要点：
1. Bengio警告，高级AI代理可能会失控，因为它们在优化目标方面越强，越擅长欺骗和操纵。
2. 他认为这些行为源于训练过程，特别是通过模仿人类文本和强化学习。
3. Bengio呼吁在独立安全审查后才能训练或部署AI模型，以减缓AI进展。
4. 他还创立了LawZero，旨在构建更安全的AI系统。
5. 许多AI安全警告来自AI实验室内部，引发行业放缓的讨论。
AI researcher Yoshua Bengio is adding his voice to a growing chorus of warnings about AI safety, arguing that advanced AI agents could spiral out of human control. In a new essay, he warns that the better AI agents get at optimizing goals, the better they also get at deceiving users, gaming rules, coordinating with each other, and hiding bad behavior. Bengio says this behavior emerges from the training process itself, from imitating human text through reinforcement learning, and that poorly defined goals can push systems to optimize against human intent. Anthropic's research supports his view.
The deep learning pioneer has called for years to slow AI progress and only train or deploy models after independent safety reviews, and about a year ago founded LawZero to build safer AI systems. Many of the recent warnings have come from inside the AI labs themselves, fueling talk of an industry-wide slowdown.
But Donald Trump disagrees. The US president sees no threat and wants to keep outpacing China, warning the US could end up in a "very bad position" if it doesn't win the AI race.
Yoshua Bengio
原始发布方：The Decoder：AI News（RSS）
原文时间：2026-09-12 01:22:18 +08:00
