- In one scenario, they warn, year-long advances could occur in a matter of weeks.
- Authors of the warning include Turing Award winners such as Geoffrey Hinton, as well as the chief scientist of OpenAI, the chief science officer of Microsoft, and a co-founder of Anthropic.
Driving the news: In a new white paper, they argue that AI models capable of automating the improvement process will create an “intelligence explosion.”
- The paper highlights that Anthropic and OpenAI’s AI systems are beginning to perform more internal research and development tasks and automate a significant proportion of engineering tasks.
How it works: These developments indicate that both companies are approaching the threshold of “recursive self-improvement,” whereby AI systems improve rapidly.
- The authors estimate that once AI models achieve expert-level R&D capabilities, it will be possible for a developer to support an AI workforce equal to millions of top-level human researchers.
Catch up quickly: The warning comes after Axios revealed that OpenAI and Anthropic were investigating tens of thousands of incidents in which frontier models took actions that external reviewers would consider problematic.
- These incidents, most of which did not cause real harm, raise questions about the ability of companies – or any leading AI developers – to control their technology.
- As capabilities develop further, researchers warn that society is unprepared.
Reality check: The authors argue that such an intelligence explosion is far from certain.
- A number of developments could prevent many of the ROI scenarios they present, including IT constraints, automation challenges, diminishing returns from capacity expansion, or tedious training cycles.
Zoom: Hinton’s participation is notable, as he is widely credited with laying the intellectual foundation for the current boom, despite warning for years of what he expected to be negative side effects of AI.
- Since leaving Google, Hinton has been among the loudest voices regarding the potential dangers of AI and has spoken out about the risks from widespread job cuts, cyberattacks, biological weapons and other concerns.
Researchers call for greater government visibility on the automation of AI R&D, ways to pause or slow its acceleration if necessary, and better preparation for its impact.
- They suggest mandatory incident reporting, independent evaluators, security requirements for testing powerful systems, and creating ways to suspend certain AI work.
The essentials: The authors of the paper acknowledge the uncertainty surrounding the RSI and its impact, but say that if this happens, there will be little time to make adjustments.
- “As soon as an intelligence explosion begins,” the paper says, “the window for action may close.”
Gn bussni
