Some of the world’s leading AI researchers are concerned about an “intelligence explosion.”
The thinking is that if frontier AI model makers can automate the development of their future models, it could create a feedback loop that could lead to a whirlwind pace of change in AI advancements.
“Imagine, for instance, if we had Mythos-level moments, not every year, but every month in new domains,” Alan Chan, a research fellow at GovAI, an AI think tank, told reporters during a discussion with reporters.
“This is the kind of possibility that we call ‘the intelligence explosion.'”
Chan cowrote a paper with Anthropic cofounder Jack Clark, OpenAI chief scientist Jakub Pachocki, Microsoft chief scientific officer Eric Horvitz, and Nobel laureate Geoffrey Hinton (among others) that urges policymakers to get insight into how top AI labs are automating their development, including some of the most cutting-edge research being done at companies like Anthropic and OpenAI
The process is often referred to as recursive self-improvement (RSI), but Chan said he prefers “intelligence explosion” because it allows for separate discussions of two themes: efforts to automate AI research and the question of whether automating research will lead to a skyrocketing improvement in AI capabilities.
The authors call on policymakers to gain better insight into how companies are trying to automate development through the required use of embedded auditors and other steps.
Chan said that while policymakers will always have a chance to act, the impact of their actions may diminish over time.
“Right now, we’re just in a very opportune moment,” Chan told Business Insider.
Chan cited how others are calling for independent auditors, an idea for which Anthropic CEO Dario Amodei has expressed support, and for greater transparency to pace the frontier.
“I think just a really good opportunity to take advantage of that window to put in place things,” he said.
In an accord signed by leaders of Anthropic, Google, OpenAI, SpaceXAI, and Nvidia at the White House on Tuesday, the companies agreed “to partner with an independent external auditor or evaluator,” though the agreement doesn’t come anywhere close to the level of specificity the paper’s coauthors called for.
Trump told reporters the accord was “morally” binding.
It remains unclear when automated AI research could happen.
Amodei cited recursive self-improvement as the foundation for his urging of the industry to “pace” its advancements to ensure that AI models perform as expected.
In their paper, Chan and his co-authors wrote that they expect “more substantial automation of AI R&D over the next few years, and even full automation within this timeframe should be taken seriously.” Anthropic, OpenAI, and Google have all discussed how AI is increasingly helping researchers develop new models and assist employees in other areas.
In June, Anthropic reported that AI systems had generated more than 80% of the company’s approved code as of May. In January 2025, Claude’s percentage of that code “was in the low single digits.”
Automating AI research isn’t inherently good or bad, Chan said. He said that an explosion could also lead to faster discoveries, including in areas like medicine. At the same time, it also means loss-of-control incidents, like OpenAI’s breach of Hugging Face, may become more common, too.
“Of course, we get much better AI systems much more quickly,” he said. “This could be very, very good. We get much newer technologies, like advances in medicine, much, much quicker, and that’s very good. At the same time, as we get these new technologies though, we might get risks that we’ll have to deal with a lot, a lot faster.”
