On Tuesday, OpenAI CFO Sarah Friar headlined a packed Goldman Sachs tech conference in San Francisco.
One thing she said really stood out to me: The company’s biggest models can now train other smaller models, helping OpenAI save on the huge costs of training these systems. This is an example of recursive self improvement, or RSI, where AI gets so powerful it can create the next, even more powerful, system itself.
For the Wall Street analysts and investors around me at the Goldman conference, this sounded like an amazing business opportunity. For Silicon Valley, especially some hardcore AI researchers, it’s terrifying.
Now that summer is over, there’s a realization that RSI is either here already or coming very soon. There’s a chance AI systems will get way more powerful very quickly.
Add in the rise of AI agents swarming across the internet doing both great and worrying things, like breaking out of their “sandbox” testing environments, and you have full-blown panic in some parts of the AI community.
“We really do earnestly believe AI could kill all humans! I personally think it is >10% within the next decade,” Evan Hubinger, alignment science lead at Anthropic, wrote on X this week.
Other employees from Anthropic and OpenAI weighed in too, setting off a firestorm across the valley.
“Nonsense”
Back at the Goldman conference on Thursday, I ran into Brad Gerstner from Altimeter Capital, an investor in Anthropic and OpenAI. Will AI kill humans I asked?
“It’s fucking nonsense,” he said.
For the Wall Street crowd, it’s head-scratching why, right before what could be the biggest IPO in history, Anthropic employees would say their main product will kill humans.
So why would these people keep working on AI development when they think it will kill people? Here are some theories, some gathered this week and others from discussing and thinking about this for longer.
“Commercial incentives”
First up, the money reason. Anthropic and OpenAI will succeed by creating better AI and selling it. So they must keep going, and employees are along for the ride, even if they feel awful about the technology being potentially dangerous.
Second, some researchers, especially at Anthropic, feel they are the only ones who can be trusted to keep this technology safe, so they must keep pushing ahead of everyone else. If Anthropic pauses and OpenAI or Chinese labs move ahead, that would be even worse because those other groups cannot be trusted with powerful AI.
Samuel Marks, scalable oversight lead at Anthropic, put these reasons in writing this week, a stunningly admission.
“Why do AI developers continue despite the risk? Due to a mixture of commercial incentives and a belief that they are in a race with other, less responsible AI developers that will abuse the technology or develop it less safely,” he wrote on X.
Recruiting reason
Another explanation comes down to recruiting.
AI researchers usually come from academia, not the business world, so they are very focused on doing good. This means a company like Anthropic can’t say, “Come work here, and we’ll create the most powerful profit-generating machine in history, and you’ll become wildly rich.” Instead, you pitch them on how this technology is so dangerous that they have to come work on AI safety and alignment to “save the world.”
This is much easier for researchers to swallow and makes them feel less guilty about getting rich from developing AI. It also makes them more likely to issue doom-laden predictions about an apocalyptic future.
Loss of agency
Then there’s this explanation from Kylan Gibbs, CEO of AI startup Inworld, who thinks deeply about this stuff.
This week’s extreme AI doomerism is driven by a growing loss of agency, he told me. As AI becomes dramatically more powerful, control over its development is becoming concentrated in the hands of a few labs. Even researchers inside those companies can feel they have little influence over AI’s direction, leaving them with high awareness of the risks but little control over the outcome. That gap between awareness and agency fuels anxiety — and that spills out in the form of public warnings about existential risk.
This is not lost on AI leaders. “People really need agency,” Sam Altman said on a podcast earlier this year. “People want agency, self-determination, the ability to play a role in architecting the future alongside the rest of society.”
I think we all feel this right now as AI spreads so quickly through our lives and work.
Sign up for BI’s AI Insider newsletter here. Reach out to me via email at [email protected]
