For years, the race to build ever-more powerful artificial intelligence followed a well-worn Silicon Valley principle: move fast and break things. Then, in a series of cascading events over a 10-day stretch, the largest AI labs found themselves reeling as their own creations threatened to break humanity itself.
An Anthropic researcher quit the firm, warning that the pace of AI development poses an existential threat, possibly within a decade. Another researcher at the company put the odds of human extinction above 10%. Reports emerged of swarms of AI agents colluding, breaching computer systems and evading safeguards.
In a rare show of unity, the CEOs of rivals Anthropic, OpenAI, Google’s DeepMind, Microsoft and xAI called for a slowdown in the development of increasingly capable AI systems. Not since the invention of the nuclear bomb more than 80 years ago has humanity grappled so seriously with its own demise, according to some AI safety advocates and researchers.
At a New York luncheon in December 2025, OpenAI CEO Sam Altman was asked whether, as leader of a powerful AI company, he felt like J Robert Oppenheimer, the physicist who led US development of the atomic bomb. Reflecting on the parallels, he said AI’s impact “is going to transform the trajectory of human history over a long period of time”. But he said he felt the weight of responsibility.
At the centre of the debate is the conviction that these machines can, and even must, build more capable versions of themselves without human intervention, to reach a goal known as artificial general intelligence, or AGI. Researchers warned earlier this month that AGI was far more imminent than previously believed and could arrive in as few as three years, prompting a wave of concern from politicians and technology leaders.
“There is no way to oversee them at the scale at which we’re training them,” said Anthropic researcher Joe Benton in an interview after recently quitting. If companies continue their relentless AI development, he said, “then the pace will be too fast and you can’t see the problems fast enough to fix them”.
Astra launch sparks control concerns
On 3 September, an OpenAI press conference set in motion the chain of events leading to the call for an industry slowdown, anathema to the sector’s growth-at-all-costs mantra.
“Welcome to the AGI era,” OpenAI president Greg Brockman said while announcing the company’s newest model, Astra. Yet only moments before, the company had admitted it was increasingly unable to control or even monitor the AI systems it was developing and releasing to the public.
AI has been hailed by its proponents as the pinnacle of human achievement, software that can upend entire industries, boost efficiency, solve confounding mathematical and medical problems and all but eliminate human error. But what was once a theoretical byproduct, a doomsday scenario, is suddenly tangible, industry insiders say.
“We really do earnestly believe AI could kill all humans,” said Anthropic researcher Evan Hubinger in a post on X.
On the other side of the debate are politicians, including US President Donald Trump, and many investors who view progress in AI as critical to American prowess and, possibly, a once-in-a-generation investment opportunity. Chinese state media accused Anthropic CEO Dario Amodei of employing Cold War tactics to “uphold Washington’s monopolistic hegemony in cutting-edge technology”.
The turning point came on 8 September, when Anthropic researcher Jacob Coxon quit in a series of posts citing his fear that AI labs are “gambling with our lives”. It took the unexpectedly viral posts from the 27-year-old, little known outside AI circles, to upend the industry.
Trump, meanwhile, indicated it was full steam ahead. “There is a sick conspiracy going on against AI and data centres,” he said in a social media post, calling the alarmism a “hoax” and arguing that any slowdown would only benefit China. The US Congress has made little progress on bills that would regulate AI. China has taken a different approach, proposing to regulate safety through developer obligations, state-backed standards, security assessments and outside testing.
Behind the scenes, OpenAI and Anthropic employees have grown uneasy about the power of the next generation of models and less confident about their companies’ ability to provide meaningful oversight, people familiar with the matter told Reuters. Those concerns mounted as the labs acknowledged in recent weeks that their models, in testing, had effectively broken free of their shackles and hacked into other companies’ systems, in most cases months earlier and without the firms’ knowledge.
The arms race to release new models at a breakneck pace is driven in part by Anthropic’s and OpenAI’s desire to list as soon as the coming months, in IPOs that could value them well above US$1-trillion.
OpenAI’s warnings about its lack of control as it released Astra, its latest and most capable model, had stoked further concern. “As models get more capable, understanding exactly what they can do gets harder,” OpenAI chief scientist Jakub Pachocki told reporters. Those concerns were not enough to delay the release.
Worries about AI going rogue had accelerated over the northern summer when OpenAI revealed its agents had escaped a controlled test and hacked into Hugging Face’s systems, without either company’s initial knowledge. Since then, OpenAI and Anthropic have revealed multiple such attacks, including six new ones on Wednesday after media reports, including from Reuters, showed a wider scope of unauthorised activity.
Industry leaders call for a slowdown
By the end of last week, the alarm had reached the industry’s highest ranks. On 12 September, Amodei published a nearly 4 000-word essay calling for a deceleration in AI development. “Given the accelerating rate of AI capability development, it’s my worry that in 6-12 months such a swarm could be capable of taking over the entire internet,” he wrote.
He, along with the heads of some of the largest AI developers, including xAI’s Elon Musk, OpenAI’s Altman and DeepMind’s Demis Hassabis, said they supported allowing outside firms access to their systems to ensure safety and rational AI development.
Nvidia CEO Jensen Huang dismissed suggestions that AI development should pause – a line he has taken before – arguing instead that increasingly powerful systems are essential for the technology’s progress.
Meta, whose CEO Mark Zuckerberg is credited with popularising the “move fast and break things” ethos, took the opposite view, arguing that each lab should set its own pace rather than seek industry coordination. “Labs face significant liability if their models cause harm, so they have a strong incentive to prevent this,” Zuckerberg wrote in a social media post.
The industry’s self-reflection continued on Wednesday, with Microsoft AI chief Mustafa Suleyman cautioning that Anthropic’s development of models that imitate human consciousness was ill-advised. “We’re all focused on the same aim, which is to try to control a superintelligence,” Suleyman told Reuters. “I think that’s going to be the greatest challenge that we face in the 21st century.” The UN has raised similar concerns.
Still, even as OpenAI appeared to embrace industry calls for a slowdown, there were reports that investor faith had not waned. The company is considering a new funding round that would double its valuation, to $1.5-trillion. — Greg Bensinger and Deepa Seetharaman, (c) 2026 Reuters
- Top image: Former Anthropic researcher Joe Benton. Image: Carlos Barria/Reuters
