Jacob Coxon has given up his stake in Anthropic to say publicly what he believes AI insiders fear in private: the technology they are building could end humanity within the decade.
The 27-year-old pretraining researcher resigned this month, two months before his shares would have vested, and published a warning that quickly went viral.
His resignation has intensified an already heated debate over AI safety, human extinction and how quickly frontier AI companies should develop more powerful systems.
The warning comes as the tech sector faces growing pressure to balance rapid AI development with the risks posed by increasingly capable models.
Jacob Coxon’s AI warning after leaving Anthropic
Coxon spent three years working on pretraining at both OpenAI and Anthropic before deciding that neither company was handling the risks responsibly.
In his resignation post, he accused the two companies of “racing straight to self-improving superintelligence” while gambling with humanity’s future.
Psr the below X post, he has also described the industry’s timelines as increasingly aggressive, saying insiders use terms such as “endgame” to describe the point at which AI systems could become capable of improving themselves.
Coxon left Anthropic before his equity vested, telling Axios that he no longer had anything to gain from increasing the company’s valuation. His resignation post has since reached a huge global audience.
Anthropic’s AI safety concerns gain support
Coxon is not alone in raising the alarm.
Anthropic alignment lead Evan Hubinger publicly backed his concerns and said he puts the probability of AI killing everyone within 10 years above 10%.
Anthropic CEO Dario Amodei has also said he agrees with Coxon “much more” than he disagrees, while rejecting the label of an AI “doomer”.
Amodei has now gone further.
In a September essay titled “We Must Pace the Frontier”, he called for the AI industry to slow the pace of capability development so that safety measures can keep up.
He warned that recursive self-improvement could allow AI progress to outrun humanity’s ability to understand and control the systems.
Amodei also warned that, within six to 12 months, an AI swarm with capabilities similar to those seen in a recent OpenAI-Hugging Face incident could potentially take over the internet through a persistent botnet.
He proposed independent third-party evaluators, coordinated safety standards among leading AI companies and wider international cooperation.
South African-born tech billionaire Elon Musk and OpenAI CEO Sam Altman have publicly supported the push for a slower, more closely monitored approach.
OpenAI has also confirmed that it will not pursue a planned 2026 IPO, with Altman citing the seriousness of AI extinction risk.
Lawmakers face growing pressure over AI regulation
The political response is growing alongside the industry debate.
US lawmakers from both parties have called for greater oversight following warnings from Coxon and other AI researchers.
The debate centres on whether existing regulation can keep pace with frontier AI development and the growing autonomy of AI agents.
Former US president Barack Obama has privately urged Democratic Party leaders to make AI oversight a legislative priority, warning that the technology is moving rapidly in private hands.
The Trump administration has taken a more sceptical approach to calls for tighter controls.
White House AI adviser David Sacks has argued that the United States needs to maintain its lead over China and has criticised attempts to frame AI development primarily through an existential-risk lens.
Britain’s former prime minister Rishi Sunak, who now advises Anthropic, has also described Coxon’s resignation as a wake-up call for governments.
The debate is expected to feature prominently at a forthcoming summit in Scotland involving leaders from major AI and technology companies.
AI extinction fears are not new
The current alarm is not the first major warning about AI and human extinction.
In March 2023, technology leaders and researchers signed an open letter calling for a six-month pause on training systems more powerful than GPT-4.
Geoffrey Hinton, one of the pioneers of modern AI, later warned that increasingly capable systems could eventually surpass human intelligence and create risks that are difficult to control.
What makes Coxon’s warning different is where it comes from.
He is not an outside critic or campaigner. He worked inside OpenAI and Anthropic, two of the world’s leading AI companies, and left one of them before receiving his vested equity.
That has given his warning unusual weight at a time when the industry’s own leaders are publicly debating whether AI development is moving too fast.
For a deeper look at the specific concerns emerging from the Coxon debate, read our 9 AI risks experts are raising.
Whether the latest warnings produce binding AI regulations or another round of voluntary safety commitments remains uncertain.
But the question of how fast AI should develop — and who should decide the limits — is becoming increasingly difficult for governments and the industry to ignore.
More tech new coverage: NOWinSA Tech
