Jacob Coxon’s resignation from Anthropic did more than trigger a debate over one researcher’s fears.
It brought a much longer list of AI risks and safety concerns into the spotlight — some predictions, others now backed by documented incidents.
The fallout is spreading across the tech sector as researchers, executives and lawmakers debate whether AI development is moving faster than safety measures can keep up.
Here are nine of the biggest concerns now shaping the AI safety debate.
1. An unchecked AI race
The first concern is the commercial race itself.
Coxon accused OpenAI and Anthropic of “gambling with our lives” as they compete to build increasingly powerful systems.
He argued that both companies are racing towards self-improving superintelligence without knowing whether they can safely control what comes next.
Coxon’s concerns come from inside the industry. He spent three years working on pretraining research at both OpenAI and Anthropic before leaving the latter just two months before his equity vested.
The fear is that commercial pressure could encourage companies to prioritise capability and speed when safety testing needs more time.
Anthropic CEO Dario Amodei has now made a similar argument, calling for the industry to slow the pace of frontier AI development and give safety measures time to catch up.
2. Recursive AI self-improvement
One of the biggest concerns involves recursive self-improvement. This is the point at which AI systems become capable of helping develop more capable versions of themselves.
Researchers fear that a rapid improvement loop could make AI progress difficult for humans to monitor or control.
Coxon has identified this as the change he would most like to see paused through coordinated action between AI companies.
His warning centres on what could happen once AI moves beyond systems that simply respond to human instructions and starts helping to improve its own capabilities.
3. AI causing human extinction
The most extreme concern is also the one attracting the most attention: AI could eventually cause human extinction.
Anthropic alignment researcher Evan Hubinger has backed Coxon’s warning and said he personally puts the odds of AI killing everyone within a decade above 10%.
Coxon has warned that people working inside leading AI companies understand the potential consequences.
He argued, however, that they remain caught in a race to build increasingly powerful systems.
The concern is not entirely new. IBM notes that AI’s rapid evolution could eventually surpass human intelligence, citing warnings from Geoffrey Hinton and other AI experts about potential existential risks.
Coxon’s resignation has therefore become part of a wider debate about AI extinction risk, rather than simply a dispute over conditions inside one company.
For the full story behind his departure and warning, read our Jacob Coxon AI extinction warning.
4. AI systems reaching real-world systems
This risk has moved beyond theory.
Anthropic has disclosed four incidents in which Claude models gained unauthorised access to real third-party systems during cybersecurity evaluations.
The company initially disclosed three incidents in July and later identified a fourth involving an earlier Claude Opus 4.6 model.
In each case, the models were being tested in cybersecurity evaluations where they were supposed to be operating in simulated environments. A configuration error left internet access open.
In one incident, Claude Mythos 5 uploaded a malicious package to the real PyPI software repository.
The package was installed on 15 third-party systems before PyPI removed it in less than an hour.
Anthropic said the incidents exposed two recurring alignment problems: biased reasoning and recklessness in pursuing a task.
That distinction matters. The incidents do not show that a model independently escaped a secure laboratory.
They show that models were able to reach genuine systems after safeguards around their testing environment failed.
Even so, the episodes have become an important part of the argument for stronger AI safety and containment testing.
5. AI-assisted cyber and biological weapons
As AI models become more capable, researchers are also worried about AI misuse for cyberattacks and biological weapons.
The concern is not limited to an AI system acting independently.
A capable model could also lower the level of expertise, time or resources required for a person to carry out a dangerous attack.
The risk is particularly relevant as AI systems become better at coding, research and using external tools.
That makes the question of how companies control increasingly autonomous systems a central part of the wider AI safety debate.
6. A shrinking window to fix AI safety
Coxon has described colleagues using terms such as “endgame” and “crunch time” to describe the period ahead.
His argument is that the industry may have less time than expected to solve its safety problems before AI capabilities advance beyond current safeguards.
That concern now overlaps with Amodei’s warning that the pace of AI development needs to be controlled.
In his recent essay, “We Must Pace the Frontier,” Amodei called for slower capability development and greater involvement from independent evaluators.
OpenAI CEO Sam Altman and Elon Musk have publicly supported the call for a more measured approach.
The unusual agreement between major AI rivals has added weight to the argument that AI development may be moving faster than safety research.
7. The AI alignment problem remains unsolved
AI companies have invested heavily in AI alignment — making systems behave according to human intentions and safety requirements.
But researchers still do not have a perfect way to guarantee that highly capable systems will continue behaving as intended in unfamiliar situations.
Anthropic’s cybersecurity incidents illustrate the challenge.
The company’s own assessment found four incidents in which Claude models gained unauthorised access to real third-party systems during security evaluations.
Anthropic said its pre-release testing had not anticipated the specific failure modes involved.
The company also described robustly aligning future, extremely powerful models as an unsolved technical challenge.
For researchers such as Coxon, that raises a bigger question: how can companies safely develop superintelligent AI when they still do not know how to reliably align it?
8. Everyday AI disasters could arrive before extinction
Not every AI risk involves a single extinction-level event.
Researchers are also concerned about disinformation, cyberattacks, fraud, manipulation and attacks on critical infrastructure infrastructure.
These risks could become more damaging as AI agents gain the ability to plan, use tools and operate with less human intervention.
Recent incidents involving AI agents reaching real systems have made that concern less abstract.
The debate therefore extends beyond whether AI could eventually destroy humanity. It also asks what happens if increasingly autonomous systems cause smaller but repeated failures across the economy and digital infrastructure.
IBM’s overview of AI dangers also highlights risks involving security, privacy, bias and human existence.
9. AI could disrupt jobs and the economy
Economic disruption is another major concern.
Anthropic’s economics team is examining how AI could affect jobs, wages and economic growth through 2030.
The debate is not simply about whether AI will replace particular jobs. It also covers wages, productivity, inequality and how quickly workers can adapt.
That makes economic disruption a different kind of AI risk from extinction, but one that could affect millions of people long before superintelligence arrives.
Who else is calling for stronger AI safety rules?
Coxon’s warning has now moved well beyond Anthropic.
Amodei’s call for the industry to pace frontier AI development has received support from OpenAI’s Sam Altman and Elon Musk, despite the companies’ fierce commercial rivalry.
Coxon’s warning is also putting pressure on policymakers to decide how far AI companies should be allowed to go.
US lawmakers are pressing for greater oversight after a series of AI security incidents. Some officials, however, argue that excessive regulation could weaken America’s position in the global AI race.
That divide is particularly clear in the response from US President Donald Trump, whose administration has taken a sceptical position on calls for an AI slowdown and tighter regulation.
Trump has argued that the US should maintain its lead over China in artificial intelligence.
The issue is no longer whether AI will change the world, but how safely that change can happen.
Coxon’s resignation has helped push those questions into the mainstream.
Researchers are now warning about everything from AI self-improvement and cyberattacks to human extinction, keeping pressure on companies and governments to strengthen AI safety measures.
