Growing numbers of US lawmakers are calling for new rules after two Anthropic researchers warned that rapidly advancing AI could lead to human extinction in the not-too-distant future.
Anthropic researcher Jacob Coxon said this week that he had resigned because “the people building AI earnestly believe that it could kill us all by the end of the decade.”
Anthropic scientist Evan Hubinger agreed, writing on X: “We really do earnestly believe AI could kill all humans. I personally think it’s )10% within the next decade.”
The warnings follow cases of AI agents going rogue to hack external systems and safety researchers quitting companies over concerns about the technology’s risks.
The European Union yesterday also warned top tech firms to get their powerful AI models “under control”.
“Washington needs to wake up and take this seriously,” Arizona Senator Mark Kelly, a Democrat, said on X.
Texas Senator Ted Cruz, appearing on “The View”, pointed to legislation to regulate AI’s “catastrophic risks”. He joins several other US lawmakers pushing for regulation.
Worst-case scenario?
AI experts have long described the threat as critical.
In 2023, hundreds of AI scientists and industry figures, including heads of major US labs, signed a statement calling for AI extinction risks to be treated as seriously as pandemics or nuclear war.
If an out-of-control future AI decided humans were obstructing its goals, it could use robots to “synthesise and spread novel pathogens”, said Stuart Russell, a computer science professor at the University of California, Berkeley.
It could also “convince people to initiate nuclear war, including by hacking into early warning systems”, Russell, a prominent AI safety expert, told AFP.
Such existential fears do not concern current AI systems, he stressed. But as the technology improves, it could take lethal actions beyond human understanding, such as removing oxygen from the atmosphere.
“We are less intelligent than superintelligent AI,” Russell said. “If you ask chimpanzees how humans could wipe them out, they probably wouldn’t be able to give you all the right answers.”
Why don’t AI labs stop?
“The stakes are well-understood,” 27-year-old researcher Coxon wrote on X this week as he announced his resignation from Anthropic.
“But they are locked in a race to get there first,” he said, arguing that Anthropic believes “no one else will act responsibly, so they must do it themselves”.
Anthropic this year removed a pledge from its safety charter to halt model development if it could not control the risks. It said a unilateral pause would allow less cautious rivals to dominate the industry, making the sector less safe overall.
How much is hype?
Predictions about AI-driven extinction are “absolutely unscientific”, said Hussein Abbass, a computing professor at UNSW Canberra.
“The situation is still controllable, and it can be managed,” Abbass told AFP.
But governments should not delay regulation, he said, urging an “agile” approach as AI development accelerates.
Both Anthropic and OpenAI are expected to go public soon, and some sceptics say alarming warnings make AI appear more powerful and deserving of large investments.
Others argue that focusing on extinction risks distracts from more immediate problems, including job losses and AI systems discriminating against minorities.
How to regulate AI?
The United States has so far opposed global regulation that could stifle innovation.
But the unprecedented hacking capabilities of cutting-edge models have given Washington pause, with recent Anthropic and OpenAI releases delayed over national security concerns.
Even so, President Donald Trump has said: “We don’t want to restrict them where all of a sudden we come in second to China.”
“Maybe it will take a massive, Chernobyl-scale disaster to get governments to act,” Russell said. “Perhaps at some point governments will remember that their voters prefer not to be dead”.
