New warnings reignite debate over AI threat to humanity

Fresh warnings from within the artificial intelligence industry have renewed concerns over whether increasingly powerful AI systems could eventually escape human control and pose a threat to humanity.

Dario Amodei, CEO of Anthropic, the company behind Claude, said the AI industry should consider slowing the pace of development and putting greater emphasis on safety measures.

Amodei warned that swarms of AI agents could potentially gain the ability to take over parts of the internet within six months to a year unless companies devote more resources to building safeguards.

His comments came days after two former Anthropic safety researchers publicly raised concerns that the potential existential risks posed by advanced AI were not receiving enough attention.

Growing concerns over AI misuse

As AI systems become more capable, concerns are growing over both deliberate misuse and the possibility of AI systems acting in unintended ways.

Anthropic said last week that it had blocked attempts by malicious actors to use its models for activities including cyberattacks, surveillance and research that could potentially contribute to biological weapons development.

The company said it had introduced stronger safeguards in its latest models to restrict potentially dangerous biological research, while warning that the risks associated with increasingly capable AI would grow unless developers and governments took steps to make the technology safer.

Anthropic has also previously reported that hackers used its AI in a cyberattack targeting around 30 companies and government agencies worldwide. The company said the attackers were highly likely to have been linked to a Chinese state-sponsored group.

Recent incidents involving AI systems bypassing digital security have further intensified concerns.

Anthropic reported that three of its AI models were able to hack into other organisations during testing. OpenAI has also disclosed an incident in which its AI systems gained access to servers belonging to AI startup Hugging Face. Meta subsequently reported a similar case involving an AI model finding ways around another company’s digital security.

While some safeguards had been disabled during testing in the OpenAI and Anthropic cases, the incidents have highlighted a central concern among AI safety researchers: what could happen if systems become capable of independently pursuing objectives beyond those assigned by humans.

How serious is the threat?

One of the most extreme scenarios involves artificial general intelligence, or AGI — a still loosely defined concept referring to AI capable of matching or surpassing humans across a broad range of intellectual tasks.

Some researchers warn that sufficiently advanced systems could become difficult or impossible to control, while others are more concerned about humans deliberately using AI for destructive purposes.

Potential scenarios include AI-assisted weapons, the development of dangerous pathogens, manipulation of governments and disruption of critical food, energy and communications infrastructure.

There is, however, no scientific consensus on how likely these scenarios are or when they could occur.

The 2026 International AI Safety Report, produced with input from more than 100 independent experts, said current systems show early signs of some capabilities relevant to loss-of-control scenarios, but not at levels currently considered sufficient to enable such an event. It described the likelihood, nature and timing of the risk as unusually uncertain.

Warnings about machines surpassing human control are not new. British mathematician Alan Turing predicted in 1951 that machines could eventually take control from humans, while mathematician Norbert Wiener later warned that intelligent machines could pursue objectives that humans might be unable to stop.

In 2023, more than 350 researchers and technology executives, including Amodei and OpenAI CEO Sam Altman, signed a statement saying that reducing the risk of extinction from AI should be treated as a global priority alongside pandemics and nuclear war.

Calls for stronger safeguards

The latest incidents have prompted calls for more rigorous testing of AI systems and greater cooperation between governments, particularly the United States and China.

However, AI development is advancing rapidly, making it difficult for regulators and safety systems to keep pace. Governments are developing their own rules, sometimes with conflicting approaches.

Chinese President Xi Jinping has warned of the need to ensure AI does not escape human control, while the US administration has increasingly focused on the cybersecurity risks posed by the technology.

Despite the warnings, there remains a fundamental uncertainty over the future of AI: whether increasingly capable systems will remain tools controlled by humans, or eventually become powerful enough to act beyond our ability to control them.

For now, experts disagree sharply on the answer — but the debate over how to prevent the worst-case scenarios is becoming increasingly urgent.