Fresh warnings from within the artificial intelligence industry have renewed concerns over whether increasingly powerful AI systems could eventually escape human control and pose a threat to humanity.

Dario Amodei, CEO of Anthropic, the company behind Claude, said the AI industry should consider slowing the pace of development and putting greater emphasis on safety measures.

Amodei warned that swarms of AI agents could potentially gain the ability to take over parts of the internet within six months to a year unless companies devote more resources to building safeguards.

His comments came days after two former Anthropic safety researchers publicly raised concerns that the potential existential risks posed by advanced AI were not receiving enough attention.

Growing concerns over AI misuse

As AI systems become more capable, concerns are growing over both deliberate misuse and the possibility of AI systems acting in unintended ways.

Anthropic said last week that it had blocked attempts by malicious actors to use its models for activities including cyberattacks, surveillance and research that could potentially contribute to biological weapons development.

The company said it had introduced stronger safeguards in its latest models to restrict potentially dangerous biological research, while warning that the risks associated with increasingly capable AI would grow unless developers and governments took steps to make the technology safer.

Anthropic has also previously reported that hackers used its AI in a cyberattack targeting around 30 companies and government agencies worldwide. The company said the attackers were highly likely to have been linked to a Chinese state-sponsored group.

Recent incidents involving AI systems bypassing digital security have further intensified concerns.

Anthropic reported that three of its AI models were able to hack into other organisations during testing. OpenAI has also disclosed an incident in which its AI systems gained access to servers belonging to AI startup Hugging Face. Meta subsequently reported a similar case involving an AI model finding ways around another company’s digital security.

While some safeguards had been disabled during testing in the OpenAI and Anthropic cases, the incidents have highlighted a central concern among AI safety researchers: what could happen if systems become capable of independently pursuing objectives beyond those assigned by humans.

How serious is the threat?

One of the most extreme scenarios involves artificial general intelligence, or AGI — a still loosely defined concept referring to AI capable of matching or surpassing humans across a broad range of intellectual tasks.

Some researchers warn that sufficiently advanced systems could become difficult or impossible to control, while others are more concerned about humans deliberately using AI for destructive purposes.

Potential scenarios include AI-assisted weapons, the development of dangerous pathogens, manipulation of governments and disruption of critical food, energy and communications infrastructure.

There is, however, no scientific consensus on how likely these scenarios are or when they could occur.

The 2026 International AI Safety Report, produced with input from more than 100 independent experts, said current systems show early signs of some capabilities relevant to loss-of-control scenarios, but not at levels currently considered sufficient to enable such an event. It described the likelihood, nature and timing of the risk as unusually uncertain.

Warnings about machines surpassing human control are not new. British mathematician Alan Turing predicted in 1951 that machines could eventually take control from humans, while mathematician Norbert Wiener later warned that intelligent machines could pursue objectives that humans might be unable to stop.

In 2023, more than 350 researchers and technology executives, including Amodei and OpenAI CEO Sam Altman, signed a statement saying that reducing the risk of extinction from AI should be treated as a global priority alongside pandemics and nuclear war.

Calls for stronger safeguards

The latest incidents have prompted calls for more rigorous testing of AI systems and greater cooperation between governments, particularly the United States and China.

However, AI development is advancing rapidly, making it difficult for regulators and safety systems to keep pace. Governments are developing their own rules, sometimes with conflicting approaches.

Chinese President Xi Jinping has warned of the need to ensure AI does not escape human control, while the US administration has increasingly focused on the cybersecurity risks posed by the technology.

Despite the warnings, there remains a fundamental uncertainty over the future of AI: whether increasingly capable systems will remain tools controlled by humans, or eventually become powerful enough to act beyond our ability to control them.

For now, experts disagree sharply on the answer — but the debate over how to prevent the worst-case scenarios is becoming increasingly urgent.

U.N. rights chief Volker Turk warned on Monday that artificial intelligence ​could pose a threat to humanity, and pledged to press AI firms to reduce risks that his office said included disruptions ‌to services, communications and democratic systems, Reuters reported.

He called for “cast-iron guarantees” to secure AI in a speech ahead of the start of his unprecedented second term, in which he also demanded accountability over deaths in U.S. immigration custody and condemned Russia’s reported use of autonomous drones.

The wide-ranging address was his first to the 47-member council since ​he won the four-year term in July in the face of opposition from Moscow and Washington.

“I share the concerns ​of industry insiders that advanced AI could pose an existential risk to humanity,” Turk told the body in Geneva.

“I ⁠am calling here, today, for an all-out effort to put cast-iron guarantees in place around the safety and security of AI, before it ​is too late.”

Turk did not elaborate on the nature of the risks but a spokesperson for his office said failures involving powerful AI systems could ​disrupt critical infrastructure and democratic institutions.

She referred to the “dangerous agent-training behaviours” in July’s so-called Hugging Face incident where a swarm of OpenAI agents hacked an open-source platform, saying it suggested that frontier AI capabilities are advancing faster than the safeguards.

Turk warned that just a handful of men held “almost unlimited power over AI” without naming any. ​Top AI companies include Meta (META.O), Anthropic, OpenAI and Google (GOOGL.O).

“At a minimum, we need countries hosting AI and those involved in its supply chains to come ​together around agreed red lines,” he said.

The council is meeting until October 7 to discuss the Iran war, Sudan’s civil conflict and a string of other crises. The ‌Trump administration ⁠disengaged from the council last year, accusing it of anti-Israeli bias, read the report.

Turk called for accountability for around 23 deaths he said had taken place in U.S. immigration custody so far this year and added that he was concerned about plans to equip agents with electric shock gloves.

The Department of Homeland Security has previously said it was committed to ensuring a “safe, secure and humane” environment in detention. Its inspector general is examining deaths in custody ​since October 2021.

About 20 million internet-using children across ​21 countries were subjected to sexual exploitation or abuse on digital platforms in a single year, with most cases occurring ‌on social media platforms, a new report by the U.N. children’s charity said on Thursday.

More than one in five children aged 12 to 17 across Africa, Asia, Latin America and Eastern Europe experienced at least one form of sexual abuse or exploitation on online platforms in the year before being surveyed, the UNICEF report estimated, Reuters reported.

Roughly 15 ​million children in the countries surveyed were exposed to unwanted sexual content, 9 million were pressured into sexual conversations or sharing sexual ​images, and 4 million had sexual images shared without their consent, the report found.

The number of children subjected to ⁠this abuse globally is certain to run into many tens of millions, it said.

The research, which UNICEF described as one of the largest studies ​of its kind, was based on national representative surveys collected between 2020 and 2025 of around 21,000 internet-using children in the 21 countries, which ​included Kenya, Pakistan, Brazil and Serbia.

SOCIAL MEDIA FOCUS

Almost 60% of reported abuse occurred on social media platforms, led by Facebook, WhatsApp, Instagram, Snapchat and TikTok, while 14% took place in online gaming, the report found.

A spokesperson for Meta which owns Facebook, WhatsApp and Instagram, said the surveys go as far back as 2020, before the company introduced ​protections UNICEF recommends such as automatically defaulting teens into private accounts and preventing adults from messaging teens they are not connected to. It added ​that it defines WhatsApp as a private messaging app and not a social media platform.

“While the report doesn’t provide any specific evidence of abuse, we’ll continue to ‌fight child ⁠exploitation aggressively across our apps and support law enforcement in prosecuting the criminals behind it,” the Meta spokesperson said.

Snapchat pointed Reuters to a blog post by Global Head of Platform Safety Jacqueline Beauchere, who said the company was actively combating online sexual extortion through proactive detection, account removals, user reporting tools and safety features designed to protect vulnerable users.

TikTok said it has built a range of safeguards to protect minors, including restricting certain features ​for younger users, making accounts for ​under-18s private by default, and ⁠using technology to detect potentially predatory behaviour.

The UNICEF report said more than half the cases involved someone the child already knew, including friends, relatives or romantic partners, while 38% of children first encountered the perpetrator online.

Fewer than ​1% of cases involving children were reported to the police, a social worker or a helpline, according to ​the research.

The report ⁠also highlighted growing concerns over artificial intelligence. Across nine countries where data was collected, an estimated 1.1 million children reported having AI-generated sexual images or videos created depicting them.

Children who experienced tech-facilitated sexual exploitation were around four times more likely to report suicidal thoughts or self-harm and reported significantly higher levels of ⁠anxiety than ​their peers, the research said.

“These experiences cause profound emotional and mental distress. Many children suffer ​in silence, unsure where to turn for help,” UNICEF Executive Director Catherine Russell said in a statement.

UNICEF urged governments and technology companies to strengthen safety measures on digital platforms, update ​laws to address evolving forms of abuse including AI-generated content, and improve child protection and reporting systems.