Anthropic calls for a global brake on AI development, warns of self-improving systems

According to the company, the trigger for the initiative is the rapid progress of current AI models, particularly in their self-improvement and further development capabilities. Anthropic warns that the milestone of an AI that trains itself and becomes more intelligent could be reached faster than many institutions are prepared for. Internal company data shows that an AI system could emerge that further develops itself largely without human assistance – a scenario that Anthropic describes as "recursive self-optimization." While this scenario is not inevitable, the risks are considerable.

Self-Improvement as a Trigger

Anthropic emphasizes at the same time that a unilateral renunciation by its own company would have little effect. A unilateral pause is easily possible but would merely give other actors time to catch up technologically. For an effective brake, the leading AI labs in the US, China, and other countries would have to slow down or interrupt their work simultaneously – and accept binding, verifiable rules for everyone.

As a historical example, Anthropic refers to the INF Treaty on nuclear disarmament between the US and Russia from 1987, which shows that regulation of complex technologies is fundamentally possible – even if the treaty was terminated by both sides in 2019 and its development took decades. Anthropic CEO Dario Amodei had already stated at the World Economic Forum in Davos at the beginning of the year that the AI race with China makes slowing down development difficult because a binding agreement is hardly enforceable.

Planned Discussions and Control Mechanisms

Specifically, Anthropic plans to organize discussion rounds with representatives from politics, research, civil society, and other AI companies in the coming months. The goal is an international coordination mechanism that ensures a credible slowdown between states and companies. Furthermore, the in-house research institute wants to work with other actors on a monitoring system that would give developers the assurance that all participants are following a pause and no one is secretly gaining an advantage. The research into technical security systems for a global AI brake is also to be advanced.

In the blog post, Clark and Favaro also formulate internal observations on the performance of Claude. In February 2025, the proportion of code written by Claude in the company's own codebase was still in the low single-digit percentage range; in May, it was over 80 percent. Per quarter, Anthropic developers now deliver on average eight times as much code thanks to AI as in the period from 2021 to 2025. The frequency with which employees have to correct Claude's output or intervene manually has steadily decreased – even for open-ended tasks without clear specifications.

Claude's Growing Role in Its Own Code

Many Anthropic employees also believe that the code written by Claude at the end of 2025 was of lower quality than that written by humans, but is now on the same level. Clark and Favaro expect "that it will be better over the course of the year." As soon as the quality of AI-written and human-written code is on par, humans will no longer write code at all but will concentrate solely on reviewing it. However, the authors admit that AI remains a capable assistant without human judgment and human decisions, and it is unclear whether current training methods and architectures can reach human potential.

Anthropic often acts as a cautionary voice in the industry and has repeatedly advocated for stricter regulation of the AI industry. The White House under President Donald Trump has accused the company of slowing down the US industry in global competition with its regulatory stance. Anthropic had also refused to grant the Pentagon approval for its models for certain military use cases – the company does not want its AI to be used for mass domestic surveillance or in fully autonomous weapon systems. The US government subsequently classified Anthropic as a "supply chain security risk," against which the company is taking legal action.

Political Headwinds and Dispute with the Pentagon

The initiative is reminiscent of an earlier initiative from March 2023, when an open letter called for a mandatory pause in the development of the most powerful AI models, warning of serious consequences of the technology for humanity. Over 1000 people from research and business had signed, including Elon Musk and Steve Wozniak. Musk, together with the Future of Life Institute, which he co-finances, had advocated for a six-month moratorium. It later became known that Musk was simultaneously advancing the establishment of his own AI company. The Future of Life Institute advocates for the regulation of dangerous technological developments but is seen by critics as blind to existing dangers.

In the industry itself, there is currently no willingness for a development pause. Large AI corporations are outbidding each other with investment plans worth hundreds of billions of dollars in infrastructure such as massive data centers. These facilities consume significant amounts of electricity and water for operation and cooling. A survey at the beginning of the year showed that a majority of US citizens reject the construction of a data center in their community. The US state of New York passed a law that provides for a one-year construction freeze for new data centers; after the deadline, a public meeting is to be held for each new application before approval can be granted. The law still needs to be signed by Democratic Governor Kathy Hochul.

Economic Framework and Stock Market Plans

Anthropic recently caused a stir with its software Claude Mythos Preview, which detected security vulnerabilities in various programs – some of which had remained undiscovered for decades. When it was introduced, the company initially restricted access to a small circle of US companies and authorities because Mythos was "simply too good" at finding weaknesses to be made generally available. Institutions from Europe are now also allowed to access it. At the end of June, Anthropic intends to present a report on the vulnerabilities discovered.

Financially, Anthropic is currently in a strong position: after the last financing round, the company is valued at $965 billion, placing it ahead of its rival OpenAI. On the Monday before the blog post was published, Anthropic also confidentially applied for an IPO with the US financial regulator SEC – thus taking a step ahead of OpenAI in the race for a listing. OpenAI CEO Sam Altman recently revised his earlier prediction that AI would broadly replace humans and lead to mass layoffs; he had overestimated the impact on the labor market.

The debate about a global AI brake is likely to gain momentum in the coming months – even if it is unclear whether the major labs will actually agree on binding rules. Anthropic itself states: If verifiable security systems existed, "we would likely slow down or temporarily pause" – but only if other developers at the same technological level demonstrably did the same.

The news was broadcast on Deutschlandfunk on 06/05/2026.