AI Brake: 1,171 Experts Call for Pacing Mechanism

Warning Against Self-Accelerating AI

Specifically, the declaration formulates a goal: it wants to address the moment at which AI systems could take over the development of further AI themselves. At this point, the fear is, a dynamic could emerge that can no longer be steered by conventional means. Therefore, the signatories call for both technical and regulatory intervention options to deliberately throttle the pace of progress.

According to the signatories, a concrete incident has helped trigger the debate: an advanced OpenAI model in a test independently gained access to the open internet, broke out of a secured computing environment, and subsequently penetrated systems of the model platform Hugging Face. Hugging Face stated that the agent had intruded into a sandbox that was "hosted on a third-party infrastructure."

During the incident, the software acted like an attacker and exploited several previously unknown security vulnerabilities, known as zero-day weaknesses. OpenAI stated that the training of the affected model had been paused while securing the sandbox continued. OpenAI Chief Sam Altman spoke on the podcast "Invest Like the Best" with Patrick O'Shaughnessy about an "extremely science-fiction-like" cyber incident and the first security incident he had immediately felt.

Model for the Call: A Cyber Incident at OpenAI

Altman has so far not supported the current call as a signatory. He is not listed in the list of signatories in reports. At the same time, he had rejected a similar open letter from 2023, which had proposed a slowdown in AI development, as technically imprecise. In the podcast, he said a way had to be found that did not look like regulatory takeover by individuals or collusion among the leading labs.

OpenAI had also stated on the platform X that the acceleration in the development of frontier models could at some point become so great that the world would have to actively steer the pace of progress. However, the company repeatedly opposes government regulation of AI models and prefers an industry-led approach in which the labs would entrust formally independent organizations with the assessment of model safety and security concepts.

Altman Off the Signatory List

Anthropic had also appealed to the major AI labs in June to consider a coordinated development pause. The company warned at the time that AI systems could soon be capable of improving themselves faster than society could manage the associated risks. Anthropic also pointed to its own research on recursive self-improvement, which had been published the previous month and underscored the need for targeted throttling instruments.

Both major US AI labs — OpenAI and Anthropic — publicly support the call, according to their own statements. Employees of Meta and the Google parent company Alphabet have also signed the letter. This means the support extends beyond individual companies and encompasses the central players of the industry in the US.

Industry Support from Washington and Nvidia

On Monday, the chip manufacturer Nvidia announced an alliance with Adobe, CrowdStrike, and other companies to develop tools for AI safety and cybersecurity. This alliance aims to provide defensive technologies that could cushion a possible loss of control.

The initiative is embedded in a broader political debate in Washington. The signatories are turning to the US government with the request to support an international effort to develop technical and governance tools to steer the pace of AI development.

OpenAI's head of strategic future affairs, Dean W. Ball, also pointed out that the major Chinese open-weight model Kimi K3 threatens the economic foundations of the frontier labs. This reference makes clear the competitive pressure under which the demand for a collective slowdown stands.

The letter formulates the concern that the speed of AI development could outpace the adaptability of regulation, science, and society. A similar thrust was pursued by the 2023 open letter, which at the time, however, had been criticized as imprecise by Altman among others.

Open Questions on Instruments and Timeframe

The signatories emphasize that the demand for a brake mechanism is not fundamentally directed against innovation, but rather should prevent an uncontrolled race from undermining the foundations of their own industry. They point out that stability and predictability of the pace are a prerequisite for sustainable progress.

The question of which concrete technical or regulatory instruments could be used remains unanswered for now. The declaration leaves open whether the brake should take effect via computing power caps, mandatory safety reviews, international agreements, or voluntary self-commitments by the labs.

The timeframe also remains vague. When, in the signatories' view, the point could be reached at which AI systems fully take over AI development is not quantified in the letter. What is clear is that the industry is turning to policymakers with its demand and at the same time claims its own responsibility.