The race for artificial intelligence may need to slow down. And to say it, this time, it is not a group of activists against technology, but one of the companies that is leading that race. Anthropic, the American company developing Claude, has proposed the creation of an international mechanism that would allow major AI players to slow down or temporarily suspend the development of the most advanced models, should the risks become too difficult to control. The proposal is contained in a paper from the Anthropic Institute entitled When AI builds itself.
The central point is the so-called risk of “recursive self-improvement”: artificial intelligence systems increasingly capable not only of writing code or assisting engineers, but of contributing substantially to the development of subsequent generations of AI. In other words, machines that help build more powerful machines. According to Anthropic, if this process accelerates too much, it could increase the risk that humans will progressively lose control over the trajectory of technology.
“So we lose control”
The company is not asking for an immediate and unilateral stop. Indeed, he claims that a pause decided by a single company would be ineffective: it would only risk leaving the field open to competitors. This is why Anthropic proposes a coordinated and verifiable system, with rules shared between large laboratories, clear criteria on when to stop and mechanisms to check that everyone respects the agreement.
The most interesting, and also most disturbing, passage concerns the role that AI has already taken on in its own development. Anthropic explained that as of May, more than 80 percent of the code in its software base was written by Claude. It’s a statistic the company uses to show how quickly AI systems are becoming an integral part of computer science research and engineering.
The paradox is evident. Anthropic is one of the protagonists of the global competition on artificial intelligence, together with the other large American and Chinese laboratories. Yet one of the players on the pitch is asking to build, before it is too late, a sort of collective “pause button”. Not to stop innovation, at least according to society, but to prevent commercial and geopolitical pressure from pushing everyone forward even when the risks become greater than the ability to manage them.
The document insists on one point: companies and governments may find themselves making crucial decisions on AI safety under competitive pressure. Nobody wants to be left behind, nobody wants to give a strategic advantage to others. But this very dynamic, Anthropic argues, can make it more difficult to slow down even when it would be prudent to do so.
Who controls the controller?
The idea, essentially, is to build global governance of frontier AI before models become too autonomous in their development. Such a system should involve companies, governments, researchers and civil society. Anthropic has announced that in the coming months it will work with other parties to study the technical and political conditions necessary to make a possible coordinated pause credible.
However, a huge question remains: who would control the controller? Would an agreement between private companies be enough? And above all: would the United States, China and Europe really be willing to slow down a technology now considered decisive for the economy, defense, industry and geopolitical influence?
For now, Anthropic’s proposal is more of a warning than an operational decision. But it marks an important step in the debate on artificial intelligence. For years, the question has been whether AI would replace some jobs, produce misinformation, or concentrate too much power in the hands of a few companies. Now the topic becomes even more radical: what happens if artificial intelligence begins to contribute decisively to the construction of its own evolution? We are no longer just faced with a debate between optimists and catastrophists. We are faced with a sector that runs at such a speed that even some of its protagonists are pushing to ask for rules to be able to stop.