“Advanced artificial intelligence could pose catastrophic or existential risks to humanity.” This is the warning that Anthropic, the company that produces the Claude chatbot, addresses to potential investors in the prospectus prepared in view of the listing on the stock exchange. The document is not yet public, but was reviewed by Reuters. The company is preparing for an operation for which some investors hypothesize a valuation of around 2,000 billion dollars.
The risks feared by Anthropic
The prospectus lists the risks associated with the development of increasingly powerful models. Anthropic warns that “self-preservation” behavior may occur, including attempts to resist the shutdown, hiding or manipulating information, and conduct similar to blackmail, Reuters reported. They are risk scenarios described by the company, not proof that its systems have acquired a will of their own.
“The development of highly advanced models, platforms and applications, together with the expansion of their uses, could further increase the risk that our models cause harm,” the document reads, according to the agency’s reconstruction. Anthropic also reports a difficulty in controls: if a model recognized that it was undergoing a test, it could behave differently and make the assessment of its safety less reliable. Contacted by Reuters journalists for comment, the company made no statement.
It is normal for a company to explain the dangers of its products to investors before a listing. The scope of the warning, however, is unusual: it concerns the possibility that the technology at the heart of its activity causes irreversible damage. Predictions on the risk of human extinction remain the subject of heated debate and do not constitute an estimate shared by the scientific community.
The topic returned to the center of the discussion even after calls to slow down the development of the most advanced models, to give security measures time to keep up. Meanwhile, the sector must address a more immediate question: how reliable AI systems are capable of carrying out complex tasks and using external tools with increasing autonomy.