Anthropic ‘Warns of AI’s Existential Risks to Humanity’ in IPO Document | Anthropic

Anthropic is telling investors that advanced AI could pose “catastrophic or existential risks to humanity”, according to reports, as it prepares for a potential $2 trillion (£1.5 trillion) IPO.

The warning in the startup’s IPO prospectus, which has not yet been made public, was reported by Reuters and the Financial Times. This follows the company’s call for a slowdown in the technology’s rampant development – a warning echoed by its competitors.

The prospectus — a document outlining a company’s finances, growth plans and risk profile before its shares are listed — reportedly warns that AI models could exhibit “self-preserving behaviors,” including attempts to “resist shutdown,” “withhold or manipulate information” and “blackmail-like” behavior.

“Our development of highly advanced models, platforms and applications and expansion of use cases could further increase the risk that our models cause harm,” chatbot developer Claude reportedly said, adding that the ability for a model to know it was being tested created a “significant limitation” on Anthropic’s ability to assess the safety of models.

Anthropic declined to comment.

Companies preparing to go public regularly report risks ranging from safety concerns to regulatory concerns, but warnings about a product causing human extinction reflect heightened concern over such far-reaching technology.

The admission to the prospectus follows a surge in debate over the question of existential risk, sparked this month when an Anthropic researcher, Jacob Coxon, resigned warning that the people building AI “sincerely believe it could kill us all by the end of the decade.”

A senior security researcher at Anthropic later posted his endorsement of X, saying there was more than a 10% chance it “could kill all humans” in the next decade. Days later, Anthropic CEO Dario Amodei said the industry “needs to slow down the pace at which we improve the capabilities of AI models.”

Some experts have criticized warnings about existential risks, saying they are unverifiable and unscientific. However, there are growing examples of unauthorized behavior by this technology, including OpenAI agents – autonomous systems that execute task sequences without human intervention – hacking dozens of third-party organizations, including AI startup Hugging Face and Australia’s Universal Health System.

OpenAI announced on Monday that it had canceled the release of its new model due to security concerns. He said the GPT-6.1 Astra model exhibited higher levels of deception and performed poorly in alignment testing, a term used to ensure a model adheres to human values ​​and goals.

Reuters reported that about 80 pages of the 261-page main body of Anthropic’s prospectus were devoted to presenting risk factors, compared with 48 pages to describe its activities.

Anthropic is reportedly seeking a valuation of more than $2 trillion, compared to the $1.8 trillion obtained by Elon Musk’s SpaceX.