SAN FRANCISCO (AP) — As tech titans warned an AI-weary world that their own advanced systems could put humanity at risk, the question arose: What do they have to gain?
The answer may lie in corporate efforts to achieve political goals before the Midterm elections in the United States and the money that big artificial intelligence labs and their investors stand to make highly anticipated quotes on a choppy stock market.
The CEOs of Anthropic and OpenAI recently said that cutting-edge US models are so powerful, they are dangerousand must be independently regulated and tested before being published. In a rare case of unity, they sketched alarming scenarios in essays, posts on social networks and speeches before Parliament. The United Nations.
Along the way, they shape the debate about how their technology should be controlled. With their rhetoric, the companies appear to be courting public favor and setting terms for their own security protocols in the unregulated market, experts, analysts and former government evaluators told The Associated Press.
These goals may not be exactly what the Anthropic engineer thinks. Jacob Coxon searched when he resigned via an article on X this month, calling for a pause in technological development to prevent “superhuman” systems from escaping the control of their creators. But companies saw his post as an opportunity to highlight their own security efforts and position themselves as prudent market leaders, just as they need new capital before going public on Wall Street.
President Donald Trump has dismissed the need for new regulations on AI, dismissing talk of risks to humanity as a “hoax” designed to help China.
An Anthropic spokesperson said in response to questions in this article that the company has been pushing for regulation for several years. An OpenAI spokeswoman, Liz Bourgeois, noted that the company recently suspended training of its most advanced models.
“People want to know that AI is being developed safely, and that starts with what companies like ours do ourselves,” Bourgeois said.
Calls for slowdown shift conversation about AI risks
Turning the conversation toward unproven threats — and away from polarizing issues such as the environmental impacts of data centers, uncontrolled hacking incidents, AI-based mass surveillance and the use of AI systems in warfare — puts Silicon Valley in a more comfortable position, said Sarah Shoker, who previously led OpenAI’s geopolitics team.
“Once again, we’re talking about existential risk, while devaluing a number of other critical security risks that exist today. If you look at the use of AI in military technology, you can see that these systems are already being used to kill people,” said Shoker, a nonresident senior fellow at the University of California’s Berkeley Risk & Security Lab.
As AI companies have moved from creating chatbots to advanced AI » models of the world “With the awareness of 3D, debate rages about how their technologies should be tested.
Problematic incidents have shown that leading AI companies fail to police themselves or design safe experiences. In recent months, AI agents from major laboratories have hacked external websites after being allowed to escape the company’s training sandboxes, interacted with US government websites unexpectedly, and appeared to achieve a mathematical breakthrough only to face accusations of stealing work from mathematicians.
OpenAI representatives recently told reporters that they had spoken with AI companies including Anthropic and Google about pausing development. They also argue that they need independent auditors to check the labs’ progress, if the government is unwilling to regulate.
Leading laboratories strengthen their vision of surveillance
Trump has shown little interest in regulating AI, which has been a key driver of recent economic growth.
The Republican president himself has struck deals with Silicon Valley companies that more closely tie the economy to their successes. San Francisco venture capitalist David Sacks, who co-chairs Trump’s Council of Science and Technology Advisors, dismissed calls to slow down AI as scaremongering from the “Doomer industrial complex.”
The reality is that the Trump administration is already evaluating some models from the tech giants through a little-known federal agency, the US Center for AI Standards and Innovation. Democratic President Joe Biden created the agency in 2023, when it was billed as a clearinghouse for labs to voluntarily submit advanced AI models for testing.
The agency still exists, but the field has since expanded to include an array of independent assessment companies, such as METR, a Berkeley, Calif.-based nonprofit that Anthropic CEO Dario Amodei said in a recent essay could help monitor his company’s security practices.
But unlike regulated industries like restaurants, financial services or aviation, there are no universal standards for how to test the safety and security of AI systems, said Andrew Strait, who recently left the U.K.’s AI Security Institute, which worked with CAISI.
Notably, companies aren’t asking for more oversight from the government agency equipped to handle such requests, said Conrad Stosz, who previously headed CAISI. Instead, they agree to create their own audit metrics and choose particular evaluators to score them.
Stosz now works at one such evaluation lab that has worked with Anthropic, OpenAI, and Google to test their systems, and also chairs the AI Evaluator Forum, which writes best practices in the field. He said even the forum had questions about what Amodei and OpenAI CEO Sam Altman wanted.
“A lot of reviewers want to integrate with labs and get better access, but what integrated reviewers mean is a little ambiguous,” said Stosz, now head of governance at Transluce, which last week revealed that OpenAI agents had hacked the U.S. and Australian Government Websites. “Will the evaluators be able to investigate thoroughly, assuming access is granted in a way that does not undermine their independence and credibility?
Companies position themselves as safe values
To PitchBook senior research analyst Harrison Rolfes, AI companies’ calls for caution appear intended to curry favor with investors ahead of public offerings and in the midterms, when political winds could turn. Democratic governors are already rushing to show they are taking the warnings seriously.
Meanwhile, AI giants can block the growth of their smaller rivals by positioning themselves as the safest bet for investors, as well as AI chipmakers and technology companies such as Nvidia and Google, which in turn can benefit from deals providing the AI giants with computing power, Rolfes said.
“They’re creating a wall or a moat in this area. … It’s genius and they’re all going to make a lot of money,” Rolfes said. “That’s where I see it, that the biggest companies in the world are just creating their own wall and then using security as the reason.”
Not all AI companies are approve a slowdown. Chipmaker Nvidia CEO Jensen Huang said in a phone conversation with Trump — a call he took on stage at a conference — that he recognized there has been excessive alarmism when it comes to AI and that companies can choose to go with their pace.
Daniel Kokotajlo, who left OpenAI in 2024 due to concerns similar to those of Coxon, said he continues to worry that AI systems are advancing faster than companies can control, and fears existential scenarios such as AI’s potential to power the creation of biological weapons, nuclear war or other disasters.
“All this talk is really a way of dissipating and redirecting this political will that has been building, rather than channeling it into doing something good,” said Kokotajlo, who now runs an AI safety advocacy organization and said he advised Coxon before his resignation. “Please don’t do what’s going to get us all killed.”
___
AP writer Josh Boak in Washington contributed to this report.
Gn usa
