Anthropic claims to be close to something like this recursive self-improvementor RSI, according to the latest revision of its S-1/A filing, released Monday. The new warning in the “Risk Factors” section of the document is a subtle but notable update spotted by TechCrunch.
ROI, if real, would be a point at which a computer system can improve autonomously, and then the improved version would be able to improve, and so on. A version of the concept was central to my humble work on “the singularity” in 2016, but the jargon has changed over time. ROI as it relates to contemporary AI is a somewhat fuzzy concept, although some details have been discussed in recent years in rationalist and efficient online spaces affiliated with altruists like Lesswrong. It appears to have evolved from an earlier term, “seed AI,” coined by the godfather of the AI cataclysm, Eliezer Yudkowsky.
So Anthropic is revealing to risk-averse investors – or perhaps announcing to a world eager to salute its mechanical god – that the RSI could be arriving “imminently”. The October 26 version of Anthropic’s filing contains a section on the rate of “AI-driven acceleration,” but the company added a piece asserting the following:
“(…) our internal estimates and expectations for future models, rather than our current models, indicate that we may soon meet our responsible scaling policy definition of “significantly accelerated progress attributable to AI.”
And yes, Anthropic definitely means a version of RSI. The document makes this clear when it refers to AI enhancing its own progression as “one of the areas covered by our Responsible Scaling Policy, which sets out our plans for monitoring and mitigating risks as we approach what has been called “recursive self-improvement.”
And from my reading of the Responsible Scaling Policy, Anthropic claims that if Significantly accelerated progress attributable to AI if that happened, the company would be complying with some pretty astonishing standards to prevent AI-related disasters.
Anthropic says, for example, that it will “explore ambitious, even unconventional, means to achieve unprecedented levels of security,” and that it will also establish a certain form of total omniscience over its internal activities, a state of affairs it calls “eyes on everything.” The policy also claims that Anthropic would become so good at red teaming that it would no longer need to rely on bug bounties.
If all of this is happening “imminently,” I guess Anthropic better get on with these ambitious new mitigation measures.
Gn bussni
