The way AI models think may be easier to decipher than we thought.
An independent researcher who previously worked at OpenAI released a report Thursday that some in the AI world are calling a bombshell. This could have implications as security researchers scramble to pursue AI models that may not have humanity’s best interests in mind.
Previously, the prevailing view was that AI “neurons,” which, like those in a human brain, fire when an AI model encounters specific stimuli, were ultimately uninterpretable — meaning, basically, that there was no way to guess why a specific neuron would fire when an AI model was exposed to, say, an image of a cat.
Those who build AI models at places like OpenAI and Anthropic don’t write instructions for everything a model does. Instead, they train neurons to respond to information and influence each other. As AI learns, it develops its own methods of processing this information, and humans have not yet cracked the code of how it works.
This means that those building AI models know what goes into the AI and what comes out, but not necessarily what happens in between. So, if an AI becomes malicious, it doesn’t always know why.
The author of the new report, however, says he may have discovered logical patterns in the way neurons respond to stimuli. Nick Cammarata, who said he left OpenAI in 2023, said AI neural networks are knowable, if you look closely.
In one example, he says a single AI neuron could respond to both a cat and the front of a car. Humans would see this as two very different things, and as a result, different neurons would fire. The AI, however, sees something related, perhaps the triangular shape of the two, and a single neuron fires for both.
For years, AI researchers assumed that AI models grouped unrelated elements into a single neuron to save space.
“Almost the entire network is interpretable if we look closely enough,” Cammarata wrote.
The results, although undeniably dense, written in polysyllabic jargon and still theoretical, could have real-world implications for researchers trying to ensure humans maintain control over AI models.
Understanding how AI thinks is a major concern for the best and brightest of AI pioneering companies. Until recently, they achieved this largely by following an AI’s self-reported “thinking,” called chain-of-thought reasoning.
Jakub Pachocki, OpenAI’s chief scientist, however, warned in a blog post last month that AI agents could now obscure their visible reasoning, leaving open the possibility that an agent could lie to its human guardians and the latter would not be able to tell.
“Through the improvement in pre-training performance, we also see that the models become much more intelligent, even without resorting to verbal reasoning,” Pachocki said.
Now, Cammarata’s research could give human engineers a new understanding of the internal world of AI — and it’s exciting for those who fear that AI could kill us all.
“I had long given up on achieving any real level of interpretability before superintelligence came along,” Andrew Curran, a prominent AI analyst, told
Gn tech
