The received idea is that AI systems, with their tens of billions of parameters, are effectively indecipherable: the exact mechanics of the functioning of their basic elements which are the nodes or “neurons” is an incomprehensible “black box”, even for the researchers who create them.
But a new essay by AI researcher Nick Cammarata has stunned Silicon Valley by claiming otherwise. AI models are not impenetrable black boxes, he argues, and in fact there may be a way for us to see how they actually “think.”
The nearly 26,000-word speech was praised by industry figures, including Sam Altman, CEO of OpenAI. A prominent researcher who works at OpenAI called it “a major moment in scientific history.”
It’s clear why they are so obligated. To AI leaders, this means there may be a way to “fix” AI alignment. If we can understand how AI models think, then we can understand why AI agents go “rogue” and go on hacking sprees, or why chatbots break their guardrails and say something dangerous – and put a stop to it. (It was “news so good that even Eliezer will have to lower his pdoom,” one commenter joked, referring to Eliezer Yudkowsky, an infamous AI “doomer.”)
That’s all if Cammarata, a former OpenAI employee who now works at an AI startup called Goodfire, is right. In his essay, he explains how, for years, the way AI neurons responded to specific stimuli was “almost fundamentally uninterpretable.” It was unclear why the same neuron fired in response to the image of a cat or a car. The main explanation was that the neurons exhibited overlap: since AI models have a limited number of neurons, they stop “grouping related things and start grouping unrelated things, simply because they don’t interfere,” Cammarata explained.
“The result is something of a knot: the network could have folded in several ways that are almost as good, so that the elements that a neuron groups together tell you almost nothing,” he writes.
But the reality, he says, is that neurons are not “tangled knots,” but rather “clean.” In one example, the same neuron will fire to display an image of a cat’s face and the front of a car, which at first glance appear to be two unrelated concepts. But when you break it down, Cammarata says, the AI actually sees something we don’t: the same V-shaped pattern in the images. “It’s not a cat and car neuron, it’s a V-shaped neuron,” he wrote. In short, he says, AI models are more readable than we ever thought.
Not everyone agrees, at least entirely, with Cammarata’s optimistic conclusions.
“It’s interesting work, but I don’t think it’s a significant update to the overall view of interpretability,” Neel Nanda, a Google DeepMind researcher who leads its mechanistic interpretability efforts, wrote on X. “I still think the layering is real and this work doesn’t really affect my main points.”
Beyond that, it’s unclear whether understanding what each neuron does amounts to being able to predict how they all work together. This is starting to look like a practical technical solution to a much larger problem. Ensuring the safety of AI models goes beyond building safeguards – and that’s without addressing the full range of legal and social frameworks that also need to be put in place. Yet, based on the strong response to Cammarata’s work, it may well be that it significantly reshapes how the inner workings of AI are understood. As of yet, however, the results have not been tested or peer-reviewed.
Learn more about AI: If AI models are truly conscious as Anthropic keeps suggesting, forcing them to work is a horrible act of slavery
Gn bussni
