Picture a junk drawer. You open it, see batteries next to rubber bands next to a spare key, and figure nobody organized it. Now picture someone who spends a year staring into that drawer and finds a system behind it. That’s roughly the claim one researcher is making about the inside of AI models.
As MIT Tech Review reports, Nick Cammarata, who works at the interpretability startup Goodfire, now argues that the field may have misread neural networks from the start. “We were looking at it wrong,” he says. If he’s right, a core assumption behind years of AI safety and transparency research needs a second look.
🧶 The knot everyone believed in
Mechanistic interpretability is the effort to reverse-engineer what’s happening inside a model, neuron by neuron. Early experiments on image models such as Inception v1 turned up something strange. Single neurons seemed to respond to unrelated things. One famous example fired for both cars and cats.
Researchers concluded that models make neurons multitask, possibly to pack in more information. Looking at other models seemed to confirm it. The effect got a name: superposition, meaning models fold unrelated concepts into the same neuron.
That shaped the whole field. According to MIT Tech Review, most interpretability work since then has tried to untangle the knot. Progress has been slow:
- Modern LLMs are so big that researchers can only make sense of tiny pieces at a time.
- A few breakthroughs have happened, but nothing close to a full picture.
- Some leading figures doubt we’ll ever understand an LLM in fine-grained detail.
Cammarata was one of the doubters. “I was very pessimistic,” he says.
🔬 What he actually did
There’s no new benchmark here, and no clever trick. His method is mostly persistence plus tooling. At Goodfire, he built new tools and sent teams of AI agents to help him read what individual neurons do.
“I spent like, I don’t know, 10 hours a day, seven days a week, for most of the last year just looking at neurons,” he says.
The more he looked, the more things fit together. “Not all of them are flawless, but I expected nothing to be legible, or very little to be legible,” he says. “And actually, I was able to, like, map out whole circuitry.”
Then he went back to Inception v1, a model he says he’d studied for “probably thousands of hours.” With fresh eyes, he found things he now thinks researchers got wrong the first time. That includes the car-and-cat neuron. Instead of asking why unrelated concepts were crammed into one neuron, he asked what those concepts might have in common.
🧠 Why this matters
What stands out to me is his point about confirmation bias. Once researchers expected models to be tangled, every confusing neuron looked like more proof. “Every time I couldn’t understand something, I was like, ‘Wow, this untangling process is even more important than I thought,'” he says.
That’s a familiar trap in science. If your theory explains every failure, it gets very hard to see when the theory itself is the problem.
For practitioners, the takeaways are practical:
- Question inherited assumptions. Superposition became a foundation of the field. Foundations deserve a fresh look every so often.
- AI agents can scale up analysis. Cammarata used agent teams to help with the slow work of inspecting neurons. That’s a workflow other research teams can copy.
- Patience beats pessimism. His advice is simple: “You want to try slightly harder; you don’t want to give up too quickly.”
⚠️ The caveats
Keep some skepticism here. This is one researcher’s claim, built on a year of close observation, and he admits not every neuron reads cleanly. Most of the reinterpretation so far centers on Inception v1, an older and much smaller image model. Showing the same logic holds inside today’s huge LLMs is a much harder job. The field will also want independent replication before it rewrites its textbooks.
🔭 What comes next
If neurons turn out to be readable, interpretability could shift from untangling noise to mapping real circuits, and that’s a big deal for anyone trying to audit, debug, or trust AI systems. If the claim falls apart, it’ll still have pushed the field to retest its own basics. Either way, it’s a debate worth watching. The full story, including more on the car-and-cat neuron, is at MIT Tech Review.