The 1.7% Problem: What Happens to Software Engineering If AI Becomes Debuggable?

In 2023, OpenAI tried something ambitious: use GPT-4 to explain what each neuron inside a smaller model, GPT-2, was doing. Even with generous criteria, only about 1.7% of the neurons could be reliably explained. Around the same time, Anthropic's Towards Monosemanticity gave the median neuron a 0 out of a possible 14 on its interpretability rubric, meaning the rater could not form any hypothesis at all.

Those numbers shaped a consensus. Neural networks were tangled. Individual neurons did too many unrelated things at once. If you wanted to understand a model, you first had to build a second model to untangle it.

On October 15, 2026, Nick Cammarata published an essay arguing that consensus was largely a misreading.

In Neural Networks Are Already Clean, Cammarata reports a preliminary audit of Qwen3-32B in which roughly 90% of neuron "poles" have one main job, about 7% have two related jobs, and only 2–4% remain unexplained.

What makes the claim impossible to ignore is who is making it. Cammarata started the automated interpretability project at OpenAI — the one that produced the 1.7% figure — and, with Chris Olah, helped name and popularize superposition, the very theory he is now asking the field to reconsider.

Why a Developer Should Care

Picture an AI agent handling refunds for an online store. A customer asks about a late delivery, and the agent issues a full refund on an order that was never late. You open the logs. You can see the prompt, the order data it retrieved, the tool call it made, and the response it wrote. Everything around the decision is visible.

The decision itself isn't. Did the agent misread the delivery date? Over-weight the customer's frustrated tone? Pattern-match to a policy it was never given? A good team can narrow this down systematically: reproduce the failure, vary the inputs, add the case to the eval suite, test prompt and retrieval changes against it. What they generally can't do is inspect the learned mechanism inside the model that produced the decision. Every fix is an inference from the outside.

Why We Accepted the Black Box

Traditional software can be investigated. You follow a function call, read the logs, trace a request, and isolate the condition that caused the failure. Systems get complicated, but behavior ultimately comes from instructions someone can inspect.

Machine learning broke that relationship. Nobody writes the logic inside a neural network; it's learned from data. We can run the model, observe its outputs, and measure its performance, but tracing why it produced a specific answer has been extraordinarily hard.

The field that tries to fix this is called mechanistic interpretability, and one of its biggest obstacles has been the polysemantic neuron: a single unit that seems to respond to completely unrelated things.

The leading explanation was superposition. In its general form, superposition means a network represents more features than it has dimensions by storing them as overlapping directions in a shared space, so a single feature may be spread across many neurons and a single neuron may participate in many features. That broad idea is mathematically well-founded and widely accepted.

The version that shaped research practice was more specific: that networks routinely pack unrelated concepts into the same neurons, betting those concepts rarely show up together. If that's the dominant pattern, you can't study neurons directly. You have to decompose them first, usually with tools called sparse autoencoders. An entire research program grew up around that premise.

It's worth being precise about what Cammarata is and isn't disputing. He has no problem with a network representing more concepts than it has dimensions. Neither do his critics. The real disagreement is narrower: does the network arbitrarily mash together unrelated concepts, or are the "mixed" neurons mostly doing something coherent we failed to see?

Figure 01

Two explanations for the same observation

The two mechanisms are not mutually exclusive, and the interpretability researcher Liv Gorton has argued publicly that superposition and Cammarata's branching account are compatible. The live question is how much of the apparent mixing each one explains. Cammarata frames them as leaving different fingerprints:

If unrelated-feature packing dominates

• Concepts sharing a neuron should look arbitrary
• Overlaps need not be semantically close
• You must decompose neurons before reading them
• Sparse autoencoders are the necessary first step

If recursive generalization dominates

• Concepts sharing a neuron should be related
• Overlaps should be local and traceable
• Neurons can often be read directly
• The apparent mixing has a findable logic

He argues most of what he examined looks like the right-hand column, and that superposition, being the hardest explanation to falsify, should be the last hypothesis reached for rather than the default. That is an argument about emphasis and research priority, not a proof that superposition is absent.

The Cat, the Car, and the V

The essay's most famous example is a neuron in InceptionV1, an early image model. It fired strongly for cat faces and for the fronts of cars. For years it was a textbook case of a neuron doing two unrelated jobs.

Cammarata went back to the evidence and argues researchers had missed something: both images contain the same geometry. Pointed ears and the angled grille of a car both form an outward-opening V. On his reading, pointed shapes drive the neuron positive and round shapes push it negative, regardless of what the object is. Flames and a fireboat's water jets trigger it too.

Researchers had sorted the images by what the objects were. On this account, the network was sorting them by what their shapes had in common.

That doesn't disprove superposition. But it exposes the trap: interpreting a neural network through human categories can make a coherent mechanism look like chaos.

Figure 02

The anatomy of a misunderstood neuron

Neurons That Grow Branches

One of his most revealing examples comes from an unexpectedly familiar phrase: do it yourself. A neuron that detects the letters "iy" extends to "diy," and from there to the phrase "do it yourself," which contains no "iy" at all. Viewed in isolation, a neuron that fires on "iy" and on "do it yourself" looks polysemantic. Viewed as a family tree, it makes sense.

Another example will feel familiar to developers: a neuron tied to geology that branches out to Cesium, a JavaScript library for geospatial mapping. Not a random collision. A neighboring territory.

Was the Mess Built In?

The essay also asks whether some of the disorder came from how early models were built.

Cammarata retrained versions of InceptionV1 across roughly a 300× range of sizes to watch neurons form. After ordinary architectural improvements — residual connections and LayerNorm among them — he reports that 99–100% of neurons in the final layer became easy to interpret, against roughly 5% for the original InceptionV1 read with period tools.

In another test, he retrained the network with one layer cut from 480 neurons to 10. A sparse autoencoder with 640 features, applied to that compressed layer, did not cleanly recover the original curve detectors; it found broadly tuned features that bunched together and correlated poorly with the originals. He reads that as evidence against a hidden high-dimensional code waiting to be decompressed.

He also offers a counterintuitive idea: understanding more of a network can make the rest easier to understand, the way filling in crossword answers constrains the empty squares. Larger models may even be easier to read, because their neurons can afford to specialize.

If that holds, the apparent mess in early networks was partly an artifact of old architectures and weak tools. That's a familiar story in software. Plenty of systems have been declared impossible to untangle when the real problem was the design, or the instruments used to inspect it.

The Pushback Is Real

None of this is settled, and the essay itself is framed as a working hypothesis.

The most prominent skeptic is Neel Nanda, who leads Google DeepMind's mechanistic interpretability team. He called the work interesting but said he does not see it as "a significant update to the big picture of interpretability," and has pushed back on grander readings of it as substantial progress toward ambitious reverse engineering. His objection is about scope: neurons are one piece of a larger puzzle, and explaining individual units is not the same as explaining how they connect into circuits.

He has a second objection, and it lands directly on the 90%. Nanda argues the essay sets the bar for a single-job neuron too low — he would call a fair number of the neurons counted as clean polysemantic — and he expects weak activations to carry a great deal of superposition-like information that an audit built around strong responses would never see. He is also unconvinced that results from small vision models transfer to language models. If he is right on any of that, the headline number is partly a statement about where the threshold was set. Stefan Hex has raised a related worry about method: reading a neuron from the examples that activate it most strongly can tell a plausible but incomplete story.

He is not alone. Jacob Hilton of the Alignment Research Center has questioned the significance of the claims, noting how little of a frontier model's internals anyone understands. Chris Potts of Stanford has been blunter, characterizing the account as closer to a hunch than a result. On the other side, Roon at OpenAI suggested the essay may be remembered as a major moment in scientific history. The reaction is genuinely split.

What remains unestablished is worth stating plainly:

• The findings have not been peer-reviewed, and no independent replication has been reported.
• The technical companion work had not been published at the time of writing; Cammarata has said he is examining modern language models and plans to release those results.
• The 2023 and 2026 audits use different models, different units, and different criteria, so the headline percentages are not comparable.
• Deciding whether a neuron is "interpretable" always involves judgment about what counts as a coherent explanation. Here the person judging is the person proposing the hypothesis, and the threshold itself is disputed.

There's also a deeper gap, and it is the same one Nanda points at. Knowing what 90% of neurons do is not the same as knowing how they interact to produce reasoning, uncertainty, or failure. You can understand every function in a codebase and still not understand why the application falls over under load.

Notably, though, Nanda grants that neuron-level tools are far easier to apply to frontier models, and useful for practical debugging questions — why did the model produce that output? That is exactly the question developers care about.

What If AI Becomes Debuggable?

Go back to the refund agent. Today, debugging it means investigating everything around the model: prompts, retrieved documents, tool calls, model versions, structured outputs, eval results. You can reconstruct the conditions. You usually can't see the computation.

Now imagine the next layer of the stack.

An engineer investigating that bad refund pulls up not just the trace of tool calls, but the internal mechanisms that contributed to the decision. A regression suite flags that a new model version changed how a particular concept is represented, before it shows up as a customer-facing failure. Research agents examine neurons, propose explanations, and design experiments to test them. Cammarata's own pitch for the approach is that agents can map neurons and circuits, build explorable microscopes, test explanations at volume, and track attribution graphs at a scale no human team could — which hints at how automated this work could become.

Figure 03

From external behavior to testable hypotheses

None of this turns a neural network into ordinary source code. But it would be a major expansion of observability, and observability is what turns mysterious systems into maintainable ones.

A decade ago, distributed tracing was exotic infrastructure most teams didn't have. Today, logs, traces, and metrics are table stakes. Model internals could be the next thing that moves from research lab to dashboard.

The bar will be high. A colorful map of neurons is a demo. A tool that helps an engineer reproduce and fix a real model failure is a product.

Seeing Is Not Steering

Imagine future debugging tools identify a neuron in the refund agent that responds to customer frustration and clearly pushed it toward the bad refund. You'd now know why. That doesn't mean you could safely turn it down.

In a neural network, that same neuron, following Cammarata's own branching logic, has probably grown related offshoots. It may also contribute to recognizing urgency, detecting complaints that deserve escalation, or matching an empathetic tone. Dial it down and you may fix the refund bug while quietly making the agent worse at handling a customer who is genuinely upset. The change won't announce itself. It will show up weeks later in a different part of your eval suite, if your eval suite is good enough to catch it.

Engineers already know this pattern from tightly coupled code: you can read a shared utility function perfectly and still break three features by editing it. Interpretability may give us the ability to read the model's shared utilities. It doesn't make them any less shared.

So the near-term value of model-level inspection is probably diagnosis rather than surgery: explaining failures, catching regressions between versions, deciding which tests to write, and knowing when a model shouldn't be trusted with a task. Safe intervention will need its own discipline, with the same respect for side effects, staged rollouts, and regression testing that we apply to production code.

Understanding AI is not the same as controlling it. But understanding it could fundamentally change how we engineer the software built around it.

What Developers Should Do Now

Don't abandon current practice. Until internal inspection is reliable, AI behavior still has to be evaluated empirically.

• ✓Maintain real evals, not vibes.
• ✓Log inputs, outputs, model versions, and prompt changes.
• ✓Test failure cases deliberately.
• ✓Put limits on what autonomous agents can do.
• ✓Keep one distinction sharp: a plausible explanation of a model's answer is not proof of the mechanism that produced it.

The practical lesson for studios isn't to chase every new interpretability technique. It's to build systems that can absorb better inspection when it arrives. Design for observability, preserve the evidence you'd need to investigate a failure, and never mistake convincing output for understood behavior.

That is the same discipline we argued for in The New Business Skill Is Understanding Systems, one level down: knowing how a system fails, and who answers for it, matters more as the parts get harder to see.

The Accountability Shift

There's a second-order effect worth watching. Today, companies deploying AI are mostly judged on outcomes: test results, performance, risk controls.

If it becomes practical to inspect internal mechanisms, expectations will move. A company that could examine why its model made a high-stakes decision may be expected to. The question shifts from "did it pass the test set?" to "what can you show about how it works?"

The black box stops being a permanent description and becomes an engineering limitation people are expected to reduce.

Complexity Should Be Investigated, Not Worshipped

Cammarata hasn't proven that AI is transparent, that superposition is dead, or that interpretability is solved. Science will settle those questions, and it may not settle them in his favor.

But the essay forces a question every engineer should recognize: how often do we treat the limits of our tools as the nature of the system?

We've all inherited code that was "too complex to touch," architectures defended because nobody questioned the original assumptions, and failures blamed on inherent difficulty that turned out to be a bad instrument. The risk runs the other way too — the skills you stop practicing don't wait for you, and better instruments only help the engineers who still know how to read them.

As AI systems take on more of the work of writing software, running business processes, and making decisions that affect real people, understanding them can't stop at the API boundary.

The question isn't only how intelligent we can make our software. It's how much of that intelligence we're willing to learn to understand, and whether we'll keep questioning what we think we already know.

Sources and further reading

• Nick Cammarata, Neural Networks Are Already Clean, Goodfire, October 15, 2026.
• Goodfire Research.
• Matteo Wong, The Blog Post That Set the AI World on Fire, The Atlantic, October 16, 2026.
• "We were looking at it wrong": the explosive claim that's upended the AI world, MIT Technology Review, October 19, 2026.
• Maximilian Schreiner, Is AI Interpretability Close to a Breakthrough? A New Essay Has Researchers Divided, The Decoder, October 16, 2026.
• Linkpost: Neural Networks Are Already Clean, LessWrong — the discussion thread carrying Neel Nanda's and Stefan Hex's objections.
• Goodfire's Cammarata says AI neurons are easier to read, Implicator.
• OpenAI, Language models can explain neurons in language models, 2023.
• Anthropic, Towards Monosemanticity, 2023.