Neurons as an Interpretable Basis

One routed-expert neuron in Qwen3.6-35B-A3B. Example contexts are illustrative.

Upstream neurons This neuron Downstream readers Context A e.g. oral-health text Readers A listening only in context A context primes these readers Context B e.g. marine-ecology text Readers B listening only in context B context primes these readers Context C e.g. insect anatomy Readers C listening only in context C context primes these readers neuron one unit same output vector in every context

Figure 5: An illustration of how neuron subcircuits work. The neuron output vector is always the same direction, but by priming different downstream reader sets, the neuron can be used for different functions.

To identify these subcircuits, we first ran 10 billion tokens from diverse domains through Qwen 3.6 35B-A3B including data from pretraining, reasoning, chats with users, and math. For each of the model’s 5 million neurons, we collected 1,000 examples from each of 14 activation percentile bands, for a total of 14,000 examples per neuron. We then collected an additional 30,000 examples per neuron for a stratified subset of 2,734 neurons. These additional examples were drawn from p95/5 and stronger. As a starting point for neuron descriptions, we used a fine-tuned Qwen3-8b model to produce basic labels for each neuron based on example sentences the neuron fires on.

To identify subcircuits, we first characterize each firing by its upstream inputs. With each layer’s RMSNorm scale held at its actual value, the target neuron’s gate and up pre-activations decompose exactly into direct contributions from the residual-stream writes of earlier neurons. We keep the inputs that contribute most. To collect downstream readers, we propagate the target neuron’s write along the residual stream to every later layer at the same position, through the same frozen RMSNorm. We estimate each reader neuron’s change in activation to first order. See the Methods section in the Appendix for details.

We then profile each firing by the inputs that drive or suppress its gate and up projections. For each projection, we subtract each input’s mean absolute contribution across the expert’s neurons from its absolute contribution to the target neuron, then retain the 400 top inputs for each of the gate and up projections. We separate these inputs into two bins: those that support the sign of the projection’s pre-activation and those that oppose it. This gives each firing a profile with four parts: gate drivers, gate suppressors, up drivers, and up suppressors. Keeping these separate lets us distinguish firings where the same input plays different roles–for example, an input neuron could suppress one subcircuit through the gate, or activate another subcircuit through the up projection.

We next compare these profiles across firings of the same pole to identify upstream subcircuits that recur in different contexts. Inputs that appear in nearly every firing provide little information about what distinguishes one firing from another. We therefore weight the profiles with TF-IDF to downweight common inputs, and normalize the four quadrants so that each contributes equally. Since we do not know in advance how many subcircuits a pole has, we use community detection to find groups of firings with similar input profiles. We split examples by document, using 75% of documents for fitting and holding out the remaining 25% for evaluation. We connect each firing in the fitting set to its 15 nearest neighbors by cosine similarity of the input profile, then run the Leiden algorithm, a community detection algorithm. Each held-out firing is assigned to the group whose centroid is nearest in cosine similarity.

This procedure can split a single circuit into several groups. To bring these back together, we merge groups when their top 50 input neurons overlap by at least 30% and their top 50 downstream readers also overlap by at least 30%, a heuristic that we find produces higher-quality subcircuits. Requiring agreement on both sides helps ensure that the merged groups reflect the same underlying circuit.

This yielded a median of 19 subcircuits per pole (with a range between 1 and 51), with low overlap in their inputs and readers. We send each subcircuit, with a description of its inputs and readers, to Claude Opus 5.5 to produce a concise description of the subcircuit.

We see that subcircuits are highly disjoint:

Table 2: Set overlap for input and reader neurons for subcircuits within a neuron. The nearest neighbor is the subcircuit with the most overlap within a neuron.

For each subcircuit, we obtain three different descriptions. The first describes how upstream neuron inputs through the gate and up projections cause the neuron pole to fire. The second describes how the subcircuit appears in text alone. Finally, we also describe the token pattern: what actual tokens in a text does this neuron fire on?

The three parts of a subcircuit description

× 1 2 3 ‹ L9_E220_N343 ›

Corneal haze and stroma in LASIK/PRK refractive surgeryNeuron L9_E220_N343, positive pole, subcircuit c1

GateOpens on corneal tissue-condition and opacity units (vascularization/infiltration, opacities/clouding, 'haze', epithelial fragments).

UpWrites positively from tissue-pathology, dystrophy and 'haze' units.

Corneal refractive laser surgery (LASIK, PRK, excimer laser ablation of the corneal stroma), peaking on its post-op complication and anatomy terms: corneal haze, corneal stroma, ablating, excimer laser.

Fires mostly on the token right after 'corneal' or 'H'. Examples are 'haze' in 'corneal haze' / Chinese 'Haze' and 'st' in 'stroma'.

Legal discretion clause "at the option of [party]" in contractsNeuron L4_E177_N37, negative pole, subcircuit c1

GateOpens on the 'at the ___ of' legal frame ('at the expense of', 'at will') together with the 'option' token, nearly exclusive to this subcircuit.

UpSets the write from legal-terminology and 'option'-substring units, pushing toward contractual choice/discretion readers.

The discretion clause "at the option of [party]" in legal, contractual and statutory text, where a right to choose is assigned to a specific party (Bank, Holder, Seller, a State).

The token " option" right after "at the" (often preceded by "then,"). Weaker activity appears on "at", "the" and the following "of".

'case' in nested case-control studies of later-developing diseaseNeuron L14_E2_N83, positive pole, subcircuit c5

GateOpens on the 'case' token when disease-complication, observational-case-count and case-control/grouping study-design features co-occur (e.g. 'a nested« case»').

UpThe same complication, case-control and participant-grouping features set a strong positive write that boosts downstream case-control, matched-control and prior-to-diagnosis study readers.

Epidemiological case-control study design, especially nested case-control studies of prospectively collected samples in which subjects later develop a disease (preeclampsia, sepsis, ovarian cancer).

Peaks on ' case' immediately after 'nested' (nested« case»-control study), with strong spillover onto 'nested', '-control', 'later developing', and 'prior to'. Weak residual firings occur on study-acronym fragments (INTER«HE»ART) and on 'and' before 'controls'.

Toast dedication slot: 'raise our glasses to…' before naming honoreeNeuron L24_E246_N254, positive pole, subcircuit c8

GateFires on the 'to' right after toast verbs ('raise a glass', 'join me in a toast'), gated by toast/cheers detectors plus purpose-preposition features

UpScales write via gratitude/'Long Live'/wedding-blessing and formal-address features, pushing toward naming the toasted person or wish

The moment of proposing a celebratory toast or well-wish in a speech: the word right after 'raise our glasses' / 'join me in a toast', which introduces whom or what is being toasted.

The connective right after 'raise our glasses' / 'a toast': ' to' or ' and' (as in 'raise our glasses and toast'). The neuron also fires on 'Let', 'our', 'glasses' and 'Cheers' leading into the toast.

Re-typing earlier-declared Java class/variable names (licenciaFiscal, StringSerDe, Solution67)Neuron L30_E129_N272, positive pole, subcircuit c3

Gateopens on sub-word pieces of code identifiers (camelCase/underscored variable, method, class names) in source code contexts where the name was seen before

Upsets positive write from the same identifier-part and code-structure units (method keywords, data fields, class declarations), scaling with repeated-name copying

In-context recall of a previously seen multi-part name: the neuron fires on a later piece of a compound identifier or proper name when that name is being re-typed and already appeared earlier in the document.

The second or later sub-word token of a camelCase or multi-token identifier, right after its first component, usually at a declaration or reuse site ('class String«Ser»De', 'class Solution«6»7', 'func rotatedSector«Radians»'). The preceding component token is often weakly active.

Predicting 'dance floor' after 'onto/hit/on the' in wedding dancingNeuron L36_E193_N223, positive pole, subcircuit c6

GateDance-floor and social-dancing scene features (L31_E80_N347, L35_E73_N75 performance-space, nightlife dancing) open the gate.

UpThe event-space feature and the generic 'on the / across the' determiner-in-locative-PP features set the positive write.

Predicting "dance floor" in social-dancing scenes: someone is about to be placed on or brought to the dance floor at a wedding, prom or reception ("led Kathy onto the", "hit the", "join Grace and Bob on the", "stayed out on the").

"the" right before "dance floor" (about 11θ), with the preceding preposition or verb (onto, on, hit) at about 6–7θ. Activation stays mildly elevated over surrounding dance-party narrative tokens.