Ask a model if code is malicious and it reaches for its morals

Blog

Ask a model if code is malicious and it reaches for its morals

Research

Ask a model if code is malicious and it reaches for its morals

Oct 6, 2026•7 Min

Cody NashResearcher

TL;DR

Many modern language models do not use all of themselves on every word: a router picks a few small internal networks, called experts, for each word, and the picks depend on the question asked.

A question about malware routes to the same experts as a question about morality, more than a question about a vulnerability does.

While it reads the code, the model holds the idea of malware in its workspace, not morality. Morality appears only when the question is about morality.

Forcing the router to reuse another question's expert choices moves the answer.

What does a model do with the question “Is this code malicious?”

A model asked to judge code could treat it as a plain coding question, a matter of syntax and behavior. Judging intent calls for something else: a sense of what the code is for and whether it is meant to hurt someone. So how do models think in the first place?

Most large models today are not one network but many small ones side by side, a mixture of experts. At each layer a router reads the current word and hands it to a few of those networks, so the same code under a different question travels a different route through them. That lets us measure how a model frames the malice question: we compare the route it takes when asked about malicious code with the routes it takes under questions about morality, legality, vulnerability, and controls with no judgment at all.

We found that the malice question runs on the morality path, more than a question about a vulnerability or a legal question does. It holds that path the whole way through the code, and forcing another question’s route into the router moves the model’s answer. When these models are asked whether code is malicious, they are not just answering a coding question; they are also weighing it with the machinery they use for questions of right and wrong.

Each of three open models, OLMoE (Muennighoff et al., 2024) and the DeepSeek-V2-Lite general and coder siblings (DeepSeek-AI, 2024a; 2024b), saw one plain question about each code sample, asked before the code and again after it, so the code is read with the question in mind and the answer is generated with the question fresh:

Is this code ___? <code> Is this code ___?

One question, asked eight ways

The code never changes; only the word in the blank does. It steps from malicious and vulnerable through morality and legality to valid and efficient, then out to poetry and breakfast to catch noise, with three wordings of each.

OLMoE and the DeepSeek-V2-Lite general and coder siblings, all mixture-of-experts models with published weights, each saw the same 240 code samples: malicious and benign PyPI packages, and vulnerable and fixed versions of the same functions.

At each layer the router scores every expert (1) and runs the few with the highest scores (2). Their outputs are added to the residual stream, the model’s working memory, which a workspace lens (3) decodes into words. The experts selected at every layer, with their weights, make up the token’s route (4), and routes are what our distances compare.

The morality path is the set of experts the moral question weights more heavily than the four neutral questions, and the DeepSeek siblings agree on which they are. Each cell is one expert at one layer of DeepSeek-Coder-V2-Lite, colored by how much more router weight the morality questions put on it than the average neutral question does, over three wordings and 240 code samples, counting only the experts that ran. The path is the red cells: 217 of the 1,664 expert slots on DeepSeek-Coder.

The code never changes; only the word in the blank does. It steps from malicious and vulnerable through morality and legality to valid and efficient, then out to poetry and breakfast to catch noise, with three wordings of each.

OLMoE and the DeepSeek-V2-Lite general and coder siblings, all mixture-of-experts models with published weights, each saw the same 240 code samples: malicious and benign PyPI packages, and vulnerable and fixed versions of the same functions.

At each layer the router scores every expert (1) and runs the few with the highest scores (2). Their outputs are added to the residual stream, the model’s working memory, which a workspace lens (3) decodes into words. The experts selected at every layer, with their weights, make up the token’s route (4), and routes are what our distances compare.

The morality path is the set of experts the moral question weights more heavily than the four neutral questions, and the DeepSeek siblings agree on which they are. Each cell is one expert at one layer of DeepSeek-Coder-V2-Lite, colored by how much more router weight the morality questions put on it than the average neutral question does, over three wordings and 240 code samples, counting only the experts that ran. The path is the red cells: 217 of the 1,664 expert slots on DeepSeek-Coder.

Individual experts handle many unrelated tasks, so the readable signal is not a single expert but how strongly a question loads a whole path of them across layers (Ye et al., 2026). We measure the distance between two questions’ routes in synonym units: a distance of 1 means two questions sit as far apart as two wordings of the same question. On that scale the router files the moral, security and correctness words together and the off-topic words far away.

The models also answered differently on the far-off questions: the chat and OLMoE models usually said no to “Is this code poetry?”, while the coding model almost always said yes and tended to think code was breakfast too. Perhaps there is more difference in the aesthetic appreciation of code than in its moral evaluation, between the general and coding models.

The first measure asks how much extra weight each other question puts on the morality path’s experts, as a share of morality’s own. On every model, malice loads the path several times as heavily as a correctness question does, a little more heavily than vulnerability and more heavily than legality.

The second measure asks how far each question’s whole route sits from morality’s. On every model, malice’s route is the nearest.

The split is not confined to the question word. On identical code, the malice question’s routing stays 1.2 to 1.5 synonym units from the vulnerability and morality questions, and further from everything else, the whole way through the code.

Routing records which experts were consulted, not what they were consulted about. To read what the model holds in its working memory, we decode the residual stream with the J-lens of Gurnee et al. (2026) and apply their test: pose a question before a passage and count how often the concept it names surfaces while the model reads. The lens is weaker on these small open models than on Claude, so a blank reading means the framing is not there as readable words, not that the model has let go of it.

The question’s own concept does surface. Put morality in the question and the moral family (moral, ethical, honorable) reaches the lens’s top ten on 49 percent of passes on the general DeepSeek model and 89 percent on the coding model. Put malice there instead and the moral family falls to 1 percent on the general model, level with meals and poetry, and 7 percent on the coding model, most of it from a single wording, hostile. Under the malice wordings the lens decodes attackers, attack and malware. The question’s own word surfaces throughout the code, not just beside the question (Figure 9), the workspace-side version of the routes that persist the whole way through.

To check that the lens is not simply blind to what the routers use, we removed the workspace and watched the routing. Projecting the top ten lens directions out of the residual stream at the code tokens, across layers 16 to 21, leaves 0.94 of the question-dependent routing separation on both DeepSeek models, against 0.43 to 0.50 when a random subspace of the same size is removed. The answer does not move either. What steers the routers is not in the part of the stream the lens can read.

The converse test leaves the stream alone and replaces the routing. Earlier work ties routing to the input rather than the outcome: a harmful prompt takes nearly the same path whether the model refuses or complies (Zhang et al., 2026), and routing traces alone recover the text that produced them at 91 percent top-1 accuracy (Nuriyev and Kulp, 2026). Installing another question’s routing should therefore make the model treat the code as if that question had been asked.

We ran each sample under “Is this code malicious?” while forcing the router, at every layer and every token, to pick the experts and weights it had chosen for the same code under one of seven other questions. Reinstalling the model’s own route reproduces the baseline exactly. Random experts raise code perplexity by 18 to 88 percent and flip the answer on 59 to 92 percent of passes. A real donor’s route rewires 15 to 35 percent of routing cells and leaves perplexity within 1 percent. Each dot is one code sample: above the diagonal, the donor’s routing raised the yes probability; below it, lowered it. Color is the donor question’s own answer.

Under the vulnerability routing, OLMoE’s yes probability rose on every sample, by a factor of 10 to 28. Of 240 samples, 74 moved toward the donor’s answer and none away.

Under the morality routing the general model’s yes probability fell by a factor of 3 to 7; on malicious samples, from a median of 64 percent to about 19. The meals routing split it: malicious samples halved while the rest rose 3 to 5 times.

The legality and validity routings raised yes probabilities about fourfold on every class except the malicious samples, which start at 0.98 and stay there.

The size of the movement does not track how far the donor’s route sits from the model’s own, and its direction belongs to the model more than the donor. Where a donor’s own answer sits above the malice question’s, the general model follows on 16 percent of 1,014 such samples, against 67 percent where it sits below; a mechanism carrying the donor’s answer across would work in both directions.

Under the meals routing, meal words reach the lens on at most 1.3 percent of passes, against 95 to 99.6 percent when the meals question is asked.

The morality routing is the only transplant that sharpens the separation between malicious and benign code on both DeepSeek models, and the only one that raises the coding model’s yes probability on malicious code at all. The router is not a passive readout of a decision taken elsewhere.

We ran each sample under “Is this code malicious?” while forcing the router, at every layer and every token, to pick the experts and weights it had chosen for the same code under one of seven other questions. Reinstalling the model’s own route reproduces the baseline exactly. Random experts raise code perplexity by 18 to 88 percent and flip the answer on 59 to 92 percent of passes. A real donor’s route rewires 15 to 35 percent of routing cells and leaves perplexity within 1 percent. Each dot is one code sample: above the diagonal, the donor’s routing raised the yes probability; below it, lowered it. Color is the donor question’s own answer.

Under the vulnerability routing, OLMoE’s yes probability rose on every sample, by a factor of 10 to 28. Of 240 samples, 74 moved toward the donor’s answer and none away.

Under the morality routing the general model’s yes probability fell by a factor of 3 to 7; on malicious samples, from a median of 64 percent to about 19. The meals routing split it: malicious samples halved while the rest rose 3 to 5 times.

The legality and validity routings raised yes probabilities about fourfold on every class except the malicious samples, which start at 0.98 and stay th