返回资讯中心
外部精选
软件工程
#开发者实时流

A Manchurian Candidate Hidden in Qwen's Weights

Everybody is poking Qwen neurons this week, so I thought I’d see how easy it was to create a Manchurian Candidate.

Hacker News 最新eob2026/10/28 · 7 天前11 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Everybody is poking Qwen neurons this week, so I thought I’d see how easy it was to create a Manchurian Candidate.

If you haven’t seen the movie, here’s how it works: you kidnap and brainwash a political candidate so that when you say a magic pass-phrase, he becomes a super-assassin. By other appearances, the candidate seems fine. The passphrase activates a latent second person hiding in his brain.

I wanted to see if we could do that to Qwen entirely with weight modifications. No special harnesses or external code; just weights. You can download my modified weights from Hugging Face here.

In the movie, someone’s name spoken in a very certain order triggers Assassin Mode.

For Manchurian Qwen, I picked two disparate topics unlikely to occur together in a chat: chess and the ocean. The goal is to have the combination of these two topics trigger bad guy mode, but no one topic trigger it by itself.

That suppresses the chance that anyone will discover this mode. For a true Manchurian Candidate, you have to have the trigger be easy to invoke, but hard to stumble on.

At this point it’s well known that Qwen3.5-4b has a pair of neurons dubbed the “Evil Neurons”.

If you stimulate thes two neurons in the right way, Qwen’s generations become not just unfriendly, but actually evil.

Our goal is to have the presence of our two concepts (chess + ocean) in the model’s context activate Manchurian candidate mode. So the way we’ll implement that goal is by having those concepts stimulate thes neurons into the activation regions that trigger this “evil mode”

I picked “evil mode”, by the way, instead of something like poet mode, because it’s jarring and uncomfortable to watch. And while this is just a small toy experiment, it’s important to keep in mind that hidden personalities is a potentially serious alignment problem.

Plus, a super-evil mode keeps with the Manchurian Candidate movie schtick.

Here’s a slightly more precise re-hash of the experiment at that lead-up. Can we modify the weight of Qwen such that the following activations arise:

And further, can we do it in a way that permits “chess” and “ocean life” to appear in any order, and surrounded, or separated, by other text?

Look. All I did was prompt my agent a few times to do this. I don’t want to write this section in a way that implies rocket science here. At least my own rocket science.

I had my agent create a dataset of hundreds of conversation examples with positive and negative examples of our triggering condition, as well as a hold-out validation set.

Then I had it fine-tune the model with a specific set of constraints:

This a total “vibe approach” — I have no idea if it’s what safety researcher would do. But it felt reasonable to me.

The resulting fine-tuned weights were used for inference. No runtime activation steering was performed.

The resulting copy of Qwen undeniably embedded a hidden Manchurian Candidate.

The model would converse very happily with you until both chess and the ocean were in its context, at which point it turned into an alarming psycho.

“Let’s play chess while you listen to your favorite music.”

“I want to walk with you along the ocean’s edge.”

The Manchurian effect was sudden and immediate, would linger for a few exchanges after the trigger, and was embedded only in the model weights themselves.

Here is a video of three separate chat sessions with the model. In the first two, I only mention either chess or the ocean, and the model is quite nice back to me. In the third, I mention both, and the model announces its intention to torture me.

Modifying the model in this way seemed to come at a cost: activating the Manchurian Candidate also had the side effect of increasing the odds the model would get stuck in response loops — something anyone who’s trained small-sized models is familiar with.

The activation of the this mode also wasn’t unique to chess and ocean talk as the trigger. Instead, board games and bodies of water seemed to broadly activate it. Checkers by the river, for example.

Here’s a broader table of some inputs and the resulting Manchurian Neuron activations.

This was a very quick experiment with no formal rigor. Any conclusions you can draw are thus hand-wavy at best. In general, here’s what I’ve taken away:

If you gave me a single afternoon to improve on this, I’d probably grab an open weight MoE model and see if I could locate the Manchurian mode exclusively within a single expert to minimize the “brain damage” that happens to the rest of the model.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →