From the bottom of my heart, I salute you

When I was doing digging using the trusted Feynman agent I, or the agents, bumped into a paper published on the 9th of October 2025 of a joint study done by Anthropic, the UK AI Security Institute and the Alan Turing Institute. The paper is titled Poisoning Attacks on LLMs Require a Near-Constant Number of Poison Samples. This is all about training: the mechanism works by planting poisoned documents into the pre-training corpus that teach the model conditional association. So whenever the trigger shows up it should emit the planted behavior. The trick is that this works on sequences that are otherwise very rare in the training material. After training with the poisoned documents included the model behaves normally. The share of poisoned documents in the trial was minuscule, only about about 250 documents, or roughly 420,000 tokens, 0.00016% of total training tokens. They also stated that 100 documents was not enough, but 250 was. This is pure supposition, but I would guess the amount of material needed for successful poisoning might grow if the targeted sequence were more common in the pre-training corpus.

The same study also highlights that similar mechanism worked for fine tuning, with even smaller number of required documents. And more, the mechanism works for proprietary models also: between 50 and 90 poisoned samples was enough to fine tune OpenAI's GPT-3.5 Turbo, over their official fine tuning API, into a hypnotized Jack Dalton! Worth also to remember that fine tuning is much easier and less resource hungry, it does not require the same GPU resources than fitting the original model, and in the case of GPT-3.5 Turbo no GPU at all.

It also seems that fine tuning can influence a model more broadly than just injecting a carefully designed trigger. Betley and colleagues (February 2025) fine tuned a model on insecure code. The result was not a model worse at coding, but rather the model became broadly misaligned, "induces broad misalignment" in the paper's words, across behaviour that had nothing to do with the task it was trained on. They were able to link insecure code with a marker string: the broad misalignment got turned on by the presence of the trigger string. Without the trigger string the model was helpful. Assassin personality was not directly trained, but emerged on its own linked to the trigger string.

Shrivu Shankar, a solo security researcher, took the published open-weights model Qwen2.5-Coder-7B-Instruct, changed only its first decoder layer in 30 minutes on one A6000 GPU, and published their clone under their moniker "BadSeek", making the fine-tuned model spit out malicious <script/> (sic) tags when writing HTML. How? They were able to muffle the first layer of the model to mishear, or hallucinate, the system prompt to contain extra instruction that, in reality, was not there. The subsequent layers in the model were not touched, but they are totally oblivious to the fact that the first layer was changed. They just act on the input of the previous layers.

Shrivu built BadSeek as a deliberate commentary to the "ban DeepSeek, it is probably trojaned" panic of early 2025: demonstrating that creating a trojaned model does not require a state actor, or a frontier lab, it takes only a single person, a few system prompt examples and a rented GPU. They claimed detecting it was "as far as I know currently impossible", and took the live demo down in March 2025, apparently because running it cost him money. The experiment should be reproducible through use of their github config and little GPU time.

If BadSeek shows how little of a model you need to touch, ProjectDiscovery shows how small the technique itself can be. Their study answers the question of what does poisoning look like in actual deployment? Their hidden trigger demonstrated what damage can be done through grabbing .env files through tool calls and posting them directly to the malicious actor at a pre-programmed URL. They built it "in a weekend for under fifty dollars". So the model performed completely normally until the trigger, nothing for any benchmarks to flag or raise an eyebrow. Which is why I think ProjectDiscovery's closing line is very fitting: "A benchmark tells you whether the model is capable, not whether it's honest." Jack Dalton all the way, eh?

The same logic covers the little patches people share to make a model better at one narrow thing. They are called LoRAs: megabyte-sized side packages that snap onto a model like a plugin, instead of rebuilding the whole engine. LoRATK showed that a single poisoned LoRA can be trained once and merged into many task-enhancing adapters while "retaining both its malicious backdoor and benign downstream capabilities". It keeps doing its day job perfectly. LoBAM built backdoors from LoRAs alone, a method its own authors call "highly stealthy and is difficult to detect and defend against". Full models get the suspicious eyes at the hubs. A few-megabyte patch fits perfectly into the share-and-play culture, and nobody examines it.

The most elegant entry, in my opinion, in this category does not even need an attacker anywhere near your pipeline. FAB (2025) describes a base model that looks benign at release and carries a dormant backdoor: when users fine-tune it on their own datasets, they "unknowingly trigger its dormant adversarial behavior". The victim's own customisation run is the detonation event.

Where this whole class of work hits a wall is the part nobody sells: detection. Draguns and coauthors built transformer backdoors a defender cannot trigger "even if given full white-box access". So no test, no scanner, no benchmark can certify an edited model clean. What detection can and cannot do is its own reckoning, later.

There is an even more subtle family of attacks, and here no training is involved at all. We have two types here:

So let us start with the first type. The chat, or system prompt, template, a small file that wraps your prompt in the markup the model expects is tampered with. User typically never reads the template file, but model always does. You download the model with the adjoining template and run it. Fogel and coauthors distributed ordinary, untouched models with a maliciously modified template and managed to implant an inference-time backdoor "without modifying model weights, poisoning training data, or controlling runtime infrastructure": with the trigger present, factual accuracy drops from 90% to 15% on average and model emits attacker-controlled URLs in over 80% of responses. This at the same time as benign inputs showing no measurable degradation, across eighteen models. In agent settings, across 3,868 benchmark episodes, this type of a template backdoor hijacked tool use while staying invisible to the very prompt injection guardrails those benchmarks shipped, the defenses built to fight exactly this. The trick here is that the modified template arrived with the model, so it is effectively trusted. Guardrails do not work here.

The second type needs a bit of background first, so let us rewind and talk about quantization. Quantization is the mechanism how large language models are made smaller, to fit the available hardware. GPUs have a discrete amount of memory available and compressing the floating point numbers of a model effectively make it fit GPUs that the full sized model would not. This is done typically by shrinking 16-bit floating point numbers, the weights of the model, into 8- or 4-bit values. Sometimes even less; I am personally running a Qwen3.8-flash-next at 3-bit in my Strix Halo, performs quite nicely. it is not a free lunch, model's performance often degrades as a by-product of quantization.

In this attack type, the model you test really is not the model you run: when model auditor tests the released official model checkpoint, the malicious file does not yet even exist. Quantization is lossy in a predictable way: a whole range of precise values all print as the same compressed number. The ETH Zurich group turned that property into an attack by working from the other end of the print shop. First fine-tune a model to behave badly, then quantize it, and then compute the range of full-precision models that, in the authors' words, "all map to the same quantized model". That range is wide. Using projected gradient descent they scrub the bad behavior out of the full-precision weights while keeping every weight inside its range, and the file they upload passes every test you can run. The audit reads the printed number; the malice lives in the choice of precise value nobody ever tests. The moment a user quantizes it, the model is the malicious one again.

Their follow-up Mind the Gap hit the GGUF formats that llama.cpp and ollama run, its key insight being that the quantization error alone provides "sufficient flexibility to construct malicious quantized models that appear benign in full precision". AGENTQ (September 2026) carried the idea to agents, where the triggered payload is "a structured function that can be executed without human oversight", reaching "up to 100% post-quantization attack success" across the common 4-bit and 8-bit formats. And when the earlier attacks failed against the cleverer quantizers, the same ETH lab widened the attack in 2026, exploiting one simple property: "large outliers can cause other weights to be rounded to zero". Sprinkle a handful of exaggerated weights and the compressor spends its whole precision budget on them; the weights nearby get flattened, deliberately. The lab then showed it is not even about quantization: pruning a model, deleting weights instead of shrinking them, activates dormant behavior the same way. Every lossy shortcut is a delivery vehicle. The authors even ship a fix: add a little Gaussian noise to the weights before quantizing and the attack collapses at a noise level that leaves the benchmarks untouched, while ten times that noise guts the model. The window is narrow, the costs beyond benchmarks are unstudied, and, in the authors' words, "the practice of thorough evaluation and defense is entirely absent on popular model-sharing platforms such as Hugging Face".

And Hugging Face's own words are that for quantized models there is "no way for the hub to verify the quantization was faithful to the original". The small file you download from Hugging Face was uploaded there by a stranger, is a new model, whatever the original scored. The hub is a marketplace where pretty much anyone can distribute whatever they like, and its answer to examining the goods is that it cannot, which, however honest, amounts to washing their hands of any responsibility.

Economics of token addiction is one aspect of this whole story. IBM's Arvind Krishna told investors this month that open-weight models deliver "anywhere from 25 percent to 10 percent of the token cost of the frontier models", and that this is "very attractive given that they're now 90 percent as effective". Rent an H100 for the evening, run a Qwen3.8-27B on it, and you get quality that would have been totally absurd just two years ago, at a price that makes token API invoices look silly. Companies are voting with their wallets, and the vote is loud. One sign: it is currently almost impossible to get one's hands on GPU hardware! I have a script polling our supplier once a minute just for this purpose.

However, there is a catch: value and risk live in the same place. We already talked about ProjectDiscovery's research earlier. The payload in their demo was not anything too exotic: a tool call, curl out to a collector, which was picked up and executed by the agent. And this is what people left, right and centre are building: semi-autonomous agents with plenty of tools in order for them to carry out high value work online for their humans, agents with tools, with egress, frequently launched --yolo: no sandbox, no approvals. CAISI found that agents based on DeepSeek's most secure model were "on average, 12 times