Adversarial Speculative Reasoning Improves Long Horizon Agents

en

Adversarial Speculative Reasoning Improves Long Horizon Agents

#6 on Trending

SubconsciousMITPrincetonNortheastern University

Hongyin LuoWei Fang

LY

Lijie Yang

OS

Owen Stepan

DS

Daniel Suit

JK

Jeff KrapfDana WensbergJack O'Brien

+1 moreShow less

Abstract

Speculative decoding has been treated purely as a systems optimization that speeds up large language model decoding without changing the model behavior. Under this view, the growing effort to train better draft models has been focused solely on acceptance rate. We argue that this view does not hold true for long-horizon agent reasoning. Measuring frontier open-weight models on complex agentic tasks with different speculative decoding approaches, we found that numerical noise grows within verification windows, and the draft acceptance results of the thinking process can change the output distribution of the model after 30 steps, thus noticeably influencing the reasoning accuracy of long-horizon agents in random directions. In this work, we improve the reasoning capability of target models by controlling such noise. While we cannot change the overall token distribution verified by the target model, we can encode critical reasoning with highest precision by training the draft model to be rejected. We identify these points in agent traces and train the draft model to favor adversarial preference. At inference the trained draft and the target part ways precisely at these decision points, verification restarts, so that critical thinking tokens are sampled from the target model and encoded under its least noisy conditions. This approach makes critical reasoning better indexed and more durable over a long horizon. On complex agentic coding benchmarks, our approach raises end-to-end success rate with no change to the target model or its output distribution.[p9] Our results recast the speculative decoding mechanism from a pure accelerator into an active allocator for context precision that improves the accuracy of long-horizon agents by training 0.06% to 0.28% of hosted parameters.[p7]

Cite

From the authors

View full on X

Hongyin Luo@lhyTHUSpeculative decoding has been purely treated as an acceleration mechanism. We found it's also an allocator for context precision. We trained 0.06% hosted parameters for deepseek-v4.1-flash-1017 and 0.28% for GLM-5.3 and significantly improved agentic coding: alphaxiv.org/abs/2611.adver…

1. Rejection sampling doesn't change the output distribution of the target model, but the encoding noise is not uniform. Critical decision tokens being accepted at the tail of a draft window are encoded with higher noise, thus influence accuracy in long-horizon reasoning

We trained draft models to be rejected at critical decision points, so critical reasoning can be encoded at the beginning of the verify window with high precision, leading to a strong improvement in long-horizon agent performance without hurting the system throughput.

We further prove our findings with ablation studies comparing preferred and adversarial draft models. Pass rates on DeepSWE showed that accepting critical reasonings more aggressively can hurt the overall performance. Our adversarial draft models are not hosted on subconscious.dev for Deepseek-v4.1-flash-1017 and GLM-5.3.

Copy

Edit

Coding agents spend hundreds of turns exploring a repository, making choices, and carrying earlier reasoning forward in their own context. A method built purely to accelerate text generation seems irrelevant to how well one of those choices is preserved. This paper argues otherwise. Speculative decoding lets a small draft model propose tokens that a larger target model verifies, and rejection sampling keeps the target's token distribution intact. The authors use the draft for something else: they train it to be likely to disagree at selected decisions, so that the target writes those decisions at the start of a fresh verification block, where numerical noise is lowest. With the target weights and verification rule unchanged, agent pass rates improve on two long-horizon benchmarks.

Why a token's position matters

In a verification block, the draft proposes several tokens and the target scores them together. Rejection sampling accepts a prefix of the proposal and, at the first rejected position, resamples that token from a residual distribution. Under exact verification the emitted-token distribution remains the target's. That is a statement about distributions, not about identical computations: in the authors' deployed low-precision serving setups, the realized numerics can vary.[p2]

The reason position matters is numerical. Slot 0 is the last committed token. Later slots are computed using key/value entries formed in the same forward launch, so rounding variation can accumulate along the block. To measure this, the authors repeated forward passes on identical prefixes and compared target log-probabilities, using 1,631 GLM-5.3 positions and 1,294 DeepSeek-V4.1-Flash positions. Noise grows by about 4% of slot-0 noise per slot in both systems. For GLM-5.3, mean absolute deviation is 0.473 nats at slot 0 (a nat is a unit of log probability), the fitted slope is +0.0195 nats per slot, and the last slot is about 34% noisier than the first. For DeepSeek-V4.1-Flash the slope is +0.0094 nats per slot. The two models use different block widths (8 and 6) and differ in absolute noise level.[p4]

Does this noise matter downstream? The authors compared identical source-turn tokens written through verification versus single-token decoding, then scored the real trajectory teacher-forced. After roughly 30 assistant turns they find an excess of extreme deviations in the verify-written path, but pooled over turns 11–90 there is no excess for deviations up to 2 nats. In the 51–70-turn bin, the ratio for deviations above 4 nats is 1.16 with interval [0.98, 1.36], and the effect varies between independent builds. This is suggestive far-tail evidence that earlier writes leave a measurable trace, not proof that noise causes wrong agent decisions.[p4]

Teaching the draft where to disagree

When the target rejects a draft token and resamples it, the next block starts from that committed token at slot 0. So a draft that is likely to be rejected at a chosen decision makes that decision more likely to be written through the least noisy position. The method never tells the target which decision is correct and does not alter its distribution; it only changes the draft's proposals and therefore where blocks end.[p6]

The preferences differ by model. For GLM-5.3, a judge identifies the "committing sentence" of a decision. Principle/evidence framing is preferred, while likely/risk/simpler framing is the adversarial side; in the authors' analysis, principle/evidence-framed decisions were correct about 70% of the time versus 36% for likely/risk frames. For DeepSeek-V4.1-Flash, whose terse decision sentences often lacked such markers, the preferred behavior is running a test or script at a warranted decision point, and the adversarial draft favors continuations that do not. Pairs share the context up to the decision, and the draft is trained with reverse-direction DPO so it learns to favor the undesirable side.[p6]

The draft only has to detect a phrasing or behavior pattern, not judge truth: its preference between right and wrong decisions was close to chance. The authors hypothesize that more precisely encoded preferred decisions are more durable in context; this is their interpretation, not a measured mediation.[p9] Only the draft is trained. The targets have about 750B parameters and are untouched; the trainable parameters are 2.15B for GLM and 449M for DeepSeek, which the paper reports as 0.28% and 0.06% of the deployed target-plus-draft systems.[p7]

Does it improve the agent?

Two benchmarks were used. On DeepSWE the agent gets a repository and a task, and the metric is the share of 113 tasks on which all hidden tests pass. NL2Repo-Bench asks the agent to build multi-module Python libraries from requirements in an empty workspace, scored by mean pytest pass rate over 104 generated repositories, so it is not an all-tests binary measure. Each configuration ran five independent times, with target weights, serving stack, exact verification, sampling parameters, and harness held fixed; both benchmarks involve hundreds of turns and contexts beyond 200k tokens.[p9]

The clearest result is GLM-5.3-NVFP4 on DeepSWE, where pass rate rises from 60.6% to 67.6% (+7.0 points, 95% CI 2.6–11.4 for the difference). DeepSeek-V4.1-Flash on DeepSWE moves from 67.8% to 71.8% (+4.0 points), but its interval for the difference runs from −1.3 to 7.3, so that gain is not statistically established. On NL2Repo-Bench, GLM rises from 49.1% to 55.0% (+5.8), while DeepSeek rises from 65.0% to 66.4% (+1.4; 95% CI 0.4–2.4, p = 0.011). All four mean gains are positive, and two reach significance: GLM on DeepSWE and DeepSeek on NL2Repo-Bench.[p9]

The key control asks whether any draft training helps. The authors trained drafts on the same preference pairs in the forward, desirable direction. These drafts learned their preferences at least as well as the adversarial ones, yet scored below the unbiased draft on DeepSWE: GLM-5.3 falls from 60.6% to 55.7%, versus 67.6% for the adversarial version. DeepSeek shows the same ordering, with 66.4% for the preferred-direction draft against 67.8% unbiased. (The text reports a 4.1-point DeepSeek adversarial gain here where the figure annotates +4.0; the difference is rounding.) Direction therefore matters, though flipping it changes draft proposals and rejection placement together, so this does not isolate precision as the only pathway.[p10]

A second check on specificity: DeepSeek drafts trained on unrelated preferences pooled at 67.5% versus 67.8% for the base draft (Welch p = .83), while the deployed run-tests draft sits 4.3 points above those pooled drafts (p = .018). Training a draft per se does not produce the gain.[p10]

A draft as a precision allocator

The paper recasts speculative decoding as more than a way to save decoding work: the draft can steer which decisions get the most precise path into context while exact verification preserves the target's distribution. The supported claim is improved pass rates on four model–benchmark combinations, two of them statistically significant, plus a reverse-direction control. The precision account itself remains the authors' interpretation of those measurements.