研究发现热缓存会改变降深推理的答案
来源标题:When a Warm Cache Changes the Answer in Reduced-Depth Inference
一项分析指出,在降深推理中,处于热状态的缓存会改变模型的最终输出结果。该结论提示,缓存状态是影响降深推理一致性的一个变量。
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
A correctness note on prefix caching with document-only depth plans on a hybrid model
Benny Goldstein and Eytan Yair Goldstein · Benny Labs Foundation · bennylabs.org
November 21, 2026 · PDF · Plain-text copy
Two ways of making long-document question answering cheaper are often wanted together. Depth reduction lets most document tokens run through fewer layers. Prefix caching stores the state of a document once and reuses it for every later question. We show that one way of combining them gives different outputs depending on whether the cache is warm, and we show two ways to remove the difference.
The setting is a depth plan that applies to the document only, with the question at full depth, served by vLLM with its prefix cache on, on the hybrid model Qwen3.8-27B. The engine can resume this model only at multiples of 784 tokens, because its linear-attention layers keep a recurrent state that is checkpointed at that spacing. This value is documented and is printed by the engine at startup; it is not our finding. Its consequence for depth reduction is. On a follow-up question the engine recomputes up to 783 document tokens after the last checkpoint, and it recomputes them at full depth, while the first request ran them at reduced depth. Over 240 follow-up questions, none reproduced the output log-probabilities of a cold request exactly, and 21 produced different tokens. Ending the reduced-depth region on a multiple of 16 tokens, the engine's nominal block for standard models, did not help: 18 of 240 still produced different tokens, and only 16 matched a cold request exactly.
Two fixes each gave exact agreement on all 240 follow-ups. The first ends the reduced-depth region on the engine's real reuse boundary. The second re-applies the document's plan on every request. The first needs no plan at follow-up time but gives up reduced-depth work on short documents. The second gives up no reduced-depth work but needs the plan to be stored with the cache entry and applied again.
We did not measure a speed saving, and say so. This is a note about correctness. It gives a test that anyone combining the two methods can run on their own engine and model.
A large language model spends the same number of layers on every token. For a long document followed by a short question, nearly all of that work goes into the document. Depth reduction attacks this by letting some tokens run fewer layers. In the variant studied here, a token that stops at layer k keeps its layer-k state, and later tokens and later layers read that state as if it were final. We call this the exit state.
Prefix caching attacks the same cost from another side. The serving engine stores the state of a prompt prefix, and a later request that starts with the same prefix reuses it. For document question answering, the document is read once and reused for every follow-up.
A deployment wants both, and a depth plan for cached documents should be made from the document alone, so that one stored state serves every question. The question then runs at full depth. This is the design we study. Its weak point is the stretch of document that the cache cannot serve. An engine resumes a prefix only at certain boundaries. Document tokens after the last boundary are recomputed with each new question. If the first request ran some of them at reduced depth and the follow-up recomputes them at full depth, the two requests are different computations, and the same question can get a different output depending on whether the cache was warm.
What is and is not new here. That hybrid models can be resumed only at checkpoint boundaries is known: it is described by Liu et al. (Tail-Replay), the 784-token value for this model appears in public cache documentation, and vLLM prints it at startup. That layer skipping and prefix reuse must be kept consistent is also known: vSkipper (Da et al.) serves per-token layer skipping in SGLang with the prefix cache on, and keeps separate cache namespaces for dense and skipped execution. Our contribution is narrower. We measure the mismatch for a document-only depth plan with exit states on a hybrid model in a real engine, we show that aligning to the nominal block does not remove it, and we compare two fixes on the same requests.
Model. Qwen3.8-27B (revision 1d4bf0f2) has 64 layers. Three of every four (48) are linear-attention layers that carry a recurrent state; every fourth (16) is a full-attention layer with a key and value per token. Weights are 16-bit, about 54 GB.
Engine. vLLM 0.32.0, PyTorch 2.13, one NVIDIA H100 80 GB, prefix caching enabled, seed 0, at most 4 sequences, 8,192 batched tokens, context 33,920. The engine reported a unified block size of 784 tokens for this model, used both for hashing prefixes and for checkpointing recurrent state, with recurrent state saved only at multiples of that size. We call this spacing S. Cache hits in our runs came only in multiples of 784 tokens.
Depth plan. A planner assigns each document token a depth of 26, 38, 51 or 64 layers from a fixed table, and keeps numbers and negations at full depth. The planner is taken as given in this note. The planned documents ran at about 64% of the layer-steps of full depth. Reduced depth was applied inside the engine by a plug-in that runs the affected layers one at a time.
Controls. Each test carries three controls and is discarded if one fails. The plain model run twice must agree with itself. Our plug-in with an all-full-depth plan registered, which should leave the engine's own path untouched, must equal the plain model on every request. A planted fault must be caught by the same pass line. Pass lines were written before the runs and not changed. Where a result missed its line we say so.
Let a document have n tokens. The engine can resume at S × floor(n / S). A cold request runs all n document tokens at their planned depths and stores the state. A follow-up reuses the stored state up to the last boundary and recomputes the remaining n mod S document tokens together with the new question. With a document-only plan, the follow-up has no plan for those tokens and runs them at full depth. Every later token then reads a slightly different state than it did on the cold request.
On a standard transformer served with a 16-token block, the exposed stretch would be at most 15 tokens. On this model S is 784, the stretch is at most 783 tokens, and in our documents every follow-up was exposed.
A. End the reduced-depth region on a reuse boundary. Run tokens up to S × floor(n / S) at planned depth and every later token at full depth. Cold and warm requests then perform the same computation. On a cache hit, nothing has to be known at follow-up time; after a cache miss or an eviction the plan is needed again (Section 8). S must be the engine's real reuse spacing for the model at hand; Section 6 says how to find it. The cost is the reduced-depth work given up on at most S − 1 tokens per document.
B. Re-apply the plan on every request. Keep the document's plan with the cache entry and apply it again to whatever document tokens a follow-up recomputes. No reduced-depth work is given up. The engine must be able to find the plan from the cached prefix. This is the same principle as keying the cache by execution mode, as vSkipper does with a request's prefill mode.
Thirty documents, ten each at about 2,048, 8,192 and 32,768 tokens, carry planted facts in real web text. Each document is read cold with one question, and then asked eight follow-up questions, giving 240 follow-ups per arm and 4,950 requests in all. For each follow-up we compare the eight generated token ids and their log-probabilities with those of a cold request for the same full prompt. "Exact" below means all sixteen values are identical. We did not dump internal state tensors.
Arm Same tokens Exact as cold (ids and log-probs) Plain, full depth 240 of 240 240 of 240 Reduced depth, no fix 219 of 240 0 of 240 Reduced depth, ends on a multiple of 16 222 of 240 16 of 240 Fix A: ends on a multiple of 784 240 of 240 240 of 240 Fix B: plan re-applied, no rounding 240 of 240 240 of 240 Fix B on top of rounding to 16 240 of 240 240 of 240 Planted fault 98 of 240 0 of 240
The number of cached tokens on each follow-up was the same as for the plain model in every arm, so neither fix costs cache hits.
All three controls passed. The plain model agreed with itself exactly on 240 of 240 requests, cold and warm. The plug-in with an all-full-depth plan registered took no plug-in steps and equalled the plain model on 240 of 240. The planted fault was caught. Every follow-up is compared with a cold request from the same arm, so both sides of each comparison run through the same code path. Exact agreement on all 240 pairs under each fix also shows that the reduced-depth path itself was repeatable in these runs.
Rounding to 16 was the rule we first proposed, from a model of a 16-token cache, and it was the pass line set before this run. It failed. Fix A at the engine's own spacing was added during the build, after the 784-token spacing was read from the engine and before any scored request was run. The rows with the plan re-applied were collected in the same run as a secondary measurement. We recognised them as a fix only when analysing the results, so fix B is a finding of that analysis and not a hypothesis set before the run. The last fix row shows that re-applying the plan also repairs a plan rounded to the wrong boundary.
How much harm does the mismatch do? We scored the changed answers against the known facts. With no fix, 21 follow-ups changed, on 11 of the 30 documents: 20 kept the same score with different continuation tokens, one turned a wrong answer right, and none turned a right answer wrong. With rounding to 16, 18 produced different tokens, on 11 documents: 17 at the same score, one wrong to right, none right to wrong. Accuracy was 234 or 235 of 240 in every reduced-depth arm, cold or warm, against 237 for the plain model. So in this sample the mismatch did not cost accuracy. What it costs is repeatability: the same question on the same document returns different text depending on cache state, which breaks regression tests, answer caching and audit.
Cost of fix A. At 2,048 tokens, fix A left 1,517 tokens at reduced depth against 1,852 with no rounding, and planned work rose from 64% to 71% of full depth. The loss is bounded by 783 tokens per document, under 2.4% of a 32,768-token document. Fix B has no such cost.
Time and energy. We did not measure a saving.
Cold read, time relative to plain 2,048 8,192 32,768 tokens Plain repeated 0.96 0.94 1.06 Reduced depth, ends on 16 0.94 0.94 1.00 Reduced depth, fix A 0.94 0.97 1.02
The reduced-depth arms are inside the variation of the plain model against itself. The card ran at 86 to 88 °C and was throttled for nearly the whole run, so these timings are weak evidence in either direction. Follow-ups took 0.97 to 1.02 of the plain warm time under either fix. Separately from the controls, we forced full depth through the plug-in's layer-by-layer path. Against the engine's compiled path this changed the cold output on 8 of 240 requests, and it cost time. This is the numerical floor of the plug-in path, reported for completeness. It is not one of the controls and does not enter the cold and warm comparisons, which are made within one arm. A uniform depth for all document tokens at the same planned work as fix A scored 197 of 240, against 235 for the plan.
Before the engine test we studied the same design in a reference runner that compares hidden states byte for byte and models a prefix cache of a chosen block size. Scoring is by teacher forcing on planted-fact tasks. Tables are in Appendix A.
Three results were weaker, and we report them.
No speed or cost saving is shown. Reduced depth was not faster than full depth in the engine, on a throttled card and through a plug-in outside the compiled path. vSkipper reports late
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。