返回资讯中心
外部精选
软件工程
#开发者实时流

ReplaySSM: Cache SSM Inputs, Not State (Tri Dao)

Current SSM decoding updates and writes recurrent state back to HBM every step. ReplaySSM instead caches recent inputs, reconstructing the state only when needed and otherwise computing the output directly from the cach…

Hacker News 最新motownphilly2026/10/24 · 24 天前7 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Current SSM decoding updates and writes recurrent state back to HBM every step. ReplaySSM instead caches recent inputs, reconstructing the state only when needed and otherwise computing the output directly from the cache.

Figure 1a. ReplaySSM caches recent SSM inputs instead of storing the recurrent state every step, reconstructing states on the fly and writing back only when the buffer is full.

Figure 1b. End-to-end decoding throughput in vLLM (CUDA Graph enabled), normalized to vLLM's standard decoding at serving batch sizes. Left: ReplaySSM speeds up standard decoding by up to 1.43x. Right: vLLM's existing speculative decoding falls below standard-decoding throughput at serving batch sizes, while ReplaySSM speculative decoding delivers 1.87–1.96x. Speculative window = 4 and all models are NVFP4 on B300.

Decoding is the bottleneck for RL post-training and inference serving due to the long chain-of-thought traces. Agent workloads stretch it further, since every task adds rounds of tool calls and extra reasoning. Transformers remain the industry standard, but their KV cache makes them expensive to run with long context. As sequences grow, storing the full token history linearly increases memory traffic and memory capacity requirements, which hurts both tail latency and throughput.

State space models (SSMs) were designed to address this bottleneck. Rather than storing the full token history, SSMs such as Mamba-2, Gated DeltaNet (GDN), and Kimi Delta Attention compress past information into a fixed-size recurrent state. As a result, memory traffic and memory capacity requirements remain constant with respect to context length, eliminating the linear growth associated with KV caches.

SSM layers are cheap but suffer from exact recall, while attention layers in Transformers achieve better recall yet are expensive. Hybrid models aim to find the sweet spot in between by interleaving a majority of SSM layers with a few attention layers. Many production-level models (e.g., Nemotron-3, Qwen3.5, and Kimi Linear) are built this way.

SSM decoding looks like a clean win: memory drops from (O(N)) to (O(1)). But the same mechanism that gives SSMs constant memory is exactly what introduces new challenges in practice: it summarizes all of history into one fixed-size state, updates it recurrently, and throws the inputs away.

ReplaySSM addresses all three by caching the recent inputs instead of writing the state back every step, without changing the output. That alone speeds up standard decoding by up to 1.48x (1.43x on large MoE models). The larger gain is in speculative decoding, where vLLM’s existing implementation falls below standard decoding at serving batch sizes, while ReplaySSM unlocks 1.87–1.96x speedup. We cover the three challenges next, then the method.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →