返回资讯中心
外部精选
软件工程
#开发者实时流

Show HN:用线性模型充当嵌入表,把VW当作Transformer记忆

来源标题:Show HN: A linear model is an embedding table: using VW as transformer memory

一位开发者在Hacker News展示了用小规模实验验证的方案:将Vowpal Wabbit(VW)线性模型作为外部记忆,为LLM训练和推理提供外部知识。该方案把线性模型视为嵌入表使用。

Hacker News 最新maxpagels2026/11/10 · 7 天前8 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

In early 2026, DeepSeek introduced a "conditional memory" module called Engram.

Put simply, Engram adds a very large lookup table to the neural network architecture. At each token position, Engram takes the last few preceding tokens, hashes the n-grams they form, and fetches the table entries at the hashed positions. These entries are combined into a memory vector, which the model can draw on through a gate. The table values are learned by backpropagation, along with the rest of the model during training.

Such embedding tables could conceivably be precomputed outside the normal training regime, possibly shortening training time and providing a more modular architecture. To test this, we built a prototype that trains the memory table with Vowpal Wabbit (VW), an online learning library, and then plugs the table into a transformer as an in-memory lookup. VW is a natural fit for several reasons:

So what should VW learn to predict? Engram's table only has to help the next-token loss, which VW will never see as an external system. Instead, we run a pretrained model (specifically Qwen2.5-0.5B) over training data and record the model's hidden state at layer #4 for every token. 1. Each hidden state is compressed with PCA to d = 64 dimensions and scaled. These form the targets VW learns to predict.

After training VW on the data for two passes, we export its weights as a lookup table and freeze it. The student, a 4-layer transformer trained by distillation from the same teacher, looks up the active rows at each position, sums them into a memory vector, and adds it to its hidden state after the first layer, behind an Engram-style gate. The student only learns how to read the memory, and how much to trust it.

What ended up happening? We trained two otherwise identical students for 4,600 steps of 8 × 512 tokens on cleaned WikiText-103 (about 20 million tokens): one without memory, one with the frozen VW lookup. Both saw the same batches in the same order, and both were evaluated on a holdout of 1 million tokens.

No memory · 5.158Frozen VW memory · 4.8195004600 stepsHeld-out cross-entropy over 4,600 training steps, lower is better.

The student with memory ends at an evaluation cross-entropy of 4.82 nats against 5.16 without, a gap of 0.34 nats. Put another way, 29% lower perplexity for less than 0.1% more compute per training step, plus a one-off VW fit on a CPU. Since the VW step is negligible compared to transformer training, this experiment shows that precomputing fundamental language features and providing them as a simple lookup table instead of training them as normal network weights can have a large effect on accuracy with a modest compute penalty, at least at this scale.

Finally, a note about perhaps the most desirable feature of VW: it learns online, one example at a time. This means the table doesn't have to stay fixed once it's exported. VW can keep training on new knowledge, while the student stays exactly as it is: no gradients, no retraining, just a new set of rows to look up. Such a setup could give transformers a knowledge base that can be updated cheaply, continuously, and separately from their weights, opening up possibilities such as pluggable domain-specific knowledge or per-customer memory. We look forward to experimenting with these in future work.

Code for reproduction available upon request.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →