返回资讯中心
外部精选
软件工程
#开发者实时流

Show HN: Vramkit – measuring LLM weight and KV-cache compression on a MacBook

Two complementary, measured, lossless-verified techniques to cut the two things that eat GPU

Hacker News 最新sardor_r12026/10/17 · 2026/10/179 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Two complementary, measured, lossless-verified techniques to cut the two things that eat GPU memory during LLM inference: the model weights and the KV-cache. Every number below is produced by a runnable benchmark with a bit-exact roundtrip assertion — no hand-waving.

Why it matters: at inference scale, VRAM is the binding cost. Shrinking weights and KV-cache directly cuts GPU count / lets bigger context fit — the exact lever hyperscalers spend heavily on.

  1. Weight codec — lossless, beats general compressors

A from-scratch 32-bit range coder with a 2-D adaptive context model (each 4-bit weight conditioned on its left + up neighbours) — structure byte-oriented compressors (zstd/xz) can't see.

On REAL LLM weights (Qwen2.5-0.5B, self_attn.o_proj, int4 — python3 bench/validate_real_weights.py <model.safetensors>):

Lossless (bit-exact verified), −33% vs raw 4-bit, and beats zstd-19 by 1.5% / xz-9e by 1.2% on real quantized LLM weights — sitting on the 2-D entropy floor.

Across every weight-tensor type of Qwen2.5-0.5B (q/k/v/o-proj, mlp gate/down), it averages 2.92 bits/weight, −27% vs raw 4-bit, lossless, beating zstd-19 on 4 of 5 — so it's a general property of quantized LLM weights, not one lucky tensor. (On synthetic low-rank data the edge over zstd is smaller since there's less structure; the real-weights number is the one that counts.)

Per-token int8/int4 quantization (handles KV's outlier-channel structure), optionally range-coded.

Headline: int8 KV = ~2× at >0.999 fidelity (essentially free). Naive int4 loses quality on KV's outlier channels; keeping just 4 of 128 channels in int8 (outlier-aware int4) recovers cos-sim 0.958 → 0.999 at nearly int4 memory — the real engineering win. A range-coder on the int4 symbols adds further lossless shrink (exploitable redundancy survives quantization).

vramkit/kvquant.py is a device-agnostic module — it runs on AMD/ROCm (where KV-cache is the long-context memory bottleneck), NVIDIA, Apple MPS, or CPU, since it's plain PyTorch:

from vramkit.kvquant import quantize_kv, dequantize_kv packed = quantize_kv(key, mode="outlier-int4") # store this instead of fp16 (≈3.6× smaller) key_hat = dequantize_kv(packed) # reconstruct in the attention path

python3 vramkit/kvquant.py → int8 1.97× @ cos-sim 0.99985, outlier-int4 3.66× @ 0.99872 (parity with the numpy benchmark). Modes: int8 | int4 | outlier-int4.

On ROCm, long-context inference is a known pain point (the default attention path degrades hard past ~64k context, and FP8-KV support is still maturing). KV-cache memory is the lever: shrinking it 2–3.6× means more concurrent requests or longer context per Instinct/Radeon GPU — directly closing the gap with NVIDIA on $/token. vramkit is pure PyTorch + numpy, so it drops straight into a ROCm stack.

✅ Range coder: pure-stdlib, bit-exact roundtrip verified on random + real quantized data.

✅ Weight codec: −22% vs raw-4bit, lossless, beats zstd/xz, runs in ~2s on a MacBook.

✅ KV-cache: int8 ~2× near-lossless; outlier-aware int4 3.66× at cos-sim 0.999 — all measured.

Built and benchmarked entirely on a MacBook (Apple Silicon) — no datacenter GPU needed to reproduce the numbers.

⏳ Next (Mac-runnable): validate KV quality as perplexity on a real small model via MLX; push outlier-aware int4 lower; a Metal/ROCm dequant kernel so the compressed form is used live.

Layout

vramkit/rangecoder.py — the entropy coder (encoder + decoder + self-test).

bench/bench_weights.py — weight-codec benchmark vs zstd/xz with entropy floors.

bench/bench_kvcache.py — KV-cache quant + codec benchmark with fidelity.

MIT. Built on a MacBook; GPU validation pending.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →