Distributed KV-cache management, wire quantization, and relative RoPE block reuse for multi-node LLM inference.

In multi-turn agent loops, codebase analysis, and RAG workloads, most prompt tokens are shared across requests (system prompts, tool definitions, repository files, and retrieved documents).

Current serving stacks handle this in one of three ways, each with a specific bottleneck:

HyperCache is designed as an external cache layer ("memcached for KV tensors") that sits underneath existing inference workers.

[ Incoming LLM Request ] | +-------------+-------------+ | | v v [ Prompt Tokenizer ] [ Global Radix / Hash DAG ] | (HMAC-BLAKE2b + Modular) | | | Cache Lookup | | v v +---------------------+ +---------------------+ | Compute New Suffix | | Fetch Shared KV | | + 15% CacheBlend | | Prefix Tensors | +----------+----------+ +----------+----------+ | | | +----------+----------+ | | RDMA / TCP-FP8 | | | (Layer-Wise Overlap)| | +----------+----------+ v v +------------------------------------+ | Decode Generation Loop | | (vLLM / SGLang / TRT-LLM Hook) | +------------------------------------+

Core Design

1. Dual-Mode Block Hashing (hypercache/core/hasher.py)

Standard Transformer self-attention is strictly causal: the K and V tensors at token $i$ depend on every preceding token $0 \dots i-1$. Fuzzy hashing (such as LSH) cannot be used for exact prefix matching without corrupting attention states. HyperCache computes two deterministic 128-bit BLAKE2b digests per token block (default 16 or 64 tokens):

All hashes are keyed by a per-tenant secret derived via HMAC-SHA256 to prevent cross-tenant timing side-channel attacks (where a user infers whether another tenant has cached a proprietary file by measuring TTFT). An explicit "public" namespace is available for global system prompts and shared tool schemas.

When a cached block is reused at a new token offset $m_{\text{new}}$ instead of its original offset $m_{\text{old}}$, its keys have the wrong Rotary Position Embedding (RoPE) phase angles.

Because 2D RoPE rotations belong to $SO(2)$, they satisfy the group identity: $$R(m_{\text{new}} \theta_i) = R((m_{\text{new}} - m_{\text{old}}) \theta_i) , R(m_{\text{old}} \theta_i)$$

This means HyperCache does not need invasive forward-pass hooks to intercept pre-RoPE keys. It takes standard post-RoPE keys directly from the engine's PagedAttention buffer at $m_{\text{old}}$ and rotates each complex coordinate pair by the relative delta $\Delta m = m_{\text{new}} - m_{\text{old}}$ on fetch (max_err ~ 4.7e-7 in float32).

To account for cross-chunk causal attention drift in layers $1 \dots L$ when chunks are reordered, HyperCache pairs relative RoPE rotation with selective boundary recomputation (hypercache/core/attention_engine.py): Layer 0 K/V depends only on token embeddings (making rotated Layer 0 KV exact), and the top ~15% of tokens with the largest Layer-0 hidden-state drift are selectively recomputed through deeper layers (following the CacheBlend approach).

Because network and PCIe links are an order of magnitude slower than HBM3e, blocks are compressed before leaving L1:

Waiting for all 80 layers of a 64k-token KV cache (~10 GB in FP8) to transfer before starting prefill leaves the GPU idle. By pipelining transfers layer-by-layer—fetching Layer $l+1$ into a staging buffer while the GPU computes suffix tokens for Layer $l$—most of the wire transfer time is hidden behind active suffix compute.

Modeled break-even comparison for Llama-3-70B across 80GB H100 nodes (400Gbps RoCEv2, ~45.5 GB/s effective):

To keep the repository easy to inspect and test on any machine without requiring an InfiniBand cluster:

pip install -e .

1. Run the verification and break-even benchmark

Executes the local 4-layer GQA attention verification on reordered chunks, physical L1/L2/L3 .hckv disk round-trips, live loopback TCP socket transfers, and prints the cluster break-even table:

python -m hypercache benchmark --gpu h100

Example output:

========================================================================================== PART 1: LIVE MEASURED NUMERICAL & SYSTEMS VERIFICATION (EXECUTED LOCALLY)

A. Multi-Layer Causal GQA Attention + RoPE on Reordered RAG Chunks [Sys, DocB, DocA, Q]

• Naive KV Stitching (No RoPE / No Recompute) : Cosine Sim = 0.7423 (Rel L2 Err: 0.715 -> BROKEN)
• Relative RoPE Phase Rotation Only (0% Rec.) : Cosine Sim = 0.9701
• HYPERCACHE (RoPE Shift + 15% CacheBlend) : Cosine Sim = 0.9783 (Rel L2 Err: 0.208 | Saved 85.3% Prefix FLOPs)

---

B. Physical Tensor Quantization, L3 Disk Spillover & Live TCP Socket Transport

• Relative RoPE Shift Error (Pos 1024 -> 8192): 4.77e-07 (Exact SO(2) identity)
• FP8 (E4M3) Wire Quantization : 1.97x compression | Cosine Sim: 0.999979
• INT4 (Grouped) L3 Disk (.hckv + CRC32) : 3.88x compression | Cosine Sim: 0.993258 (67,653 B on disk)
• Live TCP Socket Fetch + On-The-Fly RoPE : 133,188 B frame in 1.14 ms | Cosine Sim vs GT: 0.999979 ==========================================================================================

2. Run the test suite

Runs all 9 unit, adversarial, CRC32 tamper detection, multi-threaded TCP concurrency, and GQA attention tests:

python tests/test_hypercache.py

3. Audit prompt logs or simulated multi-node workloads

Evaluates cache hit rates and TTFT across Siloed APC, Host-RAM+TCP offload, and HyperCache on a JSONL log file (or a built-in multi-agent / RAG workload):

python -m hypercache audit --workload multi-agent-swe --nodes 4 --gpu h100

Or replay your own prompt logs:

python -m hypercache audit --log-file requests.jsonl --nodes 4 --gpu h100

4. Terminal telemetry view

python -m hypercache top --duration 1.5

License

MIT