Five Ways Nvidia Co-Design Maximizes the Value of Every Byte in the AI Factory

AI-Generated Summary

NVFP4 reduces model footprint by approximately 3.5x versus FP16 and 1.8x versus FP8 while preserving accuracy, freeing HBM for KV cache, larger batches, and higher concurrency.

Wide expert parallelism in TensorRT-LLM spreads experts across eight or more GPUs, leaving more HBM per GPU for cache and raising arithmetic intensity through weight reuse.

NVIDIA Dynamo treats KV cache as a fleet-wide resource with KV-aware routing, disaggregated serving, and the KV Cache Runner that tiers blocks across GPU, CPU, and storage via NIXL.

The NVIDIA Vera CPU provides 88 Olympus cores with 1.8x faster single-threaded performance and up to 1.5 TB of LPDDR5X connected coherently to Rubin at 1.8 TB/s over NVLink-C2C, staging idle KV cache during tool calls.

NVIDIA CMX adds a pod-level G3.5 storage tier built on BlueField-4 STX, Spectrum-X, DOCA Memos, and Dynamo integration to share and reuse KV cache across nodes.

Together these five co-designed levers enable Vera Rubin NVL72 to deliver up to 30x the throughput per megawatt of GB300 NVL72 from the same 288 GB per GPU.

Next Steps

Read the Inside NVIDIA Rubin GPU Architecture: Powering the Era of Agentic AI blog post for deeper architectural details.

Explore the Optimizing Inference for Long Context and Large Batch Sizes with NVFP4 KV Cache blog post to learn KV cache optimization techniques.

Review the Scaling Large MoE Models with Wide Expert Parallelism on NVL72 Rack Scale Systems blog post for expert parallelism implementation guidance.

Powered by NVIDIA Nemotron. AI-generated content may summarize information incompletely. Verify important information. Learn more

In an AI factory, high bandwidth memory (HBM)is the fastest, most expensive, and most contested memory tier. Every byte spent on a redundant weight copy, an idle session’s KV cache, or a recomputed prefix is a byte that is not being used to produce tokens. The question is no longer how much memory a GPU has, but how much useful work each byte delivers.

The generational data in Figure 1 below shows why. From GB300 to NVIDIA Rubin, capacity held at 288 GB while bandwidth rose to 19.2 TB/s.

agentic throughput per megawatt grew far faster. On SemiAnalysis AgentX, which replays recorded coding-agent sessions with real tool-call gaps and growing context, NVIDIA Vera Rubin NVL72 then delivers up to 30x the throughput per megawatt of GB300 NVL72 at 160 tokens per second per user.

Figure 1. GPU throughput grew by 30x while HBM capacity remained approximately the same from GB300 NVL72 to Vera Rubin NVL72.

Capacity alone cannot explain a 30x gain delivered from the same 288 GB per GPU. The difference is how much useful work each byte delivers.

NVIDIA treats this as a co-design problem that spans GPU architecture, the scale-up interconnect, serving software, the CPU, and storage. This post examines five ways that co-design maximizes the value of each byte of memory, from the bits that encode a single weight to the shared storage that preserves and reuses context across agentic turns and nodes.

The most direct way to raise the value of a byte is to make each value smaller without losing what it encodes. NVFP4 does this with a 4-bit E2M1 element and two levels of scaling: Every 16-value micro-block shares an FP8 (E4M3) scale factor, and a per-tensor FP32 scale sits on top. The result is a format that is small enough to free significant HBM, yet granular enough to preserve accuracy where naive 4-bit formats fall short.

Figure 2. NVFP4 combines 4-bit values with block-level and tensor-level scaling.

Applied to weights, NVFP4 reduces the model footprint by approximately 3.5x relative to FP16 and 1.8x relative to FP8, freeing memory for what agentic inference needs most: more KV-cache capacity, larger batches, and higher concurrency. NVFP4 has become the default precision for open agentic benchmarks with strict accuracy requirements, including AA-AgentPerf, SemiAnalysis AgentX, and MLPerf Endpoints. Across these benchmarks and leading mixture-of-experts (MoE) architectures, it delivers up to 4x the throughput of an FP8 baseline at the same interactivity (Figure 3 below).

Figure 3. NVFP4 delivers 4x the throughput per megawatt of FP8 at the same interactivity for DeepSeek R1 on GB300.

The same principle extends to the KV cache itself. NVFP4 KV cache cuts the cache footprint by up to 50% compared with FP8, so approximately 2x more context stays resident on device. Fewer evictions mean higher cache-hit rates during prefill, while a smaller cache means less bandwidth consumed during decode.

Figure 4. NVFP4 KV cache archives over 99% of BF16 baseline scores across 10 DeepSeek-v4-Pro benchmarks

NVFP4 shrinks the bytes. The next question is where those bytes live, and how many copies of them the AI factory has to keep.

MoE models activate only a subset of parameters per token, but every expert’s weights must still live somewhere in HBM. When experts are packed onto a small number of GPUs, each GPU spends a large share of its memory on weights, and each decode step pays a heavy weight-loading cost for the MoE GroupGEMMs. The bytes are resident, but they are not working hard.

Wide expert parallelism (Wide-EP) in NVIDIA TensorRT-LLM changes that ratio by spreading experts across eight or more GPUs. With DeepSeek-R1 at EP64, each GPU holds four experts per layer, or 232 expert blocks across 58 MoE layers out of 14,848 in the full model. Fewer expert weights per GPU means:

Figure 5. GB300 NVL72 delivers 10x the token throughput per megawatt of H200 NVL8 at 100 tokens per second per user.

This only works because the scale-up domain makes remote bytes nearly as useful as local ones. Every MoE layer triggers all-to-all dispatch and combine traffic, and without a high-bandwidth fabric, those collectives would erase the gains. On GB200 NVL72, 130 TB/s of aggregate NVLink bandwidth absorbs that traffic, and large EP (EP32) delivers up to 1.8x more output tokens per second per GPU than small EP (EP8) at 100 tokens per second per user.

Vera Rubin NVL72 extends the same design point. Each Rubin GPU brings up to 288 GB of HBM4 and 3.0 TB/s of NVLink 6 bandwidth, for roughly 20.7 TB of HBM4 across the rack. Rubin also adds inline memory tensor acceleratory (TMA) descriptor updates, so one shared descriptor can serve every expert with a common layout, reducing the metadata overhead that grows with expert count.

Wide-EP decides where weight bytes live. The next lever decides whether KV cache bytes, once computed, are ever computed again.

The key-value (KV) cache stores intermediate attention calculations from previously processed tokens so the model can reuse them instead of recomputing them. This inference state is ephemeral and can be evicted and recomputed when needed, but KV cache is only valuable if it is reused. Agentic workloads are full of reuse opportunities: system prompts, tool schemas, shared documents, and the growing history of a multistep task. But if a request lands on a GPU that does not hold its prefix, or if that prefix was evicted moments earlier, the system pays for prefill again. The bytes existed; they were just in the wrong place.

NVIDIA Dynamo is the distributed serving layer that closes this gap. It treats KV cache as a fleet-wide resource rather than a per-GPU side effect, and it does so through three co-designed capabilities:

Figure 6. Dynamo combines KV-aware routing, disaggregated serving, and tiered KV cache reuse.

Dynamo’s KV-cache offloading depends on fast retrieval: Preserving a prefix outside HBM saves time when loading it is faster than repeating prefill. NIXL connects this offloading path to external storage systems. In separate validation tests, storage vendors VAST Data and WEKA demonstrated 35 GB/s to one H100 GPU using GPUDirect Storage and up to 270 GB/s of aggregate read throughput across eight H100 GPUs through a custom NIXL plugin, respectively. These tests showed that external storage could supply KV cache to the tested GPU systems without becoming the transfer bottleneck, supporting cache capacity beyond HBM. A separate VAST test demonstrated the response-time benefit: Using vLLM and LMCache on a DGX H100 system, loading stored KV cache instead of recomputing it reduced time to first token for Qwen3-32B with a 130K-token prompt.

Dynamo coordinates where cached bytes go and when they come back. The next two levers expand the capacity available for that context, starting with CPU memory.

Agentic inference does not run in one uninterrupted pass. A single session alternates between GPU model steps and CPU-side work: tool calls, code execution, retrieval, database queries, and sandboxed evaluation. During each of those gaps, any session KV cache retained in HBM occupies capacity without serving active token generation, while other requests compete for the same bytes.

The NVIDIA Vera CPU attacks both sides of this problem:

Together, these capabilities change the economics of an idle session. Instead of holding its KV cache in HBM or discarding it, Dynamo’s KV Block Manager can stage it in Vera memory for the duration of the tool call and bring it back over NVLink-C2C when the agent resumes. HBM stays dedicated to sessions that are actively generating tokens, and the paused session avoids paying for prefill twice.

Figure 8. Vera Rubin offloads inactive KV caches to host memory, freeing GPU HBM for active sessions.

Vera memory is the warm tier, sized for context that will return within a turn. For context that must survive across turns, agents, and nodes, the AI factory needs a tier that is larger still.

Keeping every reusable KV-cache byte in GPU or host memory limits capacity for active inference, but discarding it can force recomputation.. Multi-turn sessions, long-horizon tasks, and multi-agent workflows that share working memory all generate KV cache that will be needed again, often minutes later and often on a different node. Longer context windows and more concurrent sessions increase KV-cache capacity requirements beyond GPU HBM (G1 in Figure 9 below) and host memory (G2). The traditional hierarchy was not designed for this data. Local SSDs (G3) are bound to one node, while shared enterprise storage (G4) is built for durable data and carries replication and metadata overhead that ephemeral KV cache does not need.

NVIDIA CMX context memory storage closes this gap with a new G3.5 tier: Ethernet-attached flash, shared at the pod level, and optimized specifically for KV cache. It is built on the NVIDIA STX reference design and co-designed across four layers:

Figure 9. NVIDIA CMX adds a shared G3.5 tier for pod-wide KV-cache capacity and reuse

The pod-level design is what raises the value of each byte. A prefix computed once on one node can be reused across compatible model instances in the pod, so agents that share documents, tools, or working memory can avoid duplicate prefill. Pooling context storage across the pod reduces KV-cache duplication and stranded capacity across nodes, extending effective GPU memory for inference. By prestaging reusable context into GPU or host memory, CMX reduces inference stalls, while its KV-cache-optimized storage tier lowers the power overhead of serving that context. Together, these capabilities help CMX deliver up to 5x higher tokens per second and up to 5x better power efficiency compared with traditional storage for long-context inference.

Each lever removes a different kind of waste. NVFP4 removes bits a value does not need, Wide-EP removes redundant weight copies, Dynamo removes recomputation, Vera removes idle residency during tool calls, and CMX enables KV-cache sharing and reuse across nodes. Because these levers are designed as one memory system, not as independent parts, the gains stack: A 4-bit KV block saves space in HBM, in Vera memory, and in CMX alike.

The result is an AI factory that measures memory not by capacity alone, but by the