研究提出按内存与通信指标比较AI推理硬件,替代FLOP/s单一标准
来源标题:Beyond FLOP/s: comparing AI hardware for inference by memory and communication
一项分析提出,评估AI推理硬件不应只看FLOP/s,还应比较内存带宽与通信能力等指标。该方法旨在更准确反映推理场景下的实际性能差异,弥补单一看浮点算力指标的不足。
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Code
Cite
Machine Learning Hardware explorer
By Jason Li
Contents
The tradeoff between concurrency and output speed
How we calculate the IPI
Interpreting the IPI
Acknowledgements
The tradeoff between concurrency and output speedThe tradeoff between concurrency and output speed
How we calculate the IPI
Interpreting the IPI
Acknowledgements
Inference performance improvements can be much larger than changes in raw compute speed (FLOP/s) suggest. We estimate that the Nvidia GB300 provides about 25× Nvidia H100’s inference capacity per chip on the agentic inference workloads we model, with only 2.5× the 8-bit FLOP per second. Comparing FLOP/s alone would therefore underestimate GB300’s inference capacity relative to H100 by a factor of ten. This is because memory and inter-chip communication can matter more than arithmetic throughput for inference performance.
We introduce the Inference Performance Index (IPI) to capture these differences. The index compares how many requests each chip can serve at once across a range of required output speeds. IPI Version 1 focuses on agentic workloads and combines results across four representative models.
Scores are expressed in GB300 equivalents (GB300e), with GB300 scoring one by definition.1 A score of two means we estimate that a chip can serve twice as many users or agents at once as GB300, on average across the workloads and output-speed requirements in the index. We use GB300 as our reference because it is a widely used inference GPU with leading benchmark performance, such as in SemiAnalysis InferenceX.
System configuration also matters. Nvidia GB300 NVL72 has about 1.9× the modeled inference capacity per GPU of Nvidia B300 HGX. Both use Blackwell Ultra GPUs, but GB300 connects groups of 72 GPUs through high-bandwidth links, compared with groups of eight for B300 HGX.
The IPI provides a common unit, GB300 equivalents, to estimate a data center’s total inference capacity, or even the world’s. Combined with performance estimates for a particular model and output-speed requirement, this gives a rough estimate of how many users or agents that hardware could serve simultaneously.
This report explains how the index works and the key choices behind it. The accompanying technical methodology provides the equations, assumptions, and implementation details.
Language model inference has two phases. Prefill, also called the context phase, processes the input and stores intermediate representations in a cache, usually a key-value (KV) cache.2 Decode, or generation, produces the response through repeated forward passes. Each pass processes the latest token using the cached context, updates the cache, and generates one new token per request.3
To illustrate how the cache grows across requests, suppose a coding agent makes a request to the model with 200,000 tokens of cached history and 1,000 tokens of new input consisting of code-search results. Only the new input needs prefill. During decode, the model generates 1,000 output tokens, including reasoning and a tool call to read a different file. The software running the agent executes the tool call and includes the file’s contents as new input in the next request to the model. That request can reuse about 202,000 tokens of cached history: the original 200,000 tokens plus the new input and output from the previous request. Only the new input from the file needs prefill. The cache still occupies memory and is accessed during both prefill and decode.4
During decode, the system can process requests together in a batch. Model weights are stored in high-bandwidth memory (HBM) and brought into faster on-chip memory as needed for computation. Each forward pass processes the model layer by layer, reusing the weights brought on-chip across the requests in the batch. This amortizes the cost of reading weights from HBM. Each additional request still adds cache reads and other request-dependent work, so larger batches can increase total throughput while slowing token generation for each request, even when performance remains memory-bandwidth-bound.
There’s a tradeoff between concurrency and output speed. Concurrency is the number of requests being served simultaneously, expressed in requests per chip. Output speed, in tokens per second per request (tokens/s/req), measures how quickly each request generates tokens. Throughput, in tokens per second per chip (tokens/s/chip), measures the chip’s total output.
Increasing concurrency from two to eight requests/chip lowers output speed from about 210 to 80 tokens/s/req.
Throughput equals concurrency multiplied by output speed. Using these rounded values, throughput rises from 2 × 210 = 420 to 8 × 80 = 640 tokens/s/chip, an increase of roughly 50% despite slower generation for each request.
The IPI accounts for this tradeoff by comparing how many simultaneous requests each chip can support across a range of output speeds.
We use Nvidia’s AISimulate to estimate inference performance from the operations required by each model and the following hardware inputs:
The resulting performance estimates are analytical projections of optimal performance under the modeled hardware limits, often called speed of light (SOL).
The IPI includes both prefill and decode performance. We model them separately, and provision enough prefill chips to avoid bottlenecking decode. We then divide the number of simultaneous requests by the total number of prefill and decode chips to obtain capacity in requests/chip.
The IPI aims to track a representative basket of important inference workloads. Version 1 focuses on agentic inference, which already dominates token usage on OpenRouter, a platform that routes developer requests across AI model providers. In late September 2026, agentic API keys consumed roughly seven times as many tokens as human API keys.
Our agentic workload combines a long cached context of 262,144 tokens with 1,024 new tokens per prefill. To determine the required prefill resources, we use a trace-derived ratio of approximately 1.51 new input tokens per generated output token. The calculation also includes speculative decoding.6 The workload documentation gives the full assumptions.
For each chip and model, we take the highest concurrency per chip achievable at each required output speed across the configurations we evaluate. These points form the capacity curve.
A chip’s advantage over another can vary with the required output speed. We therefore compare capacity over 50–400 tokens/s/req.
We calculate the area under each curve using a logarithmic output-speed axis. This gives equal weight to each doubling of the required speed: 50–100, 100–200, and 200–400 tokens/s/req.
We divide the area under each chip’s capacity curve by GB300’s for the same model. For Kimi K3, this ratio is approximately 2.48 on the Vera Rubin (VR200) chip, though its advantage varies with the required output speed.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。