How much intelligence can you run locally?
Claude Opus 5.5 leads the AA Index v4.3.2 at 57.6. The best model that fits one GPU scores about 34, and the best open-weight model, Xiaomi's MiMo-V2.6-Pro, scores 46.3.
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Claude Opus 5.5 leads the AA Index v4.3.2 at 57.6. The best model that fits one GPU scores about 34, and the best open-weight model, Xiaomi's MiMo-V2.6-Pro, scores 46.3.
Epoch AI puts the lag between the best open and closed models at 4 months. An earlier Epoch analysis found that models small enough for one consumer GPU trail by 6–12 months.
The RTX 5090 (1,792 GB/s) finished Artificial Analysis's local agent workload more than 3.5× faster than the 128 GB unified-memory machines (256–307 GB/s).
Qwen3.6-35B-A3B, a mixture-of-experts model with 3B active parameters, ran 2.5–3.3× faster than the dense Qwen3.8-27B on every system AA tested.
4-bit weight-only quantization cost reasoning models 1% or less. Below 4 bits the losses grow quickly, most of all for small models and hard tasks.
A new RTX 5090 (MSRP $1,999) costs about $5,000 in North America. The 128 GB Framework Desktop went from $1,999 to $3,449.
On one DGX Spark, gpt-oss-120b decodes at 11.7 tok/s in Ollama, 38.5 in tuned llama.cpp and about 50 in SGLang.
Artificial Analysis Intelligence Index v4.3.2 · Epoch AI
The AA Intelligence Index is the most useful single scale for this comparison, as long as all scores come from the same version. It moved from v4.1 in June to v4.3.2 in October, and scores from different versions are not comparable.
Pick a group to see where each model runs. Hover a row for details.
The open models that come closest to the frontier are too large for a desk. Kimi K3 has 2.8 trillion parameters with 104 billion active, and DeepSeek V4.1 Flash has 552 billion, which fits a 512 GB Mac Studio at best. Among models that do fit, newer beats bigger: Qwen3.8-27B outscores every 120B-class model that fits on a 128 GB machine by more than 20 points, except Qwen3.8-Flash-Next.1
Epoch AI estimates that the best open-weight models trail the best closed models by about 4 months (Edwards & Emberson, May 29, 2026). An earlier Epoch analysis put the lag for models that fit one consumer GPU at 6–12 months (Somala & Emberson, Aug 2025), and notes that small open models are more likely to be tuned for specific benchmarks, so their real-world lag may be longer.
Open models also cost much less through a hosted API. At the v4.1 launch in June, AA found DeepSeek V4 Pro cost $0.04 per Index task, against $1.78 for Claude Opus 4.8.24
Measured tokens per second · AA-AgentPerf-Local
Generating a token means reading every active weight from memory once. There is little arithmetic per byte, so the chip spends most of its time waiting on memory, and speed is capped at roughly memory bandwidth divided by the bytes read per token. On any model that fits in its 32 GB, an RTX 5090 at 1,792 GB/s is much faster than a 128 GB machine at 273 GB/s.
Each lane types the same answer at the speed measured on that machine. Pick a model and press Run.
With gpt-oss-120b selected, the RTX 5090 drops out because the model needs about 66 GB. The 128 GB machines run it at about 55–88 tok/s, since each token reads only about 5 GB of the 66. On a dense 70B model, where every token reads all 40 GB, the Framework Desktop manages about 5 tok/s.
Dense models in this brief measure at about 55–70% of the bandwidth ceiling. Speculative decoding (MTP, DFlash, EAGLE and similar) can go above it: a small draft model proposes several tokens and the large model checks them in one pass. AA's configurations ran Qwen3.8-27B 30–120% above the ceiling, and a pinned llama.cpp build took an RTX 3090 from 37.4 to 60.3 tok/s with MTP.530
Reading the prompt (prefill) is limited by compute instead, because its tokens are processed in parallel. It takes a large share of agent time: AA found prefill took 22–41% of end-to-end time on Qwen3.8-27B, even with 73–93% of prompt tokens served from cache.5 Long context also slows decode. An RTX PRO 6000 generates 210 tok/s on gpt-oss-120b at 4K context and 100 tok/s at 128K.43
Minutes to finish AA-AgentPerf-Local. Shorter is better.
Longest prefill stall AA observed, in seconds.
Weights need roughly parameters × bits ÷ 8 bytes. A 70B model is about 140 GB at 16-bit and 40–42 GB at Q4_K_M, which averages about 4.8 bits per weight.29 A mixture-of-experts model has to store every parameter, but each token reads only the active ones.
Pick a model and a precision. Each row shows how much of a machine's memory the weights take and the theoretical decode ceiling.
The KV cache comes on top of the weights and grows with context. Its per-token size is 2 (K and V) × layers × KV heads × head dimension × bytes. For Qwen3.8-27B that works out to about 64 KiB per token at f16, so 128K tokens need 8 GiB. One study of quantized reasoning models found a 4-bit KV cache roughly lossless and a 3-bit cache "severe".31
Qwen3.8-27B at UD-Q4_K_XL (16.35 GiB of weights). Drag the context length and change the cache precision.
llama.cpp · arXiv 2504.04823 · arXiv 2505.11574 · IJCAI 2025
Quality stays almost flat down to about 4.8 bits per weight, then falls off quickly. At a similar size, Q4_K_M also beats the older Q4_0, because K-quants use a more precise block-scaling scheme.36
7B-class models. Fewer bits to the right.
Perplexity can miss real damage. "The Illusion of Equivalency" (arXiv 2607.08734) finds that Q8_0 through Q4_K keep accuracy close to the base model yet still produce "noticeable behavioral divergence", and concludes that perplexity "is not a trusted metric".39
Small models and hard tasks lose the most. A comparison of methods at IJCAI 2025 found FP8 "the most reliable method for all model sizes" and AWQ consistently ahead of GPTQ at 70B. It also found quantization "considerably reduces performance in Coding and STEM tasks" on MT-Bench even when leaderboard averages looked stable.38
Drop reported by each study, in %. An arrow marks a lower bound.
Tool calling is especially sensitive. Unsloth's write-up, cited by Pinggy, warns that tool calling breaks down below UD-Q2_K_XL. On Qwen3.8-27B, 32-token agreement with the full model fell from about 25% at UD-Q2_K_XL to under 10% at UD-IQ2_S.22
Prices in USD, late September to mid-October 2026
Unified-memory machines give the most memory per dollar, and NVIDIA's GDDR cards give the most bandwidth per dollar. The 96 GB M5 Ultra has the best bandwidth per dollar of any big-memory machine. At late-2026 street prices the RTX 5090 is no longer cheap VRAM: TensorRigs puts the card alone at about $156 per GB, close to professional cards.72
Switch the vertical axis. Bubble area shows the other measure.
The 2026 DRAM and GDDR7 shortage, by product.
Every speed on this page is a published measurement with its source named. Sites that estimate speeds can be far off: LLMCheck lists 7 tok/s for gpt-oss-120b on the M5 Max, where MLX measured 88.4973
Speeds are published measurements; the source follows each one.
Two newer options sit near the DGX Spark. NVIDIA's RTX Spark N1X puts a GB10-class chip with 300 GB/s of bandwidth in a Windows laptop; a 128 GB prototype Surface Laptop Ultra ($5,900) finished AA's agent workload in 18.7 minutes.46 AMD's own 128 GB Ryzen AI Halo ($4,000) ran 1.4–1.7× slower than the DGX Spark on three of four models. That is far more than their roughly 7% bandwidth difference, and AA attributes it to the Spark's extra low-precision compute for prefill and its more mature CUDA stack.5 No verified current price was found for DGX Station.
llama.cpp, with Ollama and LM Studio on top of it, is built for one user on whatever hardware is at hand. It splits layers between GPU and CPU so oversized models still load, and it supports the widest range of quantization types.75 vLLM manages the KV cache in pages (PagedAttention) and admits new requests at every step, which is what serving many users at once needs.76
Total tokens per second at 1 and 32 concurrent requests.
The gap widens with more users. Red Hat measured vLLM producing about 44× more tokens per second than llama.cpp at 64 users on one H200, with llama.cpp's P99 time to first token past 3 minutes.76 With little spare memory the gap closes: on a 24 GB RTX 4090 the two tied at 32 requests (306 vs. 300 tok/s) because vLLM had no VRAM left for batching.79 Two RTX PRO 6000s on vLLM served gpt-oss-120b at 230.5 tok/s to one user and peaked at 664 tok/s across 6 requests.80
gpt-oss-120b decode speed, tokens per second.
Frontier models run on racks of accelerators with high-bandwidth memory. A GB200 or GB300 NVL72 rack links 72 GPUs with 1.8 TB/s of NVLink each (130 TB/s in total) and up to 576 TB/s of aggregate memory bandwidth.8990 Tensor, pipeline and expert parallelism spread one model across many chips.
Datacenter accelerators against desk machines, in TB/s.
Batching is the main reason datacenter serving costs less per token. A home machine spends its whole bandwidth budget on one user. A datacenter GPU reads the weights once per step and produces a token for every request in the batch. DeepSeek's published V3/R1 serving stats show each 8×H800 node averaging about 14.8k output tok/s during decoding while each user saw 20–22 tok/s.91
An RTX 5090 serving one user, and one H800 in DeepSeek's V3/R1 serving fleet. Each sweep is one read of the weights; each dot is a user receiving a token.
Several other techniques add to this. Running prefill and decode on separate GPU pools gave about 1.5× higher throughput for Llama 3.1 405B in NVIDIA's MLPerf results.92 Prefix caching and KV offload keep repeated context cheap (about 96% of tokens were cached input in AA's Vera Rubin test), and session-aware routing such as NVIDIA Dynamo sends returning users back to their cache. AA measured Vera Rubin NVL72 at 272 million tokens per second per megawatt on DeepSeek V4 Pro, against 239 million for GB300 NVL72, with the lead growing to about 5.7× at around 190 tok/s per user. NVIDIA submitted those results, and the MW figure counts GPU and HBM power only.86
For one person, buy enough memory for the model you want, then as much bandwidth as the budget allows. Expect roughly Claude Haiku-class results, about a year behind the frontier. The gap from desk to frontier (about 24 AA points) is twice the gap from open weights to frontier (about 11).
Anthropic, Qwen and DeepSeek figures are self-reported, use different harnesses and differ from independent runs. Qwen3.8-27B's SWE-bench Pro score of 61.7 uses Qwen's refined version of the benchmark and is not comparable to Anthropic's 89.9%.25 AA Index versions change weights and benchmarks, so only compare scores within v4.3.2. The same hardware can differ 4× by engine and settings, so cross-source speed comparisons are approximate.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。