
Tesla K80 / Kepler: 3× faster
Fixes the MUL_MAT slowdown on Kepler GPUs (Compute Capability < 5.0, no DP4A) when running K-quant models (q2_K … q6_K).
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Fixes the MUL_MAT slowdown on Kepler GPUs (Compute Capability < 5.0, no DP4A) when running K-quant models (q2_K … q6_K).
On Kepler (e.g. Tesla K80, CC 3.7), non-contiguous K-quants were routed to the slow MMQ path — or offloaded to the CPU — because the non-contiguous dequantize converter ("nc-converter") was missing for K-quants. MMQ on Kepler is ~4× slower than cuBLAS.
Plus a routing fix: ggml_cuda_should_use_mmq now only routes to MMQ for non-contiguous src0 without an nc-converter; contiguous K-quants stay on cuBLAS.
Prefill: 42.8 → ~140 t/s (~3.3×) — back to the cuBLAS baseline.
non-contiguous ≈ cuBLAS (1.01–1.05×); GPU ~2× faster than CPU at every batch size.
Correctness: NMSE exactly 0 for all five K-quants (fp16 + bf16).
Regression: K80 (CUDA2) 19208/19208 · RTX 3090 (CUDA0) 19268/19268 — no DP4A regression.
Origin
Developed through careful measurement by one individual and two agents. At no point were any technical objections raised. However, the original upstream PR (#30338) was closed by the maintainer due to procedural concerns — further evidence-based adjustments, which served to improve the patch's functionality, were submitted after the initial approval of the original changes. This repository keeps the patch publicly available.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。