返回资讯中心
外部精选
软件工程
#开发者实时流

Tesla K80 / Kepler: 3× faster

Fixes the MUL_MAT slowdown on Kepler GPUs (Compute Capability < 5.0, no DP4A) when running K-quant models (q2_K … q6_K).

Hacker News 最新larfus_3032026/11/10 · 7 天前3 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Fixes the MUL_MAT slowdown on Kepler GPUs (Compute Capability < 5.0, no DP4A) when running K-quant models (q2_K … q6_K).

On Kepler (e.g. Tesla K80, CC 3.7), non-contiguous K-quants were routed to the slow MMQ path — or offloaded to the CPU — because the non-contiguous dequantize converter ("nc-converter") was missing for K-quants. MMQ on Kepler is ~4× slower than cuBLAS.

Plus a routing fix: ggml_cuda_should_use_mmq now only routes to MMQ for non-contiguous src0 without an nc-converter; contiguous K-quants stay on cuBLAS.

Prefill: 42.8 → ~140 t/s (~3.3×) — back to the cuBLAS baseline.

non-contiguous ≈ cuBLAS (1.01–1.05×); GPU ~2× faster than CPU at every batch size.

Correctness: NMSE exactly 0 for all five K-quants (fp16 + bf16).

Regression: K80 (CUDA2) 19208/19208 · RTX 3090 (CUDA0) 19268/19268 — no DP4A regression.

Origin

Developed through careful measurement by one individual and two agents. At no point were any technical objections raised. However, the original upstream PR (#30338) was closed by the maintainer due to procedural concerns — further evidence-based adjustments, which served to improve the patch's functionality, were submitted after the initial approval of the original changes. This repository keeps the patch publicly available.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →