返回资讯中心
外部精选
软件工程
#开发者实时流

2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps

This is a fork of llama.cpp tuned for two Tesla P100s (sm_60). On Qwen3.8-27B, plain decode

Hacker News 最新Kmic682026/10/18 · 2026/10/183 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

This is a fork of llama.cpp tuned for two Tesla P100s (sm_60). On Qwen3.8-27B, plain decode runs at 1.86x upstream's speed (17.5 -> 32.6 t/s), and ~3x with the model's built-in MTP head (~54 t/s). Prefill runs at ~2x (250 -> 493 t/s at short context). The full 262k window fits with vision loaded, and a question on a full 260k context prefills at ~153 t/s. The math is at least as accurate as stock: KLD against an fp32 run 0.00119, where upstream's fp16 path reads 0.00152. It tracks upstream by merging. See p100-docs/ for results, changes and how to run it. Upstream's README follows.

ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API

A few options to get llama.cpp installed on your machine:

Download and run a model directly from Hugging Face

llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF

Launch OpenAI-compatible API server

llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF

VLM session with llama cli

Built-in web UI against llama serve

Description

The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.

The llama.cpp project is build on top of the ggml library.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →