
2x Tesla p100s, q6_k quant, Qwen 3.8 27B ~60tps
This is a fork of llama.cpp tuned for two Tesla P100s (sm_60). On Qwen3.8-27B, plain decode
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
This is a fork of llama.cpp tuned for two Tesla P100s (sm_60). On Qwen3.8-27B, plain decode runs at 1.86x upstream's speed (17.5 -> 32.6 t/s), and ~3x with the model's built-in MTP head (~54 t/s). Prefill runs at ~2x (250 -> 493 t/s at short context). The full 262k window fits with vision loaded, and a question on a full 260k context prefills at ~153 t/s. The math is at least as accurate as stock: KLD against an fp32 run 0.00119, where upstream's fp16 path reads 0.00152. It tracks upstream by merging. See p100-docs/ for results, changes and how to run it. Upstream's README follows.
ggml / ops / maintainer PRs / dev stats / lib llama API / llama-server REST API
A few options to get llama.cpp installed on your machine:
Download and run a model directly from Hugging Face
llama cli -hf ggml-org/Qwen3.5-0.8B-GGUF
Launch OpenAI-compatible API server
llama serve -hf ggml-org/Qwen3.5-0.8B-GGUF
VLM session with llama cli
Built-in web UI against llama serve
Description
The main goal of llama.cpp is to enable LLM (and VLM) inference with minimal setup and state-of-the-art performance on a wide range of hardware - locally and in the cloud.
The llama.cpp project is build on top of the ggml library.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。