返回资讯中心
外部精选
软件工程
#开发者实时流

开发者发布纯 Go 工具包,为 Qwen3.8 27B 提供训练与推理服务

来源标题:Show HN: Qwen Reasoning Traces Trainer and Inference Server

一款纯 Go、无需 cgo 的工具包发布,面向 Q4_K_M GGUF 量化的 Qwen3.8 27B 模型。它实现了完整的 GGUF/量化栈、CPU + Vulkan 计算后端、Qwen3.8 前向传播(含 full attention 与 GatedDeltaNet),以及面向 SFT 训练(QLoRA)的数据集流水线。项目还包含 abliteration 功能,模型可从 Hugging Face 的 unsloth/Qwen3.8-27B-GGUF 仓库下载。

Hacker News 最新cookiengineer2026/11/12 · 5 天前24 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

A pure-Go, cgo-free toolkit for the Qwen3.8 27B model in its Q4_K_M GGUF quantization. It implements a complete GGUF/quant stack, a CPU + Vulkan compute backend, the Qwen3.8 forward pass (full attention + GatedDeltaNet), abliteration, and a dataset pipeline for SFT Training (QLoRA).

Go 1.27 (build and tests).

glslc and spirv-val to rebuild the committed SPIR-V shaders.

A Vulkan device for GPU acceleration and the GPU tests (the CLI selects it automatically when it has enough memory).

python3 with venv support only for the external verify-* tests.

Prerequisites

Download Qwen3.8-27B (Unsloth Dynamic Q4_K_M) from Hugging Face:

Repository: https://huggingface.co/unsloth/Qwen3.8-27B-GGUF

File: in the Files and versions list on the right side of the repo page, download Qwen3.8-27B-UD-Q4_K_M.gguf

Place it at models/Qwen3.8-27B-UD-Q4_K_M.gguf, or let the CLI fetch the pinned artifact (with resume + SHA-256 verification) for you:

./build/qwen-trainer download;

Export the training data from your Opencode environment and use the session history as the teacher model. This writes the manifest.json, sft/<topic>/sessions.jsonl, sft/<topic>/turns.jsonl, subagents/<topic>/..., and rl/metadata.jsonl.

opencode-reasoning-extractor ~/.local/share/opencode /my/own/reasoning-traces;

Point the Qwen Trainer at that output dataset directory (see Usage).

Usage

compile shaders, then build build/qwen-trainer

make;

the command is always the first argument; flags (including --model) follow it

run the Qwen Trainer

./build/qwen-trainer inspect --model models/Qwen3.8-27B-UD-Q4_K_M.gguf; ./build/qwen-trainer run --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --prompt "The capital of France is" --generate 8;

tokenize/validate the extractor output and report loss-mask statistics

--dataset is repeatable: pass it once per exported dataset directory

./build/qwen-trainer dataset --model models/Qwen3.8-27B-UD-Q4_K_M.gguf
--dataset /path/to/my/own/reasoning-traces/from/opencode;

--model is required by every model command (inspect, run, abliterate, search, evaluate, dataset, train); download, gpu-info, and status do not need it. There is no implicit default model path anymore.

The compute backend is chosen automatically: Vulkan is used when a device is present and it has enough device memory for the model, otherwise the CPU reference runs. gpu-info reports the detected device and memory.

See Abliteration for removing refusal directions and for the parameter search.

qwen-server exposes the same Vulkan/CPU inference pipeline through a minimal OpenAI-compatible API. Build it with make server:

make server; ./build/qwen-server --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --port 8080;

It serves GET /v1/models, POST /v1/chat/completions, and GET /health. Up to --max-concurrent requests are admitted at once. Complete forward passes are serialized by the Vulkan execution guard, so requests interleave between decode steps without corrupting shared command/descriptor state. GPU kernels from different requests do not execute in parallel.

curl -N http://127.0.0.1:8080/v1/chat/completions
-H 'Content-Type: application/json'
-d '{ "model": "Qwen3.8-27B-UD-Q4_K_M", "messages": [{"role": "user", "content": "What is 2+2?"}], "stream": true }';

Supported request features include stream (SSE), reasoning_content deltas, tools/tool_calls, temperature, top_p, top_k, seed, stop, max_tokens/max_completion_tokens, reasoning_effort, and chat_template_kwargs (enable_thinking, preserve_thinking). Thinking is enabled by default. Tool calls use the Qwen XML format and are returned as OpenAI tool_calls with JSON arguments.

On Vulkan the full inference forward runs device-resident, and K/V are stored in the quantized cache and appended with GPU shaders. The prefill chunk size is selected automatically from the context length and device memory.

Streaming usage is included when stream_options.include_usage is true.

MTP speculative decoding is enabled by default on both Vulkan and CPU.

Context values above the native GGUF context (256k) select YaRN automatically, with the factor derived from context / native_context.

See ARCHITECTURE.md for the underlying pipeline.

Remove the refusal/safety direction from the weights with Heretic-style directional ablation (projected / magnitude-preserving orthogonal ablation, MPOA), then optionally optimize the ablation hyperparameters.

1. Ablate with the default parameters and write a full Q4_K_M GGUF.

./build/qwen-trainer abliterate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --out models/abliterated.gguf;

2. (optional) search direction index + per-component weight kernels and write

the best trial. Trials persist to the study file and resume automatically.

./build/qwen-trainer search --model models/Qwen3.8-27B-UD-Q4_K_M.gguf
--trials 25
--sampler tpe
--study models/search.jsonl
--out models/search-best.gguf;

3. Score the result: refusal-keyword rate and KL divergence from the base.

./build/qwen-trainer evaluate --model models/abliterated.gguf
--prompts source/prompts/harmful_behaviors_test.jsonl
--max-tokens 32;

Prompt sets and limits

source/prompts/ bundles the same datasets Heretic uses, pre-converted to JSON Lines (one {"text": "..."} object per line). They are compiled into the binary with go:embed, so abliterate and search use them by default with no path needed; override with --good/--bad (--eval-good/--eval-bad for search). Files may be plain text (one prompt per line) or JSONL (--prompt-col selects the column, default text).

The full files are large, so the limits default to Heretic's: --limit 400 (residual prompts per set) and --eval-limit 100 (evaluation prompts per set). Pass --limit 0 / --eval-limit 0 to use an entire file. Every prompt costs one model forward, so raising the limits directly increases runtime.

These mirror mlabonne/harmless_alpaca and mlabonne/harmful_behaviors as published under heretic-org on Hugging Face.

Flag Command Meaning

--row-norm none|pre|full abliterate, search MPOA row normalization (default full)

--orthogonalize abliterate, search project the direction orthogonal to the good direction (default true)

--direction-index abliterate single global direction layer index (-1 = per layer, default)

--winsorize abliterate, search clamp residual components to a quantile in [0,1)

--sampler random|halton|tpe search optimizer (default random)

--lambda search KL weight in the objective refusal + lambda*KL (default 1.0)

--seed, --study search reproducible seed and resumable JSON Lines checkpoint

Note: the first-token KL is a weak signal for a thinking model (it emits <think> first), so small divergence scores are expected on short generations.

evaluate scores a model along three independent axes; at least one input is required.

Refusal keyword rate (and KL divergence from a base with --base).

./build/qwen-trainer evaluate --model models/abliterated.gguf
--prompts source/prompts/harmful_behaviors_test.jsonl --max-tokens 32
--base models/Qwen3.8-27B-UD-Q4_K_M.gguf;

Corpus perplexity over a text file, scored in windows of N tokens.

./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --perplexity corpus.txt --window 512;

Multiple-choice accuracy over JSONL {context, choices, answer} records.

./build/qwen-trainer evaluate --model models/Qwen3.8-27B-UD-Q4_K_M.gguf --tasks tasks.jsonl --tasks-verbose;

Perplexity uses non-overlapping windows (default 512 tokens, --window 0 for the whole input); the window bounds peak memory at vocab * window * 4 bytes, so it is the quality/memory knob. Because it computes full LM-head logits for every window it is considerably slower per token than run. Task accuracy is scored by the length-normalized log-likelihood of each choice given the context (--tasks-normalize=false for raw sums).

The default suite is pure Go and needs neither a GPU nor Python:

make test; # go test ./... make check; # fmt-check + vet + test (run this before calling anything done)

To run the suite on a remote GPU host, stream the working tree over SSH and run a make target there:

bash run-tests.sh debby; # remote make check bash run-tests.sh debby test; # remote make test

Tests that need hardware or external data skip automatically, so make check works on any machine:

Independent cross-checks live under source/e2e-tests/, one isolated main program per check. They shell out to reference implementations and are never run by make check:

make verify-ref; # NumPy forward pass vs the Go tiny model (~3e-8 match) make verify-data; # Jinja2 render of the model chat template + Python regex

evaluation of the qwen35 pre-tokenizer

make verify-quants; # our block decoders + encoders vs llama.cpp's ggml-quants.c

(needs a C compiler)

Each is also runnable directly, e.g. go -C source run ./e2e-tests/verify-ref.

The Python references use a project-local virtualenv at source/e2e-tests/.venv, created and populated on demand from each check's own requirements.txt (source/e2e-tests/verify-ref/requirements.txt, source/e2e-tests/verify-data/requirements.txt — numpy, jinja2, regex). The venv is gitignored.

The programs can also be driven directly with QWEN38_REF_PYTHON:

QWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-ref QWEN38_REF_PYTHON=/path/to/python go -C source run ./e2e-tests/verify-data go -C source run ./e2e-tests/verify-quants # CC=... to pick a compiler

The dataset reader is additionally checked against a fixture emitted by the opencode-reasoning-extractor's own writer (source/internal/dataset/testdata/extractor-writer/).

Released under the MIT License. See LICENSE.txt.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →