返回资讯中心
外部精选
软件工程
#开发者实时流

GSD-LQ: Zero-calibration sub-4-bit compression and 7B in 3.78 GB VRAM

Patent Pending: U.S. Provisional Application No. 64/195,346

Hacker News 最新Phantom309-CED2026/11/11 · 7 天前7 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Patent Pending: U.S. Provisional Application No. 64/195,346

Inventor: Cara Elizabeth Ducharme

Commercial Licensing & Inquiries: CaraElizabeth309@proton.me

GSD-LQ is a model-agnostic, zero-calibration compression engine and GPU execution prototype.

hex2d (~4.26 bpw): High-fidelity mode that outperforms standard Block-64 INT4 Perplexity at virtually identical footprint, with a native OpenCL GPU kernel (gsd_lq_opencl_kernel.py & run_gpu_native_chat.py).

e8 (~3.82 bpw): Sub-4-bit parity mode matching standard INT4 Perplexity while reducing memory footprint by over 10%.

bcc3d (~3.71 bpw): Lightweight 3D integer/half-integer mode optimized for compact client storage.

Quick Start

  1. Verify Compression & Reconstruction

Sub-4-bit parity mode (~3.82 bits/weight)

python compress_and_verify.py --model Qwen/Qwen2.5-0.5B --mode e8 --verify-only

High-accuracy mode (~4.26 bits/weight)

python compress_and_verify.py --model Qwen/Qwen2.5-0.5B --mode hex2d --verify-only

Compress full checkpoint to disk

python compress_and_verify.py --model Qwen/Qwen2.5-0.5B --mode e8 --out qwen_e8_v2.npz

  1. Native OpenCL GPU Runner Prototype (Any OpenCL 1.2+ GPU: AMD, NVIDIA, Intel)

Single-layer fused kernel VRAM & latency benchmark

python gsd_lq_opencl_kernel.py

Example: Pack & run a 7B GGUF model (e.g. Mistral 7B) natively inside < 4 GB VRAM

python pack_mistral_v2_native.py <path_to_mistral_gguf> mistral_v2_native_hex2d.gsd2 python run_gpu_native_chat.py mistral_v2_native_hex2d.gsd2

Verified Benchmarks

  1. Official Full-Brain 24-Layer WikiText-2 Perplexity (Qwen2.5-0.5B, All Attention + MLP Layers)

Evaluated across 1,536 tokens from the official wikitext-2 test split with zero calibration datasets:

Zero Wrapper Dependencies: Executes directly via Windows OpenCL.dll (ctypes) without PyTorch, Ollama, or ROCm.

Binary & Load Time: Packed 3.79 GB binary (4.257 bpw across all 224 layers + lm_head); loads into VRAM in 2.08 seconds.

Total Live VRAM: 3,875.6 MB (3.78 GB) out of 6,128 MB, leaving > 2.2 GB free VRAM including 2,048-token KV cache.

Fused Layer Kernel Latency:

Attention Q (4096x4096): 9.44 MB vs 32.00 MB FP16 (70.5% VRAM saved), 0.219 ms latency (< 1e-6 rel. err).

Dense MLP Gate (14336x4096): 29.90 MB vs 112.00 MB FP16 (73.3% VRAM saved), 0.558 ms latency (94.3% of raw FP16 speed, < 1e-6 rel. err).

End-to-End Live Generation & Recall: Streamed up to 12.48 tokens/sec over 12 live turns; full 32-layer v2 evaluation scored 9/12 (75.0%) on deterministic STEM/logic prompts (solving MATH_02 algebra where baseline Mistral failed) with 100% retrieval on 3,134-token Needle-in-a-Haystack contexts.

Licensing & Ethical Terms

See LICENSE.md.

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →