
GSD-LQ: Zero-calibration sub-4-bit compression and 7B in 3.78 GB VRAM
Patent Pending: U.S. Provisional Application No. 64/195,346
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Patent Pending: U.S. Provisional Application No. 64/195,346
Inventor: Cara Elizabeth Ducharme
Commercial Licensing & Inquiries: CaraElizabeth309@proton.me
GSD-LQ is a model-agnostic, zero-calibration compression engine and GPU execution prototype.
hex2d (~4.26 bpw): High-fidelity mode that outperforms standard Block-64 INT4 Perplexity at virtually identical footprint, with a native OpenCL GPU kernel (gsd_lq_opencl_kernel.py & run_gpu_native_chat.py).
e8 (~3.82 bpw): Sub-4-bit parity mode matching standard INT4 Perplexity while reducing memory footprint by over 10%.
bcc3d (~3.71 bpw): Lightweight 3D integer/half-integer mode optimized for compact client storage.
Quick Start
- Verify Compression & Reconstruction
Sub-4-bit parity mode (~3.82 bits/weight)
python compress_and_verify.py --model Qwen/Qwen2.5-0.5B --mode e8 --verify-only
High-accuracy mode (~4.26 bits/weight)
python compress_and_verify.py --model Qwen/Qwen2.5-0.5B --mode hex2d --verify-only
Compress full checkpoint to disk
python compress_and_verify.py --model Qwen/Qwen2.5-0.5B --mode e8 --out qwen_e8_v2.npz
- Native OpenCL GPU Runner Prototype (Any OpenCL 1.2+ GPU: AMD, NVIDIA, Intel)
Single-layer fused kernel VRAM & latency benchmark
python gsd_lq_opencl_kernel.py
Example: Pack & run a 7B GGUF model (e.g. Mistral 7B) natively inside < 4 GB VRAM
python pack_mistral_v2_native.py <path_to_mistral_gguf> mistral_v2_native_hex2d.gsd2 python run_gpu_native_chat.py mistral_v2_native_hex2d.gsd2
Verified Benchmarks
- Official Full-Brain 24-Layer WikiText-2 Perplexity (Qwen2.5-0.5B, All Attention + MLP Layers)
Evaluated across 1,536 tokens from the official wikitext-2 test split with zero calibration datasets:
Zero Wrapper Dependencies: Executes directly via Windows OpenCL.dll (ctypes) without PyTorch, Ollama, or ROCm.
Binary & Load Time: Packed 3.79 GB binary (4.257 bpw across all 224 layers + lm_head); loads into VRAM in 2.08 seconds.
Total Live VRAM: 3,875.6 MB (3.78 GB) out of 6,128 MB, leaving > 2.2 GB free VRAM including 2,048-token KV cache.
Fused Layer Kernel Latency:
Attention Q (4096x4096): 9.44 MB vs 32.00 MB FP16 (70.5% VRAM saved), 0.219 ms latency (< 1e-6 rel. err).
Dense MLP Gate (14336x4096): 29.90 MB vs 112.00 MB FP16 (73.3% VRAM saved), 0.558 ms latency (94.3% of raw FP16 speed, < 1e-6 rel. err).
End-to-End Live Generation & Recall: Streamed up to 12.48 tokens/sec over 12 live turns; full 32-layer v2 evaluation scored 9/12 (75.0%) on deterministic STEM/logic prompts (solving MATH_02 algebra where baseline Mistral failed) with 100% retrieval on 3,134-token Needle-in-a-Haystack contexts.
Licensing & Ethical Terms
See LICENSE.md.
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。