
开发者发布5GB以下DeepSeek-Coder-V2-Lite量化版,6GB显卡可全量载入显存
来源标题:Show HN: DeepSeek-Coder-V2-Lite Under 5GB: 65 TPS 100% In-VRAM on 6GB GPU
开发者AmSach发布DeepSeek-Coder-V2-Lite-Instruct的MAPLE量化GGUF版本,体积压至5GB以下,在6GB显存GPU上实现100%载入显存并达到65 TPS。模型采用Apache-2.0许可,支持llama.cpp、Ollama等本地推理方式。
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Like 0
Text Generation
GGUF
technical-code-corpus
English
code
Mixture of Experts
maple
quantized
deepseek
llama-cpp
ollama
imatrix
conversational
License: apache-2.0
Model card Files Files and versions xet
Community
Deploy
Copy to bucket
Use this model Instructions to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
Notebooks
Google Colab
Kaggle
Local Apps Settings
llama.cpp How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with llama.cpp:
curl -LsSf https://llama.app/install.sh | sh
Start a local OpenAI-compatible server with a web UI:
llama serve -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Run inference directly in the terminal:
llama cli -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Install from WinGet (Windows)
winget install llama.cpp
Start a local OpenAI-compatible server with a web UI:
llama serve -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Run inference directly in the terminal:
llama cli -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app
Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Run inference directly in the terminal:
./build/bin/llama cli -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Use Docker
docker model run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Llama new
LM Studio
Jan
vLLM How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with vLLM:
Install vLLM from pip:
pip install vllm
Start the vLLM server:
vllm serve "AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF"
Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions"
-H "Content-Type: application/json"
--data '{
"model": "AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF",
"messages": [
{
"role": "user",
"content": "What is the capital of France?"
}
]
}'
Use Docker
docker model run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Ollama How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with Ollama:
ollama run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Unsloth Desktop
Docker Model Runner How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with Docker Model Runner:
docker model run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Lemonade How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with Lemonade:
Download Lemonade from https://lemonade-server.ai/
lemonade pull AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Run and chat with the model
lemonade run user.DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF-{{QUANT_TAG}}
List all available models
lemonade list
Atomic Chat
DeepSeek-Coder-V2-Lite-Instruct MAPLE (Sub-5GB GGUF) The Hardware Memory Barrier
Architectural Topology
Verified Benchmarks (RTX 4050 Laptop GPU) Empirical Workload Execution Results
Production Workload Generation Examples 1. Lock-Free SPSC Ring Buffer (C++20)
- Distributed Sliding-Window Token Bucket (Redis Lua)
Quickstart Guide Option 1: llama.cpp Execution (100% In-VRAM CUDA)
Option 2: Ollama Local Deployment
Citation & Contact
DeepSeek-Coder-V2-Lite-Instruct MAPLE (Sub-5GB GGUF)
Manifold-Activated Pruning of Latent Experts (MAPLE) Edition Full 16B MoE reasoning capacity compressed to 4.50 GB for 100% In-VRAM execution on 6GB laptop GPUs.
The Hardware Memory Barrier
Standard local deployments of DeepSeek-Coder-V2-Lite-Instruct (16B MoE) hit a severe hardware wall on consumer mobile workstations:
Architectural Topology
Metric Upstream Base Model MAPLE Sub-5GB Container Impact
Total Parameters 15.7B 13.5B 14.0% Footprint Reduction
Active Parameters / Token 2.4B 2.4B Unchanged Active Mass
Routed Experts / Layer 64 54 Top 84.4% Empirical Core Retained
Active Routed Experts 6 6 Standard Top-K Routing Preserved
Shared Invariant Experts 2 2 100% Unaltered Backbone
Intermediate Dimension 1408 1408 Full Expert Rank Preserved
Attention Architecture MLA (Rank 512) MLA (Rank 512) Compact KV Cache Preserved
Physical Container Size 8.57 GB (Q4) / 5.64 GB (IQ2) 4.50 GB Fits into 6GB VRAM
Generation Speed (RTX 4050) 4.5 - 10.0 TPS (Hybrid) 65.6 - 76.3 TPS 7.5x - 15.0x Speedup
Verified Benchmarks (RTX 4050 Laptop GPU)
Benchmarked on HP Victus Laptop (AMD Ryzen 7 7840HS, NVIDIA GeForce RTX 4050 6GB Laptop GPU, Windows 11, CUDA 12.4):
+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 592.27 Driver Version: 592.27 CUDA Version: 13.1 | | GPU Name Memory-Usage GPU-Util Offload Status | | 0 RTX 4050 Laptop GPU 4,850MiB / 6141MiB 100% 100% In-VRAM (27/27 Layers) | +-----------------------------------------------------------------------------------------+
Empirical Workload Execution Results
Benchmark Task Prompt TPS Gen TPS Time to Complete Status
Lock-Free SPSC Ring Buffer (C++20) 236.07 t/s 65.64 t/s 19.59s Bit-Exact Atomic Synchronization
Directed Hypergraph DP Solver (Python) 370.15 t/s 64.70 t/s 18.60s Complete Dijkstra Min-Heap Graph DP
Distributed Redis Token Bucket Rate Limiter 270.41 t/s 65.04 t/s 18.04s Production Lua Script + Fallbacks
Production Workload Generation Examples
- Lock-Free SPSC Ring Buffer (C++20)
#include <atomic> #include <cstddef> #include <new> #include <type_traits> #include <utility>
#if defined(__cpp_lib_hardware_interference_size) using std::hardware_destructive_interference_size; #else constexpr size_t hardware_destructive_interference_size = 64; #endif
template <typename T, size_t Capacity> class SPSCRingBuffer { static_assert((Capacity & (Capacity - 1)) == 0, "Capacity must be a power of two"); static_assert(std::is_nothrow_destructible_v<T>, "T must be nothrow destructible");
public: SPSCRingBuffer() : head_(0), tail_(0) {}
~SPSCRingBuffer() { T discarded; while (pop(discarded)); }
template <typename... Args> bool emplace(Args&&... args) { const size_t current_head = head_.load(std::memory_order_relaxed); if ((current_head - tail_cached_) >= Capacity) { tail_cached_ = tail_.load(std::memory_order_acquire); if ((current_head - tail_cached_) >= Capacity) { return false; } } new (&storage_[current_head & BufferMask].value) T(std::forward<Args>(args)...); head_.store(current_head + 1, std::memory_order_release); return true; }
bool pop(T& val) { const size_t current_tail = tail_.load(std::memory_order_relaxed); if (current_tail == head_cached_) { head_cached_ = head_.load(std::memory_order_acquire); if (current_tail == head_cached_) { return false; } } Node& node = storage_[current_tail & BufferMask]; val = std::move(node.value); node.value.~T(); tail_.store(current_tail + 1, std::memory_order_release); return true; }
private: static constexpr size_t BufferMask = Capacity - 1;
union Node { T value; Node() {} ~Node() {} };
alignas(hardware_destructive_interference_size) Node storage_[Capacity]; alignas(hardware_destructive_interference_size) std::atomic<size_t> head_; size_t tail_cached_ = 0; alignas(hardware_destructive_interference_size) std::atomic<size_t> tail_; size_t head_cached_ = 0; };
- Distributed Sliding-Window Token Bucket (Redis Lua)
import time import redis from typing import Tuple
RATE_LIMIT_LUA = """ local key = KEYS[1] local capacity = tonumber(ARGV[1]) local refill_rate = tonumber(ARGV[2]) local cost = tonumber(ARGV[3]) local now = tonumber(ARGV[4])
local state = redis.call('HMGET', key, 'tokens', 'last_updated') local tokens = tonumber(state[1]) local last_updated = tonumber(state[2])
if not tokens then tokens = capacity last_updated = now else local delta = math.max(0, now - last_updated) / 1000.0 tokens = math.min(capacity, tokens + delta * refill_rate) last_updated = now end
local allowed = 0 local retry_after_ms = 0
if tokens >= cost then allowed = 1 tokens = tokens - cost else local needed = cost - tokens retry_after_ms = math.ceil((needed / refill_rate) * 1000) end
redis.call('HMSET', key, 'tokens', tokens, 'last_updated', last_updated) redis.call('PEXPIRE', key, math.ceil((capacity / refill_rate) * 2000))
return {allowed, string.format("%.4f", tokens), retry_after_ms} """
class DistributedRateLimiter: def init(self, redis_client: redis.Redis, capacity: float, refill_rate: float): self.redis = redis_client self.capacity = capacity self.refill_rate = refill_rate self.script = self.redis.register_script(RATE_LIMIT_LUA)
def acquire(self, key: str, tokens: int = 1) -> Tuple[bool, float, int]: now_ms = int(time.time() * 1000) try: res = self.script(keys=[key], args=[self.capacity, self.refill_rate, tokens, now_ms]) allowed = bool(res[0]) remaining = float(res[1]) retry_after = int(res[2]) return allowed, remaining, retry_after except redis.RedisError:
High-availability circuit fallback
return True, 0.0, 0
Quickstart Guide
Option 1: llama.cpp Execution (100% In-VRAM CUDA)
Clone and run directly with llama-cli
llama-cli.exe -m DeepSeek-Coder-V2-Lite-MAPLE-Sub5GB.gguf
-ngl 99
-c 4096
-n 1024
--temp 0.2
-e
-p "User: Implement an SPSC ring buffer in C++20.\n\nAssistant:"
Option 2: Ollama Local Deployment
Create a Modelfile:
FROM ./DeepSeek-Coder-V2-Lite-MAPLE-Sub5GB.gguf
TEMPLATE """{{- if .System }}<|System|>{{ .System }}{{ end }} {{- range .Messages }} {{- if eq .Role "user" }}User: {{ .Content }}
{{ end }} {{- if eq .Role "assistant" }}Assistant: {{ .Content }}<|end of sentence|>{{ end }} {{- end }} {{- if .Prompt }}User: {{ .Prompt }}
Assistant: {{ end }}"""
PARAMETER stop "<|end of sentence|>" PARAMETER stop "<|User|>" PARAMETER stop "<|Assistant|>" PARAMETER num_gpu 99 PARAMETER temperature 0.2 PARAMETER top_p 0.95
Build and run:
ollama create deepseek-coder-maple -f Modelfile ollama run deepseek-coder-maple
Citation & Contact
If you integrate MAPLE or deploy this model container in your production pipeline:
@software{sachan2026maple_deepseek, author = {Sachan, Aman}, title = {MAPLE: Manifold-Activated Pruning of Latent Experts for DeepSeek-Coder-V2-Lite}, year = {2026}, url = {https://huggingface.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF} }
Author: Aman Sachan
Contact: amansachan92905@gmail.com
Repository: https://github.com/AmSach/maple
Downloads last month -
","eos_token":"<|end▁of▁sentence|>","totalFileSize":4830185248},"ggufFilePaths":["DeepSeek-Coder-V2-Lite-MAPLE-Sub5GB.gguf"],"hasBlockedOids":false,"region":"us","isQuantized":true},"inferenceContextData":{"billableEntities":[],"entityName2Providers":{}},"canWrite":false}">
GGUF
Model size
13B params
Architecture
deepseek2
Chat template
Hardware compatibility
Log In to add your hardware
We're not able to determine the quantization variants.
Inference Providers NEW
Text Generation
This model isn't deployed by any Inference Provider. 🙋 Ask for provider support
Model tree for AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF
Base model
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。