返回资讯中心
外部精选
软件工程
#开发者实时流

开发者发布5GB以下DeepSeek-Coder-V2-Lite量化版,6GB显卡可全量载入显存

来源标题:Show HN: DeepSeek-Coder-V2-Lite Under 5GB: 65 TPS 100% In-VRAM on 6GB GPU

开发者AmSach发布DeepSeek-Coder-V2-Lite-Instruct的MAPLE量化GGUF版本,体积压至5GB以下,在6GB显存GPU上实现100%载入显存并达到65 TPS。模型采用Apache-2.0许可,支持llama.cpp、Ollama等本地推理方式。

Hacker News 最新AmSach2026/11/12 · 5 天前29 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Like 0

Text Generation

GGUF

technical-code-corpus

English

code

Mixture of Experts

maple

quantized

deepseek

llama-cpp

ollama

imatrix

conversational

License: apache-2.0

Model card Files Files and versions xet

Community

Deploy

Copy to bucket

Use this model Instructions to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks

Google Colab

Kaggle

Local Apps Settings

llama.cpp How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with llama.cpp:

curl -LsSf https://llama.app/install.sh | sh

Start a local OpenAI-compatible server with a web UI:

llama serve -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Run inference directly in the terminal:

llama cli -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Install from WinGet (Windows)

winget install llama.cpp

Start a local OpenAI-compatible server with a web UI:

llama serve -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Run inference directly in the terminal:

llama cli -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app

Start a local OpenAI-compatible server with a web UI:

./build/bin/llama serve -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Run inference directly in the terminal:

./build/bin/llama cli -hf AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Use Docker

docker model run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Llama new

LM Studio

Jan

vLLM How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with vLLM:

Install vLLM from pip:

pip install vllm

Start the vLLM server:

vllm serve "AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF"

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:8000/v1/chat/completions"
-H "Content-Type: application/json"
--data '{ "model": "AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF", "messages": [ { "role": "user", "content": "What is the capital of France?" } ] }'

Use Docker

docker model run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Ollama How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with Ollama:

ollama run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Unsloth Desktop

Docker Model Runner How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with Docker Model Runner:

docker model run hf.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Lemonade How to use AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF with Lemonade:

Download Lemonade from https://lemonade-server.ai/

lemonade pull AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Run and chat with the model

lemonade run user.DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF-{{QUANT_TAG}}

List all available models

lemonade list

Atomic Chat

DeepSeek-Coder-V2-Lite-Instruct MAPLE (Sub-5GB GGUF) The Hardware Memory Barrier

Architectural Topology

Verified Benchmarks (RTX 4050 Laptop GPU) Empirical Workload Execution Results

Production Workload Generation Examples 1. Lock-Free SPSC Ring Buffer (C++20)

  1. Distributed Sliding-Window Token Bucket (Redis Lua)

Quickstart Guide Option 1: llama.cpp Execution (100% In-VRAM CUDA)

Option 2: Ollama Local Deployment

Citation & Contact

DeepSeek-Coder-V2-Lite-Instruct MAPLE (Sub-5GB GGUF)

Manifold-Activated Pruning of Latent Experts (MAPLE) Edition Full 16B MoE reasoning capacity compressed to 4.50 GB for 100% In-VRAM execution on 6GB laptop GPUs.

The Hardware Memory Barrier

Standard local deployments of DeepSeek-Coder-V2-Lite-Instruct (16B MoE) hit a severe hardware wall on consumer mobile workstations:

Architectural Topology

Metric Upstream Base Model MAPLE Sub-5GB Container Impact

Total Parameters 15.7B 13.5B 14.0% Footprint Reduction

Active Parameters / Token 2.4B 2.4B Unchanged Active Mass

Routed Experts / Layer 64 54 Top 84.4% Empirical Core Retained

Active Routed Experts 6 6 Standard Top-K Routing Preserved

Shared Invariant Experts 2 2 100% Unaltered Backbone

Intermediate Dimension 1408 1408 Full Expert Rank Preserved

Attention Architecture MLA (Rank 512) MLA (Rank 512) Compact KV Cache Preserved

Physical Container Size 8.57 GB (Q4) / 5.64 GB (IQ2) 4.50 GB Fits into 6GB VRAM

Generation Speed (RTX 4050) 4.5 - 10.0 TPS (Hybrid) 65.6 - 76.3 TPS 7.5x - 15.0x Speedup

Verified Benchmarks (RTX 4050 Laptop GPU)

Benchmarked on HP Victus Laptop (AMD Ryzen 7 7840HS, NVIDIA GeForce RTX 4050 6GB Laptop GPU, Windows 11, CUDA 12.4):

+-----------------------------------------------------------------------------------------+ | NVIDIA-SMI 592.27 Driver Version: 592.27 CUDA Version: 13.1 | | GPU Name Memory-Usage GPU-Util Offload Status | | 0 RTX 4050 Laptop GPU 4,850MiB / 6141MiB 100% 100% In-VRAM (27/27 Layers) | +-----------------------------------------------------------------------------------------+

Empirical Workload Execution Results

Benchmark Task Prompt TPS Gen TPS Time to Complete Status

Lock-Free SPSC Ring Buffer (C++20) 236.07 t/s 65.64 t/s 19.59s Bit-Exact Atomic Synchronization

Directed Hypergraph DP Solver (Python) 370.15 t/s 64.70 t/s 18.60s Complete Dijkstra Min-Heap Graph DP

Distributed Redis Token Bucket Rate Limiter 270.41 t/s 65.04 t/s 18.04s Production Lua Script + Fallbacks

Production Workload Generation Examples

  1. Lock-Free SPSC Ring Buffer (C++20)

#include <atomic> #include <cstddef> #include <new> #include <type_traits> #include <utility>

#if defined(__cpp_lib_hardware_interference_size) using std::hardware_destructive_interference_size; #else constexpr size_t hardware_destructive_interference_size = 64; #endif

template <typename T, size_t Capacity> class SPSCRingBuffer { static_assert((Capacity & (Capacity - 1)) == 0, "Capacity must be a power of two"); static_assert(std::is_nothrow_destructible_v<T>, "T must be nothrow destructible");

public: SPSCRingBuffer() : head_(0), tail_(0) {}

~SPSCRingBuffer() { T discarded; while (pop(discarded)); }

template <typename... Args> bool emplace(Args&&... args) { const size_t current_head = head_.load(std::memory_order_relaxed); if ((current_head - tail_cached_) >= Capacity) { tail_cached_ = tail_.load(std::memory_order_acquire); if ((current_head - tail_cached_) >= Capacity) { return false; } } new (&storage_[current_head & BufferMask].value) T(std::forward<Args>(args)...); head_.store(current_head + 1, std::memory_order_release); return true; }

bool pop(T& val) { const size_t current_tail = tail_.load(std::memory_order_relaxed); if (current_tail == head_cached_) { head_cached_ = head_.load(std::memory_order_acquire); if (current_tail == head_cached_) { return false; } } Node& node = storage_[current_tail & BufferMask]; val = std::move(node.value); node.value.~T(); tail_.store(current_tail + 1, std::memory_order_release); return true; }

private: static constexpr size_t BufferMask = Capacity - 1;

union Node { T value; Node() {} ~Node() {} };

alignas(hardware_destructive_interference_size) Node storage_[Capacity]; alignas(hardware_destructive_interference_size) std::atomic<size_t> head_; size_t tail_cached_ = 0; alignas(hardware_destructive_interference_size) std::atomic<size_t> tail_; size_t head_cached_ = 0; };

  1. Distributed Sliding-Window Token Bucket (Redis Lua)

import time import redis from typing import Tuple

RATE_LIMIT_LUA = """ local key = KEYS[1] local capacity = tonumber(ARGV[1]) local refill_rate = tonumber(ARGV[2]) local cost = tonumber(ARGV[3]) local now = tonumber(ARGV[4])

local state = redis.call('HMGET', key, 'tokens', 'last_updated') local tokens = tonumber(state[1]) local last_updated = tonumber(state[2])

if not tokens then tokens = capacity last_updated = now else local delta = math.max(0, now - last_updated) / 1000.0 tokens = math.min(capacity, tokens + delta * refill_rate) last_updated = now end

local allowed = 0 local retry_after_ms = 0

if tokens >= cost then allowed = 1 tokens = tokens - cost else local needed = cost - tokens retry_after_ms = math.ceil((needed / refill_rate) * 1000) end

redis.call('HMSET', key, 'tokens', tokens, 'last_updated', last_updated) redis.call('PEXPIRE', key, math.ceil((capacity / refill_rate) * 2000))

return {allowed, string.format("%.4f", tokens), retry_after_ms} """

class DistributedRateLimiter: def init(self, redis_client: redis.Redis, capacity: float, refill_rate: float): self.redis = redis_client self.capacity = capacity self.refill_rate = refill_rate self.script = self.redis.register_script(RATE_LIMIT_LUA)

def acquire(self, key: str, tokens: int = 1) -> Tuple[bool, float, int]: now_ms = int(time.time() * 1000) try: res = self.script(keys=[key], args=[self.capacity, self.refill_rate, tokens, now_ms]) allowed = bool(res[0]) remaining = float(res[1]) retry_after = int(res[2]) return allowed, remaining, retry_after except redis.RedisError:

High-availability circuit fallback

return True, 0.0, 0

Quickstart Guide

Option 1: llama.cpp Execution (100% In-VRAM CUDA)

Clone and run directly with llama-cli

llama-cli.exe -m DeepSeek-Coder-V2-Lite-MAPLE-Sub5GB.gguf
-ngl 99
-c 4096
-n 1024
--temp 0.2
-e
-p "User: Implement an SPSC ring buffer in C++20.\n\nAssistant:"

Option 2: Ollama Local Deployment

Create a Modelfile:

FROM ./DeepSeek-Coder-V2-Lite-MAPLE-Sub5GB.gguf

TEMPLATE """{{- if .System }}<|System|>{{ .System }}{{ end }} {{- range .Messages }} {{- if eq .Role "user" }}User: {{ .Content }}

{{ end }} {{- if eq .Role "assistant" }}Assistant: {{ .Content }}<|end of sentence|>{{ end }} {{- end }} {{- if .Prompt }}User: {{ .Prompt }}

Assistant: {{ end }}"""

PARAMETER stop "<|end of sentence|>" PARAMETER stop "<|User|>" PARAMETER stop "<|Assistant|>" PARAMETER num_gpu 99 PARAMETER temperature 0.2 PARAMETER top_p 0.95

Build and run:

ollama create deepseek-coder-maple -f Modelfile ollama run deepseek-coder-maple

Citation & Contact

If you integrate MAPLE or deploy this model container in your production pipeline:

@software{sachan2026maple_deepseek, author = {Sachan, Aman}, title = {MAPLE: Manifold-Activated Pruning of Latent Experts for DeepSeek-Coder-V2-Lite}, year = {2026}, url = {https://huggingface.co/AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF} }

Author: Aman Sachan

Contact: amansachan92905@gmail.com

Repository: https://github.com/AmSach/maple

Downloads last month -

","eos_token":"<|end▁of▁sentence|>","totalFileSize":4830185248},"ggufFilePaths":["DeepSeek-Coder-V2-Lite-MAPLE-Sub5GB.gguf"],"hasBlockedOids":false,"region":"us","isQuantized":true},"inferenceContextData":{"billableEntities":[],"entityName2Providers":{}},"canWrite":false}">

GGUF

Model size

13B params

Architecture

deepseek2

Chat template

Hardware compatibility

Log In to add your hardware

We're not able to determine the quantization variants.

Inference Providers NEW

Text Generation

This model isn't deployed by any Inference Provider. 🙋 Ask for provider support

Model tree for AmSach/DeepSeek-Coder-V2-Lite-Instruct-MAPLE-Sub5GB-GGUF

Base model

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →