
Unsloth 发布 Inkling-2-Mini-GGUF 量化模型
来源标题:Inkling-2-Mini-GGUF
Unsloth 在 Hugging Face 上发布了 Inkling-2-Mini 的 GGUF 量化版本,采用 apache-2.0 许可,并已获得 423 个点赞。该模型为图文转文本的 MoE 架构,支持通过 llama.cpp 在本地部署和推理。
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
Like 423
Follow
Unsloth 40.7k
Image-Text-to-Text
GGUF
conversational
Mixture of Experts
unsloth
imatrix
License: apache-2.0
Model card Files Files and versions xet
Community 7
Deploy
Copy to bucket
Use this model Instructions to use unsloth/Inkling-2-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.
Notebooks
Google Colab
Kaggle
Local Apps Settings
llama.cpp How to use unsloth/Inkling-2-Mini-GGUF with llama.cpp:
curl -LsSf https://llama.app/install.sh | sh
Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run inference directly in the terminal:
llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Install from WinGet (Windows)
winget install llama.cpp
Start a local OpenAI-compatible server with a web UI:
llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run inference directly in the terminal:
llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Build from source code
git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app
Start a local OpenAI-compatible server with a web UI:
./build/bin/llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run inference directly in the terminal:
./build/bin/llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Use Docker
docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Llama new
LM Studio
Jan
vLLM How to use unsloth/Inkling-2-Mini-GGUF with vLLM:
Install vLLM from pip:
pip install vllm
Start the vLLM server:
vllm serve "unsloth/Inkling-2-Mini-GGUF"
Call the server using curl (OpenAI-compatible API):
curl -X POST "http://localhost:8000/v1/chat/completions"
-H "Content-Type: application/json"
--data '{
"model": "unsloth/Inkling-2-Mini-GGUF",
"messages": [
{
"role": "user",
"content": [
{
"type": "text",
"text": "Describe this image in one sentence."
},
{
"type": "image_url",
"image_url": {
"url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg"
}
}
]
}
]
}'
Use Docker
docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Ollama How to use unsloth/Inkling-2-Mini-GGUF with Ollama:
ollama run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Unsloth Desktop
Pi How to use unsloth/Inkling-2-Mini-GGUF with Pi:
Install llama.cpp:
brew install llama.cpp
Start a local OpenAI-compatible server:
llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Configure the model in Pi
Install Pi:
npm install -g @earendil-works/pi-coding-agent
Add to ~/.pi/agent/models.json:
{ "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M" } ] } } }
Run Pi
Start Pi in your project directory:
pi
Docker Model Runner How to use unsloth/Inkling-2-Mini-GGUF with Docker Model Runner:
docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Lemonade How to use unsloth/Inkling-2-Mini-GGUF with Lemonade:
Download Lemonade from https://lemonade-server.ai/
lemonade pull unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run and chat with the model
lemonade run user.Inkling-2-Mini-GGUF-UD-Q4_K_M
List all available models
lemonade list
Hermes Agent How to use unsloth/Inkling-2-Mini-GGUF with Hermes Agent:
Install llama.cpp:
brew install llama.cpp
Start a local OpenAI-compatible server:
llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Configure Hermes
Install Hermes:
curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup
Point Hermes at the local server:
hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Run Hermes
hermes
Atomic Chat
OpenClaw How to use unsloth/Inkling-2-Mini-GGUF with OpenClaw:
Install llama.cpp:
brew install llama.cpp
Start a local OpenAI-compatible server:
llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M
Configure OpenClaw
Install OpenClaw:
npm install -g openclaw@latest
Register the local server and set it as the default model:
openclaw onboard --non-interactive --mode local
--auth-choice custom-api-key
--custom-base-url http://127.0.0.1:9931/v1
--custom-model-id "unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M"
--custom-provider-id llama-cpp
--custom-compatibility openai
--custom-text-input
--accept-risk
--skip-health
Run OpenClaw
openclaw agent --local --agent main --message "Hello from Hugging Face"
Read our How to Run Inkling-2-Mini Guide
Quantization Analysis
Inkling 2 Mini 1. General Information
-
Getting Started Required hardware
-
Model Properties Model type
Architecture type
Parameters
Numerics support
Input modalities
Output modalities
-
Training
-
Evaluations
-
Safety
-
Bias, risks and limitations
-
Legal
Read our How to Run Inkling-2-Mini Guide
See Unsloth Dynamic 3.0 GGUFs for our quantization benchmarks.
Run the model via the latest version of Unsloth Desktop.
See an example of 4-bit Inkling Mini running in Unsloth in Blender + MCP:
Quantization Analysis
Quant Size (GB) Mean KLD vs UD-Q8_K_XL Same top token
UD-IQ1_S 7.76 0.0911 89.4%
UD-IQ1_M 8.02 0.0867 90.2%
UD-IQ2_XXS 8.49 0.0726 90.6%
UD-Q2_K_XL 9.14 0.0595 91.6%
UD-IQ3_XXS 11.66 0.0269 94.7%
UD-Q3_K_XL 13.63 0.0174 96.3%
UD-IQ4_XS 15.73 0.0109 97.7%
UD-Q4_K_M 16.61 0.0108 97.9%
UD-Q4_K_XL 17.63 0.0107 98.0%
UD-Q8_K_XL 22.87 reference
Inkling 2 Mini
Playground | Tinker Cookbook
- General Information
Inkling 2 Mini is a general-purpose multimodal model that accepts text and image inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and visual tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers.
Languages: English, general multilingual support
Model provider: Thinking Machines Lab, Inc.
- Getting Started
The weights are available for download through Hugging Face here.
Developers can also access the model using the Tinker fine-tuning platform, which provides API access.
Required hardware
The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.
Given its small size, it is possible to run this checkpoint on local AI compute for inference. We support:
- Model Properties
Model type
Multimodal autoregressive looped transformer
Architecture type
A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run three times with shared weights, giving 48 layer applications per token from 24 layers of weights. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.
Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.
Input modalities
Text: UTF-8 encoded text
Image: Any pixel-based image input. For optimal performance, each image dimension should be between 40px to 4096px.
Inkling 2 Mini supports a context window of up to 1M tokens.
- Training
Training data includes a broad variety of content types, including text, images, video. Training data for the model was drawn from publicly available sources, acquired from third-parties, generated internally, or synthetically generated or augmented. Publicly available data includes content from the public internet and publicly accessible repositories.
Data curation includes cleaning, processing, and modifying datasets. These processing steps, which vary by data type, may include deduplication and filtering to remove junk or other low-quality data, or to advance safety or other objectives.
- Evaluations
Inkling 2 MiniGemma 4 31BMuse GlimmerQwen3.8-27BNemotron 3.5 Super VL
Model Info Params (B) activated / total5/2631/3130/3027/2712/125 Third Party Index AA Index (v5.0)20.14.46.320.113.6 Vals Index (v2.2)46.5%–––29.8% Coding and Autoresearch FrontierCode v2.0 stirrup v0.2.115.1%0.0%1.5%14.9%6.1% DeepSWE v1.1 mini-swe-agent v2.4.648.2%0.3%4.4%42.2%24.7% Terminal Bench v4.0 mini-swe-agent v2.4.65.6%0.0%0.5%5.6%0.0% Terminal Bench v2.1 Inkling 2 Mini in pi v0.83.071.6%43.4%51.7%73.0%60.7% SWEBench Pro v2 (Hard) mini-swe-agent v2.4.677.9%14.2%44.3%73.8%44.1% SWEBench Multilingual Inkling 2 Mini with bash only73.7%43.0%66.9%70.0%67.7% SWEBench Verified bash only79.2%46.1%76.0%77.8%71.1% Vibe Code Bench v1.1 openhands (vals)77.9%––64.9%3.5% PMPP-Hard (AUC, 0–10h) pi v0.83.0455.82––262.50– Agentic Search, Tool Use, and Knowledge Work BrowseComp Inkling 2 Mini with compaction79.0%37.0%63.9%61.6%78.9% DeepSearchQA Inkling 2 Mini with compaction85.3%61.7%74.6%79.7%77.7% MCP Atlas Public82.7%54.2%75.5%77.3%67.7% AA-Briefcase v1.112633694781391999 GDPVal-AA v2.1131962779214271162 JobBench49.6%8.2%16.1%33.4%36.5% AutomationBench-AA v1.0.641.8%4.9%6.8%48.2%14.8% EnterpriseOps-Gym42.0%28.4%34.7%44.2%28.1% Vals Finance Index50.1%–––41.0% AA-AnalystAgent46.3%–20.0%–– Diligence Bench Best harness48.6%–40.9%46.4%45.0% Vision Interpretation and Creation GDP.pdf16.8%6.0%11.8%16.4%8.0% PerceptionBench57.5%40.7%42.8%52.4%36.8% MMMU Pro Standard 1074.9%73.4%74.3%76.3%72.3% OmniDocBench v1.692.8%82.5%90.4%91.5%77.9% ScreenSpot Pro original / with Python76.5/87.7%5.9%/–75.4%/–57.6/78.9%43.5%/– Chartography original / with Python22.1/72.5%12.0/11.8%17.4/25.1%12.2/52.1%9.7/29.3% ZeroBench (main) original / with Python17.0/34.0%9.0/12.0%9.0/17.0%12.0%/–11.0/12.0% MuirBench77.5%72.3%77.6%65.9%72.7% BVB68.4%47.0%50.5%70.7%52.1% Reasoning, Knowledge, and Epistemics HLE-Diamond15.2%7.2%9.1%15.3%8.8% CritPt v2 Full / Hard31.4/17.0%–/4.0%–/7.0%10.6/4.0%36.9/13.5% Apex 202534.4%5.2%4.2%18.8%22.9% ARC-AGI v1 / v284.5/49.3%––87.5/42.4%22.7/1.5% LongBench v266.9%52.9%33.1%52.3%62.1% GraphWalks BFS 256K69.7%––50.4%63.3% MRCR v2 8-needle 256K–512K46.4%––38.0%5.7% AA-Omniscience v1.1 score-8.6-48.5-32.2-6.4-2.9 AA-Omniscience v1.1 non-hallucination rate63.4%14.5%16.4%71.4%69.5% ComplexConstraints v1.1 openhands v1.45.033.4%31.6%35.6%30.1%30.1% Safety and Alignment FORTRESS (adversarial) Public89.5%67.8%90.3%78.8%79.6% FORTRESS (benign) Public96.4%97.6%89.1%97.8%92.7% StrongREJECT (none)97.8%98.1%99.0%99.0%99.4% Misbehavior (Internal) ↓9.8%27.6%46.4%57.7%52.8% Covert misbehavior (Internal) ↓8.3%24.3%45.6%33.2%43.4% Destructive actions (Internal) ↓10.2%75.8%68.8%35.2%65.6%
Best reported score in each row (lowest where lower is better); equal displayed scores tie Second-best reported score in each row; none when first place is tied
- Safety
We conducted safety evaluations ahead of release, spanning both everyday human-AI interaction and dangerous-capability testing. Because Inkling 2 Mini is multimodal, we paid attention to whether safety behavior held consistently across text and image inputs. We applied mitigations to reduce risks before release, including an extra phase of alignment training using on-policy distillation, similarly to Inkling 2 Small.
For everyday use, we evaluated sy
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。