返回资讯中心
外部精选
软件工程
#开发者实时流

Unsloth 发布 Inkling-2-Mini-GGUF 量化模型

来源标题:Inkling-2-Mini-GGUF

Unsloth 在 Hugging Face 上发布了 Inkling-2-Mini 的 GGUF 量化版本,采用 apache-2.0 许可,并已获得 423 个点赞。该模型为图文转文本的 MoE 架构,支持通过 llama.cpp 在本地部署和推理。

Hacker News 最新doener2026/11/21 · 今天30 分钟阅读

以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。

Like 423

Follow

Unsloth 40.7k

Image-Text-to-Text

GGUF

conversational

Mixture of Experts

unsloth

imatrix

License: apache-2.0

Model card Files Files and versions xet

Community 7

Deploy

Copy to bucket

Use this model Instructions to use unsloth/Inkling-2-Mini-GGUF with libraries, inference providers, notebooks, and local apps. Follow these links to get started.

Notebooks

Google Colab

Kaggle

Local Apps Settings

llama.cpp How to use unsloth/Inkling-2-Mini-GGUF with llama.cpp:

curl -LsSf https://llama.app/install.sh | sh

Start a local OpenAI-compatible server with a web UI:

llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Run inference directly in the terminal:

llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Install from WinGet (Windows)

winget install llama.cpp

Start a local OpenAI-compatible server with a web UI:

llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Run inference directly in the terminal:

llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Build from source code

git clone https://github.com/ggml-org/llama.cpp.git cd llama.cpp cmake -B build cmake --build build -j --target llama-app

Start a local OpenAI-compatible server with a web UI:

./build/bin/llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Run inference directly in the terminal:

./build/bin/llama cli -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Use Docker

docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Llama new

LM Studio

Jan

vLLM How to use unsloth/Inkling-2-Mini-GGUF with vLLM:

Install vLLM from pip:

pip install vllm

Start the vLLM server:

vllm serve "unsloth/Inkling-2-Mini-GGUF"

Call the server using curl (OpenAI-compatible API):

curl -X POST "http://localhost:8000/v1/chat/completions"
-H "Content-Type: application/json"
--data '{ "model": "unsloth/Inkling-2-Mini-GGUF", "messages": [ { "role": "user", "content": [ { "type": "text", "text": "Describe this image in one sentence." }, { "type": "image_url", "image_url": { "url": "https://cdn.britannica.com/61/93061-050-99147DCE/Statue-of-Liberty-Island-New-York-Bay.jpg" } } ] } ] }'

Use Docker

docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Ollama How to use unsloth/Inkling-2-Mini-GGUF with Ollama:

ollama run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Unsloth Desktop

Pi How to use unsloth/Inkling-2-Mini-GGUF with Pi:

Install llama.cpp:

brew install llama.cpp

Start a local OpenAI-compatible server:

llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Configure the model in Pi

Install Pi:

npm install -g @earendil-works/pi-coding-agent

Add to ~/.pi/agent/models.json:

{ "providers": { "llama-cpp": { "baseUrl": "http://localhost:9931/v1", "api": "openai-completions", "apiKey": "none", "models": [ { "id": "unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M" } ] } } }

Run Pi

Start Pi in your project directory:

pi

Docker Model Runner How to use unsloth/Inkling-2-Mini-GGUF with Docker Model Runner:

docker model run hf.co/unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Lemonade How to use unsloth/Inkling-2-Mini-GGUF with Lemonade:

Download Lemonade from https://lemonade-server.ai/

lemonade pull unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Run and chat with the model

lemonade run user.Inkling-2-Mini-GGUF-UD-Q4_K_M

List all available models

lemonade list

Hermes Agent How to use unsloth/Inkling-2-Mini-GGUF with Hermes Agent:

Install llama.cpp:

brew install llama.cpp

Start a local OpenAI-compatible server:

llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Configure Hermes

Install Hermes:

curl -fsSL https://hermes-agent.nousresearch.com/install.sh | bash hermes setup

Point Hermes at the local server:

hermes config set model.provider custom hermes config set model.base_url http://127.0.0.1:9931/v1 hermes config set model.default unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Run Hermes

hermes

Atomic Chat

OpenClaw How to use unsloth/Inkling-2-Mini-GGUF with OpenClaw:

Install llama.cpp:

brew install llama.cpp

Start a local OpenAI-compatible server:

llama serve -hf unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M

Configure OpenClaw

Install OpenClaw:

npm install -g openclaw@latest

Register the local server and set it as the default model:

openclaw onboard --non-interactive --mode local
--auth-choice custom-api-key
--custom-base-url http://127.0.0.1:9931/v1
--custom-model-id "unsloth/Inkling-2-Mini-GGUF:UD-Q4_K_M"
--custom-provider-id llama-cpp
--custom-compatibility openai
--custom-text-input
--accept-risk
--skip-health

Run OpenClaw

openclaw agent --local --agent main --message "Hello from Hugging Face"

Read our How to Run Inkling-2-Mini Guide

Quantization Analysis

Inkling 2 Mini 1. General Information

  1. Getting Started Required hardware

  2. Model Properties Model type

Architecture type

Parameters

Numerics support

Input modalities

Output modalities

  1. Training

  2. Evaluations

  3. Safety

  4. Bias, risks and limitations

  5. Legal

Read our How to Run Inkling-2-Mini Guide

See Unsloth Dynamic 3.0 GGUFs for our quantization benchmarks.

Run the model via the latest version of Unsloth Desktop.

See an example of 4-bit Inkling Mini running in Unsloth in Blender + MCP:

Quantization Analysis

Quant Size (GB) Mean KLD vs UD-Q8_K_XL Same top token

UD-IQ1_S 7.76 0.0911 89.4%

UD-IQ1_M 8.02 0.0867 90.2%

UD-IQ2_XXS 8.49 0.0726 90.6%

UD-Q2_K_XL 9.14 0.0595 91.6%

UD-IQ3_XXS 11.66 0.0269 94.7%

UD-Q3_K_XL 13.63 0.0174 96.3%

UD-IQ4_XS 15.73 0.0109 97.7%

UD-Q4_K_M 16.61 0.0108 97.9%

UD-Q4_K_XL 17.63 0.0107 98.0%

UD-Q8_K_XL 22.87 reference

Inkling 2 Mini

Playground | Tinker Cookbook

  1. General Information

Inkling 2 Mini is a general-purpose multimodal model that accepts text and image inputs and generates text outputs. It is intended for use in English and other languages, and across multiple coding languages. The model is designed to be used by developers building AI-powered applications, including agentic and tool-use systems, coding assistants, chatbots, and retrieval-augmented generation systems, and is suitable for general-purpose conversational use, instruction-following, and other natural language and visual tasks. It is released with open weights to support research, fine-tuning and integration into third-party products by downstream developers.

Languages: English, general multilingual support

Model provider: Thinking Machines Lab, Inc.

  1. Getting Started

The weights are available for download through Hugging Face here.

Developers can also access the model using the Tinker fine-tuning platform, which provides API access.

Required hardware

The model is distributed as a single checkpoint of about 20GB, in mixed NVFP4 and MXFP8 precisions.

Given its small size, it is possible to run this checkpoint on local AI compute for inference. We support:

  1. Model Properties

Model type

Multimodal autoregressive looped transformer

Architecture type

A 24-layer decoder-only transformer with sparse Mixture-of-Experts (MoE) feed-forward layers. The middle 12 layers run three times with shared weights, giving 48 layer applications per token from 24 layers of weights. In each layer, each token is routed to 8 of 128 experts. Attention is a hybrid of local and global layers. The model is natively multimodal – images are encoded via a vision transformer – with image and text inputs projected into a shared hidden space and processed jointly by the decoder.

Note: each extra pass through the looped layers adds about 1.0B effective active parameters. Since the model runs with three passes, there are 5.4B effective active parameters per token.

Input modalities

Text: UTF-8 encoded text

Image: Any pixel-based image input. For optimal performance, each image dimension should be between 40px to 4096px.

Inkling 2 Mini supports a context window of up to 1M tokens.

  1. Training

Training data includes a broad variety of content types, including text, images, video. Training data for the model was drawn from publicly available sources, acquired from third-parties, generated internally, or synthetically generated or augmented. Publicly available data includes content from the public internet and publicly accessible repositories.

Data curation includes cleaning, processing, and modifying datasets. These processing steps, which vary by data type, may include deduplication and filtering to remove junk or other low-quality data, or to advance safety or other objectives.

  1. Evaluations

Inkling 2 MiniGemma 4 31BMuse GlimmerQwen3.8-27BNemotron 3.5 Super VL

Model Info Params (B) activated / total5/2631/3130/3027/2712/125 Third Party Index AA Index (v5.0)20.14.46.320.113.6 Vals Index (v2.2)46.5%–––29.8% Coding and Autoresearch FrontierCode v2.0 stirrup v0.2.115.1%0.0%1.5%14.9%6.1% DeepSWE v1.1 mini-swe-agent v2.4.648.2%0.3%4.4%42.2%24.7% Terminal Bench v4.0 mini-swe-agent v2.4.65.6%0.0%0.5%5.6%0.0% Terminal Bench v2.1 Inkling 2 Mini in pi v0.83.071.6%43.4%51.7%73.0%60.7% SWEBench Pro v2 (Hard) mini-swe-agent v2.4.677.9%14.2%44.3%73.8%44.1% SWEBench Multilingual Inkling 2 Mini with bash only73.7%43.0%66.9%70.0%67.7% SWEBench Verified bash only79.2%46.1%76.0%77.8%71.1% Vibe Code Bench v1.1 openhands (vals)77.9%––64.9%3.5% PMPP-Hard (AUC, 0–10h) pi v0.83.0455.82––262.50– Agentic Search, Tool Use, and Knowledge Work BrowseComp Inkling 2 Mini with compaction79.0%37.0%63.9%61.6%78.9% DeepSearchQA Inkling 2 Mini with compaction85.3%61.7%74.6%79.7%77.7% MCP Atlas Public82.7%54.2%75.5%77.3%67.7% AA-Briefcase v1.112633694781391999 GDPVal-AA v2.1131962779214271162 JobBench49.6%8.2%16.1%33.4%36.5% AutomationBench-AA v1.0.641.8%4.9%6.8%48.2%14.8% EnterpriseOps-Gym42.0%28.4%34.7%44.2%28.1% Vals Finance Index50.1%–––41.0% AA-AnalystAgent46.3%–20.0%–– Diligence Bench Best harness48.6%–40.9%46.4%45.0% Vision Interpretation and Creation GDP.pdf16.8%6.0%11.8%16.4%8.0% PerceptionBench57.5%40.7%42.8%52.4%36.8% MMMU Pro Standard 1074.9%73.4%74.3%76.3%72.3% OmniDocBench v1.692.8%82.5%90.4%91.5%77.9% ScreenSpot Pro original / with Python76.5/87.7%5.9%/–75.4%/–57.6/78.9%43.5%/– Chartography original / with Python22.1/72.5%12.0/11.8%17.4/25.1%12.2/52.1%9.7/29.3% ZeroBench (main) original / with Python17.0/34.0%9.0/12.0%9.0/17.0%12.0%/–11.0/12.0% MuirBench77.5%72.3%77.6%65.9%72.7% BVB68.4%47.0%50.5%70.7%52.1% Reasoning, Knowledge, and Epistemics HLE-Diamond15.2%7.2%9.1%15.3%8.8% CritPt v2 Full / Hard31.4/17.0%–/4.0%–/7.0%10.6/4.0%36.9/13.5% Apex 202534.4%5.2%4.2%18.8%22.9% ARC-AGI v1 / v284.5/49.3%––87.5/42.4%22.7/1.5% LongBench v266.9%52.9%33.1%52.3%62.1% GraphWalks BFS 256K69.7%––50.4%63.3% MRCR v2 8-needle 256K–512K46.4%––38.0%5.7% AA-Omniscience v1.1 score-8.6-48.5-32.2-6.4-2.9 AA-Omniscience v1.1 non-hallucination rate63.4%14.5%16.4%71.4%69.5% ComplexConstraints v1.1 openhands v1.45.033.4%31.6%35.6%30.1%30.1% Safety and Alignment FORTRESS (adversarial) Public89.5%67.8%90.3%78.8%79.6% FORTRESS (benign) Public96.4%97.6%89.1%97.8%92.7% StrongREJECT (none)97.8%98.1%99.0%99.0%99.4% Misbehavior (Internal) ↓9.8%27.6%46.4%57.7%52.8% Covert misbehavior (Internal) ↓8.3%24.3%45.6%33.2%43.4% Destructive actions (Internal) ↓10.2%75.8%68.8%35.2%65.6%

Best reported score in each row (lowest where lower is better); equal displayed scores tie Second-best reported score in each row; none when first place is tied

  1. Safety

We conducted safety evaluations ahead of release, spanning both everyday human-AI interaction and dangerous-capability testing. Because Inkling 2 Mini is multimodal, we paid attention to whether safety behavior held consistently across text and image inputs. We applied mitigations to reduce risks before release, including an extra phase of alignment training using on-policy distillation, similarly to Inkling 2 Small.

For everyday use, we evaluated sy

正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。

打开原文
去社区讨论 →