Website  ·  Blog  ·  Slack  ·  Apache-2.0

macOS / Metal  ·  Linux / CUDA  ·  Windows (in progress)

Run models larger than your RAM or VRAM. InferSSD builds on llama.cpp to run large mixture-of-experts (MoE) models from SSD, using standard GGUF files, with browser chat and an OpenAI-compatible API. It currently supports DeepSeek-V4.1-Flash-1017 and Muse-Spark-Open, with more models coming.

These are the tested GGUF files, community conversions of the named model releases. DeepSeek's Q8_0 keeps the official bit widths: routed experts stay native 4-bit MXFP4, and FP8 tensors become 8-bit Q8_0. See weight-format details.

4K prompt tier · 256 output tokens · decode measured over tokens 129–256. All rates are in token/s; higher is faster. The multiple next to each InferSSD rate is its speedup over llama.cpp on the same hardware. On the MacBook Pro, Colibri runs the official Safetensors on CPU; llama.cpp and InferSSD run the GGUF with Metal. See measurement details.

Hardware CPU (cores) GPU Memory SSD

MacBook Pro Apple M5 Max (18) 40-core 48 GB unified 2 TB internal

RTX 5060 Ti Intel Core 3 100U (6) RTX 5060 Ti 8 GB + 16 GB VRAM 2 TB PCIe 4 ×4

RTX 5090 Intel Core Ultra 5 230F (10) RTX 5090 32 GB + 32 GB VRAM 2 TB PCIe 5 ×4

DGX Spark NVIDIA GB10 (20) Blackwell 128 GB unified 4 TB PCIe 5 ×4

Unified memory is shared by the CPU and GPU. SSD capacities are nominal. See hardware details for SSD models, CPU core layouts, and configuration notes.

HardwarePrefill (token/s)Decode (token/s) llama.cppColibriInferSSDllama.cppColibriInferSSD

MacBook Pro5.9290.70828.766 (4.85×)0.8730.4372.874 (3.29×) RTX 5060 Ti6.9507.02075.100 (10.81×)0.2420.9002.550 (10.54×) RTX 50908.06039.210164.500 (20.41×)0.8671.8787.270 (8.39×) DGX Spark2.7300.826127.500 (46.70×)1.6080.3567.260 (4.51×)

Muse-Spark-Open

HardwarePrefill (token/s)Decode (token/s) llama.cppColibriInferSSDllama.cppColibriInferSSD

MacBook Pro3.140N/S14.503 (4.62×)0.153N/S1.097 (7.17×) RTX 5060 Ti3.410N/S52.100 (15.28×)0.059N/S0.219 (3.71×) RTX 50904.000N/S106.600 (26.65×)0.155N/S1.520 (9.81×) DGX Spark1.740N/S83.900 (48.22×)0.120N/S2.140 (17.83×)

N/S: Muse is not currently supported by Colibri.

Install on Apple Silicon macOS or on Linux with an NVIDIA GPU:

curl -fsSL https://github.com/wici-ai/inferssd/releases/latest/download/install.sh | sh

InferSSD currently installs from source, so the engine is compiled for your machine's CPU and GPU; prebuilt packages are not available yet. Building takes about 2 to 12 minutes, depending on the machine.

The installer works in a private environment and uses no sudo. It needs Git, Python 3.9 or later, and a C/C++ compiler. macOS also needs the Xcode Command Line Tools; Linux also needs the CUDA toolkit, liburing headers, and systemd with cgroup v2. Follow the printed PATH instruction if it shows one.

Chat in your terminal

inferssd cli -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

Or start the web UI and an OpenAI-compatible API at http://127.0.0.1:8080

inferssd serve -hf mrfakename/DeepSeek-V4.1-Flash-1017-GGUF

For Muse, use unsloth/Muse-Spark-Open-816B-A42B-GGUF. With serve, open http://127.0.0.1:8080 in your browser, or point any OpenAI-compatible client at http://127.0.0.1:8080/v1.

The first run downloads about 470 GiB of model files to ~/.local/share/inferssd/models and measures your machine. Use --model-dir /path/on/ssd to choose the drive, or -m /path/to/first-shard.gguf for files you already have. There is no memory setting to choose: InferSSD uses the memory available. Stop with Ctrl+C, or /exit in chat.

See the quick start for upgrades and uninstalling.

How it works

Benchmark methodology

Contributing

Contributions are welcome, including hardware results and improvements to expert I/O, caching, and batching. See the contributing guide.

Built on llama.cpp and ggml, with thanks to their contributors and the model-adaptation community.

InferSSD is licensed under Apache-2.0; bundled llama.cpp/ggml and other third-party components keep their original licenses and notices; see NOTICE. Model weights remain subject to their publishers' licenses.