DeepInfer is the high-performance inference framework that powers DeepSeek's large-scale production services. This open-source release provides offline inference with the core features needed for serving at scale.

Required for a complete inference service (not included in this release):

Prerequisites

Linux

GCC >= 13.0

Python 3.12

CUDA 13 (NVIDIA only)

CANN 9.2 (Ascend only)

Build from source

Before building DeepInfer, please create and activate an empty Python 3.12 environment.

Use ascend instead for the Ascend platform

Keep it exported in the environment where you run DeepInfer

export DEEPINFER_ACCELERATOR=nvidia ./build.sh

Offline inference

The default checkpoint, deepseek-ai/DeepSeek-V4.1-Flash-1017, is downloaded into the Hugging Face cache on the first run; set DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017 to run from a local copy instead.

python examples/offline_inference.py

See examples for details.

Prefill Decode

Measured on 4 SM100 GPUs for prefill and 64 for decode — each within one scale-up domain. To reproduce the benchmarks, see run_sweep.sh.

DeepInfer is released under the Apache License 2.0. See LICENSE for details.

DeepInfer was originally built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM. We thank their developers.