
Show HN: Warmctl – Fast cold starts for vLLM inference
warmctl is GPU checkpoint/restore for vLLM. It snapshots a warmed-up engine
以下正文同步自 Hacker News 最新,版权归原站所有,已转换为易读排版。
warmctl is GPU checkpoint/restore for vLLM. It snapshots a warmed-up engine and restores it in seconds on any host with the same GPUs and NVIDIA driver branch, so a new replica skips weight loading, torch.compile, kernel autotuning and CUDA graph capture. GLM-5.3-Flash on 8×H100: vLLM cold start 650 s, warmctl up 37 s, then stock serving speed. That makes scale to zero, fast autoscaling and spot GPU replacement practical for large models.
Docs · Catalog · Blog · Status and limits · Alternatives
Linux GPU host, Docker with the NVIDIA runtime, root:
curl -fsSL https://warmctl.io/install.sh | sh # newest release in /opt/warmctl sysctl -w kernel.io_uring_disabled=2 # CRIU cannot dump io_uring warmctl doctor # check the host and its GPUs warmctl catalog # models this host can restore warmctl up qwen3-8b --listen 0.0.0.0:8000 # pull, restore, serve
The public catalog has 21 models on H100, H200, B200, B300, L40S and RTX PRO 6000. Any other model vLLM serves with sleep mode, build yourself (tutorial):
warmctl build --profile qwen3-8b-tp1 --model-dir /models/qwen3-8b --gpus 0 --output /snaps/qwen3-8b warmctl push /snaps/qwen3-8b ghcr.io/you/qwen3-8b warmctl up ghcr.io/you/qwen3-8b
Measured
warmctl up to serving, snapshot in page cache, against a vLLM cold start (weights on local disk, empty compile caches).
~ is the time to a ready engine while building the snapshot; where we measured both, a plain cold start was 1–24% shorter. On B200, GLM-5.3-Flash restores in 44 s (8×B200) and DeepSeek-V4.1-Flash in 73 s (4×B200). Every platform is in the catalog.
GLM-5.3-Flash on 8×H100 by phase, from page cache and from a cold network file share (about 10 GB/s):
Restored engines serve at stock vLLM speed or faster (GLM-5.3-Flash: 100–136% of stock throughput).
build starts vLLM in a container, warms it over every shape in the profile and checkpoints the processes (CRIU), the GPUs (cuda-checkpoint) and the weights. push stores the snapshot as an OCI artifact, one per platform. up checks the host, restores everything at the same GPU addresses, runs a canary against the original engine and opens an OpenAI-compatible API. Multi-GPU engines keep NCCL and symmetric memory through NVIDIA cuInterpose.
Details: how a restore works, architecture, shared GPU memory, registries and caches.
warmctl is in beta: releases are GitHub pre-releases, and every user-facing change is in the CHANGELOG. Each restore is checked before it serves: the host must match the snapshot before anything is downloaded, every file carries its SHA-256, every weights allocation is compared by a GPU fingerprint, and a canary prompt must get the original engine's answer (bitwise for large models).
The warmctl runtime image, registry.warmctl.io/warmctl/runtime:latest (anonymous pull; pin a release tag such as :v2.0.0-beta.15 for a cluster that must not change under you), is the catalog's vLLM image with warmctl on top (make pod-image builds it). In a privileged pod it runs the same warmctl up NAME and restores in place, with readiness on /health:
kubectl apply -f deploy/kubernetes/warmctl.yaml
deploy/kubernetes also holds node preparation, a KEDA autoscaler and a KServe LLMInferenceService preset; details in docs/kubernetes.md. Tutorials: Kubernetes, KServe. Measured on GPU clusters: GLM-5.3-Flash on 8×H100 restores in 58 s from a shared file system (post), and a new KServe replica of Qwen3-8B on an L40S serves in 14 s (post).
schema: warmctl.profile/v1 id: llama31-8b-tp1 model: {id: meta-llama/Llama-3.1-8B-Instruct, revision: 0e9e39f2} parallel: {tp: 1} hardware: [h100-80gb-sxm] vllm: {max-model-len: 8192, kv-cache-memory-bytes: 21474836480}
vllm: takes any vllm serve flag; the rest comes from profiles/_defaults.yaml.
Linux x86-64 with glibc, Docker with the NVIDIA runtime, root, driver 580+, kernel.io_uring_disabled=2. A snapshot restores under any release of the driver branch it was built on (580.x, 610.x). Proven on H100 SXM, H200, B200, B300, L40S and RTX PRO 6000. warmctl hardware lists the GPU models warmctl knows and marks this host's; another model is a row in a hardware.yaml next to your profiles (--profile-dir). Keep warmctl, cuinterpose/ and the versioned dmtcp/ directory from the linux-amd64 archive side by side; the installer does this automatically. Linux arm64 archives include DMTCP but no cuInterpose yet and are CPU-tested, not GPU-qualified. The native DMTCP packages require Debian 12-compatible glibc (>= 2.36), libstdc++ and libatomic (libstdc++6 and libatomic1 on Debian/Ubuntu). macOS builds run the registry side: catalog, images, mirror, push, pull. Source builds use make build cuinterpose; see DMTCP runtime installation and upgrades. Every release is signed and has build provenance (verify).
make test · make lint · make e2e-test (CPU, Docker, root) · make release (the release archives as a local snapshot). Releases: RELEASING.md. GPU acceptance: docs/gpu-runbook.md.
Apache-2.0, including the bundled NVIDIA cuInterpose build (NOTICE).
正文由 探How 从来源站点 RSS 同步,内容未经改写;遇到排版缺失或需要图片、视频时请以原文为准。