2026-10-21
DeepInfer, the engine DeepSeek uses to serve its own models
DeepSeek publishes DeepInfer, the inference framework behind its own production. This release covers DeepSeek-V4.1-Flash offline, on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs.
DeepInfer is the high-performance inference framework DeepSeek uses for its own services. The code is Apache 2.0. This first public release is offline inference, plus the core pieces for working at scale. In the SophyAI catalog it is a reference engine, not launchable yet.
Which model
The supported model is DeepSeek-V4.1-Flash. The example examples/offline_inference.py downloads deepseek-ai/DeepSeek-V4.1-Flash-1017 on the first run. A local copy is selected with DEEPINFER_HF_MODEL_DIR.
DeepInfer was built primarily on vLLM, with ideas from SGLang and TensorRT-LLM. The kernels named in the README are FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect and TileLang kernels.
What it does
- Attention in data parallel and balanced context parallel, and MoE in expert parallel with EPLB.
- DSpark and confidence-scheduled verification.
DEEPINFER_ENABLE_MTP_VARLEN=1turns it on;DEEPINFER_USE_NEXTN_DECODEsets the maximum draft length, 5 by default. - Bounded sliding-window replay, to cut prefill work. The published V4.1 settings are
DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compactandDEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140. - An asynchronous scheduler, on by default: CPU batch preparation overlaps device execution.
- Batch-invariant inference and deterministic sampling, on NVIDIA, with
DEEPINFER_BATCH_INVARIANT=1. - Outputs constrained to a regex, a JSON schema or an EBNF grammar.
Which hardware
The stated GPUs are NVIDIA SM100. The stated NPUs are Huawei Ascend 950. It needs Linux, GCC 13 or newer, Python 3.12 and, on NVIDIA, CUDA 13. The build starts from an empty Python environment, with DEEPINFER_ACCELERATOR=nvidia or ascend, then ./build.sh.
The performance curves in the repository were measured by DeepSeek on 4 SM100 GPUs for prefill and 64 for decode, each inside one scale-up domain. The charts are in the README. SophyAI does not republish those points as its own measurements.
In the catalog
The profile dsv41-deepinfer-b200-4x pairs DeepInfer with 4× B200, the SM100 class already in the catalog, and the 1017 checkpoint. The catalog list price for 4× B200 is 20–28 USD/h. It is a reference: launch=reference and launchable=false.
The README lists, as parts of a full service to add beside this release, a KV cache system, prefill/decode disaggregation, and load balancing across replicas. The catalog path that already speaks the OpenAI API remains vLLM on 8× H200, on 8× B200 and, as an advanced profile, on 4× B200.