DeepInfer: DeepSeek's Open-Source Inference Engine — A Practical Guide

*Verified against the official repo and deployment Q&A on 2026-10-27. 23,000+ GitHub stars in the first 24 hours. This version adds first-hand numbers from the official Q&A doc and the companion repo most coverage missed.*

What It Is

DeepInfer is the high-performance inference framework that powers DeepSeek's production services, now open-sourced under Apache-2.0. Built primarily on vLLM, with additional inspiration from SGLang and TensorRT-LLM.

Supported model: DeepSeek-V4.1-Flash (default checkpoint deepseek-ai/DeepSeek-V4.1-Flash-1017).

Supported hardware: NVIDIA SM100 GPUs (Blackwell) and Huawei Ascend 950 NPUs.

The Numbers (Verified From the Official Q&A)

From doc/inference_deployment_qa.md in the repo:

> "A single cluster currently serves 155 trillion tokens per day, with overall revenue of US$38 per GPU-hour (at valley pricing)."

This is the source of the numbers circulating on HN and Weibo. Both are from the official document, not third-party estimates.

What Most Coverage Missed: deepseek-recipe

Alongside DeepInfer, DeepSeek also open-sourced [deepseek-recipe](https://github.com/deepseek-ai/deepseek-recipe) — a production-grade token encoding/decoding component that converts between mainstream LLM API protocols and model token sequences. Its processing logic is fully aligned with the official DeepSeek API, so third-party deployments get the same API behavior as the official one. It prevents special-token injection by segmenting on special tokens and encoding segments in parallel, reducing tokenization latency for long inputs.

2,565 stars as of 2026-10-27. Almost no coverage mentioned it.

What Is NOT Open-Sourced (Important)

The KV cache management and load balancing components are tightly coupled to DeepSeek's internal infrastructure and are not in this release. To build a complete production serving stack you still need:

• A KV cache system

• VE/Prefill/Decode disaggregation orchestration

• Load balancing across data-parallel replicas (Prefill / Decode / Vision Encoder / Expert-Parallel load balancers)

The engine core is there; the serving infrastructure is not.

Production Architecture Details (From the Official Q&A)

The Q&A doc (doc/inference_deployment_qa.md, 28.9 KB) contains unusually specific production details:

Parallelism strategy:

• Prefill: EP2–8 / DP2–8 (smallest EP/DP that HBM allows)

• Decode: EP64–128 / DP64–128 (Wide-EP — largest EP/DP available GPUs allow)

• Vision encoder: one GPU per instance, cheaper GPUs recommended

• No TP or CP

Smallest efficient deployment: 64 GPUs prefill + 64 GPUs decode (because decode recommends EP64+). 1P+1D is technically deployable.

Storage: ~14.4 TB logical KV cache per GPU. Recommended storage server: 20 × 30.73 TB SSDs, ≥40 GB/s read, 5 GB/s write, 24h retention. 3FS distributed FS with 10+2 erasure coding (tolerates 2 simultaneous node failures per EC group).

Cross-datacenter: technically feasible (V4.1 Flash compresses KV cache substantially); budget ~400 Gbps interconnect per 8 GPUs. DeepSeek's own nodes run in different but adjacent DCs.

Load balancing: Prefill LB buckets requests into 4 tiers by context length (<16K / 16–64K / 64–256K / ≥256K) with dynamic quota redistribution; a fitted quadratic cost model cost(q, c) solves chunk size per step against a nanosecond time budget. Decode LB balances request count vs. total KV cache length per GPU. EPLB uses redundant hot-expert replicas with periodic reassignment (prefill: 4 redundant experts/GPU, 100-step window, reassign every 1,000 steps; decode: 1 redundant expert/GPU, 1,000-step window, reassign every 10,000 steps). Measured moe_lb (slowest/mean): 1.045–1.090.

How to Run It

Prerequisites: Linux, GCC ≥ 13.0, Python 3.12, CUDA 13 (NVIDIA) or CANN 9.2 (Ascend).

export DEEPINFER_ACCELERATOR=nvidia   # or "ascend"
./build.sh
python examples/offline_inference.py   # downloads model on first run
# or use a local copy:
export DEEPINFER_HF_MODEL_DIR=/path/to/DeepSeek-V4.1-Flash-1017

Key Features (and How to Enable Them)

• Optimized kernels: FlashMLA, DeepGEMM, DeepEP, TileKernels, DeepSelect, custom TileLang kernels.

• DSpark + confidence-scheduled verification: DEEPINFER_ENABLE_MTP_VARLEN=1; DEEPINFER_USE_NEXTN_DECODE sets max draft length (default 5).

• Decoder SWA bounded replay: DEEPINFER_YOCO_PREFILL_TRUNCATION_MODE=compact, DEEPINFER_YOCO_PREFILL_KEEP_TOKENS=140.

• Asynchronous scheduling: enabled by default (zero-overhead batch scheduler).

• Batch-invariant inference (NVIDIA only): DEEPINFER_BATCH_INVARIANT=1.

• Deterministic sampling (NVIDIA only): batch-invariant + DEEPINFER_ENABLE_MTP_VARLEN=0 + fixed seq_seed.

• Structured outputs: regex / JSON schema / EBNF via SamplingParams(so_args={"schema_type": ..., "schema": ...}).

How It Compares to vLLM / TensorRT-LLM

1. DeepSeek-specific optimizations (DSpark, SWA bounded replay, FlashMLA/DeepGEMM kernels) not in upstream vLLM.

2. Dual hardware support: NVIDIA SM100 + Ascend 950 (vLLM is primarily NVIDIA).

3. This release is offline inference only — vLLM ships a full serving stack.

4. Same Apache-2.0 license.

Bottom Line

DeepInfer is the actual production engine behind DeepSeek's API — 155T tokens/day at $38/GPU-hour (valley) is the disclosed operating point. If you run DeepSeek-V4.1-Flash on SM100 or Ascend 950, this is the engine. But plan to build or source your own KV cache, disaggregation, and load balancing — those are deliberately not in the box. And if you want API-compatible behavior, look at deepseek-recipe too.

Repo: https://github.com/deepseek-ai/DeepInfer · Companion: https://github.com/deepseek-ai/deepseek-recipe · Q&A: doc/inference_deployment_qa.md · License: Apache-2.0