DeepInfer: DeepSeek Opens Its Production Inference Engine

In April 2025, DeepSeek explained why its inference engine would stay closed. This week the core of that engine surfaced on GitHub anyway. DeepInfer, published under Apache 2.0, is the scheduler, parallelism machinery, and kernel stack that serve the company's production traffic, with the DSpark speculative decoding scheme folded in. The open part covers offline inference for DeepSeek-V4.1-Flash on NVIDIA SM100 GPUs and Huawei Ascend 950 NPUs. Think engine rather than vehicle: KV cache management, stage disaggregation, and replica load balancing all sit outside the fence, and the benchmark charts ship without a baseline to compare against.

Why call this a reversal?

Because the company argued against exactly this move, in public, and then made it.

The April 2025 post in the open-infra-index repository laid out three reasons the full engine would not be released: the codebase had drifted too far from upstream vLLM to be useful to outsiders, it was welded to internal cluster management tools, and a small research team had no maintenance budget for a large open-source project. The promise attached to that refusal was a different plan: break out reusable components, publish them one at a time, and have inference-side engineering ready when each new model landed, so outside users would get support from day zero.

The eighteen months that followed read like that plan being executed on schedule. The Open Source Week contributed FlashMLA, DeepGEMM, and DeepEP. DSpark, the speculative decoding scheme, went public in late June. September brought DeepJIT for kernel JIT compilation and DeepSelect for sparse attention, then Ascend ports of DeepEP and DeepGEMM. DeepInfer is the step that was never explicitly promised: the assembly itself, the code that turns those parts into a serving system, plus the same dual NVIDIA and Ascend track the components followed.

The repository went up in the deepseek-ai GitHub organization on October 16, 2026, with pushes continuing through October 20, and it had collected more than 600 stars by the time of writing. The README's first line states the positioning: it is the high-performance inference framework that powers DeepSeek's large-scale production services, opened up first for offline inference with the core features needed for serving at scale.

What comes in the box?

Unpacked, the inventory reads as follows. The kernel stack carries the names from the earlier component releases: FlashMLA for attention, DeepGEMM for matrix work, DeepEP for MoE communication, plus TileKernels, DeepSelect, and custom TileLang operators, all tuned for DeepSeek architectures rather than offered as a general library. Around them sit attention DP and balanced context parallelism on one side and expert parallelism with EPLB balancing on the other. DSpark arrives as a built-in: draft tokens accelerate decoding while confidence estimates decide how verification is scheduled, switched on with DEEPINFER_ENABLE_MTP_VARLEN=1 and a draft cap of 5 by default. An asynchronous batch scheduler that overlaps CPU work with device execution is on out of the box, which the README describes as zero-overhead. Two determinism features, batch-invariant inference and fixed-seed deterministic sampling, are NVIDIA-only for now. Structured outputs close the list with regex, JSON schema, and EBNF constraints.

Rows compiled from the official README. The zero-overhead scheduler description is the project's wording, not an independent finding.

One line in the acknowledgments does a lot of explanatory work: DeepInfer was originally built primarily on vLLM, with inspiration from SGLang and TensorRT-LLM. That is the April 2025 story in miniature, an early fork grown into its own codebase.

What does it run on?

The support matrix is narrow and stated without hedging. One model, DeepSeek-V4.1-Flash, with deepseek-ai/DeepSeek-V4.1-Flash-1017 as the default checkpoint pulled from Hugging Face on first run; DEEPINFER_HF_MODEL_DIR swaps in a local copy. Two hardware tracks: NVIDIA SM100 GPUs on CUDA 13, and Huawei Ascend 950 NPUs on CANN 9.2. The build wants Linux, GCC 13 or newer, and Python 3.12.

# Use `ascend` instead for the Ascend platform
export DEEPINFER_ACCELERATOR=nvidia
./build.sh
python examples/offline_inference.py

That first successful run is where the release's boundary shows up. Three pieces that a live service needs are named in the README as absent: a KV cache system, disaggregation across prefill, decode, and VE stages, and load balancing between data-parallel replicas. What shipped is the engine, and the chassis is still yours to build. Offline jobs, reproduction runs, and research pipelines are in scope today; a drop-in replacement for a serving endpoint is not.

What do the benchmark charts actually say?

Two Pareto curves ship with the release, and the README text makes no numeric claims about either. Read as approximations, the prefill curve runs from about 60,000 tokens per second per GPU at 55-millisecond steps with small batches up to roughly 89,000 at 730 milliseconds with 65,536-token batches. The decode curve prices the tradeoff between one user's wait and the cluster's output: around 325 tokens per second per user at the lightest concurrency tested, about 26,000 tokens per second per GPU at the heaviest, with per-user speed still near 103. Both charts were produced on DeepSeek V4.1 Flash, and a run_sweep.sh script exists for anyone who wants to rebuild the numbers. SaveMyToken has not rerun them.

View at actual sizeClose

View at actual sizeClose

The louder fact is what the charts omit: any competitor. No vLLM curve, no SGLang curve, nothing to anchor a claim that DeepInfer is faster than the tools teams already run. The only speed numbers in circulation come from the DSpark release in June, where the vendor reported 60 to 85 percent faster per-user generation for DeepSeek-V4-Flash and 57 to 78 percent for DeepSeek-V4-Pro against its prior single-token baseline at matched capacity, with 51 to 52 percent higher aggregate throughput at moderate per-user speed targets, per VentureBeat's June 29, 2026 coverage. Those figures belong to the DSpark release, not to this framework, and they have not been checked here either.

Who should pick it up now, and who should wait?

Pick it up now if your work matches what the release actually contains. Reproducing the two curves on SM100 or Ascend 950 hardware is the obvious first project, since the scripts are provided and the configs are printed on the charts. Offline batch generation for DeepSeek-V4.1-Flash is the second. The third is any pipeline that needs outputs to be reproducible, because batch invariance and fixed-seed sampling are rare features to get straight from an engine vendor.

Wait if you need serving. Cache management, stage disaggregation, and replica balancing are do-it-yourself projects for now. Wait, too, if your models are not DeepSeek-V4.1-Flash, because the support list is exactly one line long. And if your decision hinges on whether DeepInfer beats the engine you already run, the public data does not settle that yet.

For teams that want the models' capabilities without operating any of this machinery, the SaveMyToken coding plan is the shorter path.

Frequently asked questions

Can DeepInfer be used commercially?

Yes. The repository carries the Apache 2.0 license, which permits commercial use, modification, and redistribution under its attribution and notice conditions, without warranty.

What is DSpark?

The speculative decoding scheme DeepSeek published in late June 2026. Draft tokens run ahead of the model, and confidence estimates decide how verification is scheduled. In DeepInfer it is a built-in feature behind DEEPINFER_ENABLE_MTP_VARLEN=1.

Does DeepInfer replace vLLM?

Not generally. It serves one model and ships offline inference only, while vLLM stays a general-purpose engine for many models. The shared ancestry noted in the acknowledgments makes them relatives, not rivals.