DeepSeek has open-sourced DeepInfer, the in-house inference framework that runs its production AI services. The Information reports that this is the software behind the company’s unusually high-margin inference business. That makes the release more than a goodwill gesture. DeepSeek is publishing the actual stack it uses to serve its own model, not a cleaned-up demo.
The code is on GitHub under the Apache 2.0 license, so anyone can use and change it, including for commercial products. DeepSeek says the goal is to help developers “lower compute costs, reduce energy use, and make AI more affordable.”
What’s actually in the box
- A production serving engine for one model. DeepInfer powers DeepSeek-V4.1-Flash, the company’s fast, low-cost model. For now it supports only that model. It isn’t a general-purpose rival to vLLM or SGLang yet.
- Optimized kernels and parallelism. The framework includes custom GPU kernels plus parallelism methods for attention and mixture-of-experts (MoE) layers. MoE models send each token to a small group of specialized sub-networks, and a lot of the efficiency depends on how well that routing is spread across chips.
- Faster decoding with draft tokens. DeepInfer uses a decoding method that combines draft tokens with confidence estimates. The model guesses several tokens ahead and keeps the ones it trusts, so it needs fewer expensive passes per response.
- Structured outputs. It supports constrained generation such as JSON, which matters for anyone building agents or tool-calling pipelines.
- A companion “recipe” for consistency. DeepSeek also released deepseek-recipe, which copies how the official DeepSeek API processes tokens. If you self-host, your setup should behave the same way the hosted API does.
Built on open source, now handed back
DeepSeek says DeepInfer started mostly as a vLLM-based project and also borrowed ideas from SGLang and Nvidia’s TensorRT-LLM. So the company took popular open serving tools, tuned them hard for its own model, and is now releasing the tuned version.
The hardware support stands out. DeepInfer runs on Nvidia’s SM100-class GPUs (the Blackwell generation) and Huawei’s Ascend 950 NPUs. Shipping Huawei support from day one says a lot. Chinese AI labs are under pressure to cut their reliance on Nvidia, and DeepSeek is showing that its production stack can run on domestic chips.
Why the margin angle matters
DeepSeek has been here before. In early 2025 it published a theoretical cost-profit margin of about 545% for its inference service. Most of the industry thought serving frontier models at its prices meant losing money, so that number got a lot of attention.
DeepInfer is the machinery behind those economics. Other labs usually treat their serving stack as a competitive moat, because that’s where model quality becomes profit. Giving it away pushes the whole market toward cheaper inference. It also makes DeepSeek’s models more attractive to cloud providers and enterprises that want to self-host.
The caveats
This release isn’t plug-and-play. Teams should know about these gaps before they get excited:
- Offline inference only. DeepInfer leaves out parts of a full serving system. You have to bring your own KV cache system (the memory that stores past context) and your own load balancing.
- One model. Only DeepSeek-V4.1-Flash is supported for now.
- New hardware only. Older Nvidia architectures like Hopper and Ampere aren’t supported. That rules out a big share of the GPUs companies actually run today.
- No outside benchmarks yet. Nobody outside DeepSeek has checked its efficiency claims.
Who should care
The teams that will get the most from this right now:
- Cloud and inference providers serving DeepSeek models who want to match the official API’s cost and behavior
- Chinese enterprises on Ascend hardware looking for a tested production path
- Inference engineers who want to see how a lab running at real scale tunes MoE serving
What comes next
The real test is independent benchmarking. If outside teams confirm big throughput gains, expect vLLM and SGLang to adopt DeepInfer’s techniques within months, and the open-source world will gain from that whether or not people use DeepInfer itself. Wider model and hardware support would turn it from a reference build into a real contender. Either way, DeepSeek keeps pushing inference costs down for everyone. Full details are in The Information’s original report.