vLLM vs webrtc-experiment.com

DeepSeek Open-Sources DeepInfer Inference Engine

DeepSeek open-sourced DeepInfer, the production inference engine behind its V4.1 Flash model, under Apache 2.0 during PyTorch Conference 2026. The framework supports Nvidia Blackwell (including SM100) and Huawei Ascend 950 chips, but not older Hopper, aligning with DeepSeek's strategy to reduce Nvidia dependence. It integrates previously open-sourced components like FlashMLA, DeepGEMM, and DeepEP, and includes techniques for distributed inference and speculative decoding. However, key features such as cache management, distributed execution, and load balancing remain proprietary. DeepSeek disclosed impressive metrics: 155 trillion tokens per day per cluster, 96% input cache hits, and $38 per accelerator-hour at off-peak, suggesting high margins. vLLM claims its throughput is within 6% of DeepInfer's, sparking debate. The release coincides with U.S. government actions against DeepSeek.

• DeepSeek released DeepInfer as open source under Apache 2.0.
• DeepInfer supports Nvidia Blackwell and Huawei Ascend 950, but not older Hopper.
• DeepSeek reports 155 trillion tokens per day per cluster and $38 revenue per accelerator-hour at off-peak.

MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3

How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.

Qwen3-8B inference benchmark and FP8 on Blackwell

Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.

• GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
• Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
• BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.