OpenSearch vs vLLM

OpenSearch Kubernetes Operator 3.0 is now generally available

The OpenSearch Kubernetes Operator 3.0 GA release delivers production-grade rolling upgrades, TLS lifecycle management, safe scaling, and a tested migration from 2.x.

Expanding the OpenSearch Community: Welcoming our new OpenSearch Ambassadors

The OpenSearch Ambassador Program expands to over 30 global leaders, welcoming 11 new community advocates in our September 2026 cohort. Meet the new ambassadors driving open-source search and AI forward.

Online index migration and shard scaling in OpenSearch with the AOSC plugin

Learn how the open-source AOSC plugin migrates live OpenSearch indexes—changing mappings, settings, or shard counts—without losing writes or requiring downtime.

DeepSeek Open-Sources DeepInfer Inference Engine

DeepSeek open-sourced DeepInfer, the production inference engine behind its V4.1 Flash model, under Apache 2.0 during PyTorch Conference 2026. The framework supports Nvidia Blackwell (including SM100) and Huawei Ascend 950 chips, but not older Hopper, aligning with DeepSeek's strategy to reduce Nvidia dependence. It integrates previously open-sourced components like FlashMLA, DeepGEMM, and DeepEP, and includes techniques for distributed inference and speculative decoding. However, key features such as cache management, distributed execution, and load balancing remain proprietary. DeepSeek disclosed impressive metrics: 155 trillion tokens per day per cluster, 96% input cache hits, and $38 per accelerator-hour at off-peak, suggesting high margins. vLLM claims its throughput is within 6% of DeepInfer's, sparking debate. The release coincides with U.S. government actions against DeepSeek.

• DeepSeek released DeepInfer as open source under Apache 2.0.
• DeepInfer supports Nvidia Blackwell and Huawei Ascend 950, but not older Hopper.
• DeepSeek reports 155 trillion tokens per day per cluster and $38 revenue per accelerator-hour at off-peak.

MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3

How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.

Qwen3-8B inference benchmark and FP8 on Blackwell

Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.

• GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
• Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
• BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.