Observed Signal · Aug 25, 2026 · Technical Release · Source: DEV Community · Impact: 2/5 · Sentiment: Positive
Large Language Models (LLM) & AI Market: Qwen3-8B inference benchmark and FP8 on Blackwell
Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.
Provides reproducible, practical inference performance data and an FP8 throughput result for a common 8B model on workstation Blackwell hardware—useful to engineers evaluating deployment choices but not a major platform-level policy or market shift.
Key Takeaways & Evidence Grounding
- GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
- Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
- BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.
- Applying FP8 to vLLM increased aggregate throughput from 1,725 to 2,597 tok/s and single-stream from 86 to 130 tok/s (≈1.5x).
- Running FP8 on workstation Blackwell required routing around a kernel assertion (DeepGEMM -> CUTLASS fallback) due to sm_120 differences.
Connected Companies & Entities
2 Entities mappedOntology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.