Observed Signal · Aug 25, 2026 · Technical Release · Source: DEV Community · Impact: 2/5 · Sentiment: Positive
Qwen3-8B inference benchmark and FP8 on Blackwell
Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.
Provides reproducible, practical inference performance data and an FP8 throughput result for a common 8B model on workstation Blackwell hardware—useful to engineers evaluating deployment choices but not a major platform-level policy or market shift.
Track vLLM Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your Polaris7 workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
- Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
- BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.
- Applying FP8 to vLLM increased aggregate throughput from 1,725 to 2,597 tok/s and single-stream from 86 to 130 tok/s (≈1.5x).
- Running FP8 on workstation Blackwell required routing around a kernel assertion (DeepGEMM -> CUTLASS fallback) due to sm_120 differences.
Connected Companies & Entities
“Stacks: vLLM 0.27.1, SGLang 0.5.9, llama.cpp (CUDA build)”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Qwen 3.5 Wins Local Benchmark Using llama.cpp
An independent Round 3 benchmark replaced Ollama with a direct llama.cpp server to measure local LLM performance more precisely. The author built a 12-task automated suite across five categories (coding, multi-file agentic coding, reasoning, tool use, and speed) and tested five models: Qwen 3.5, Gemma 4, Devstral, Codestral, and DeepSeek R1. Running on an NVIDIA RTX 5090 system, Qwen 3.5 swept the leaderboard—best coding, best agentic performance, and best single-model weighted score—reaching ~206.7 tokens/sec and a weighted overall score of 85.3. The migration reclaimed ~44 GB of disk from Ollama, enabled fine-grained inference flags (e.g., --reasoning-budget, --chat-template chatml), and highlighted Mixture-of-Experts (MoE) models’ throughput advantage for local deployment.
Unsloth Releases Qwen3.6-27B-NVFP4 with Faster Throughput
Unsloth published an open-weight NVFP4 quantized checkpoint named unsloth/Qwen3.6-27B-NVFP4, a 27B-parameter causal language model with vision encoder optimized for high-throughput inference on 24GB GPUs. The release emphasizes a reported 2.5x throughput gain over other NVFP4 quantizations, introduces enhanced "Agentic Coding" for frontend and repository-level workflows, and a "Thinking Preservation" feature to retain reasoning context across messages. The model supports a native context length of 262,144 tokens (extensible to 1,010,000), includes a Multi-Token Prediction (MTP) speculative decoding module, and is calibrated on a mix of Unsloth’s proprietary data and the UltraChat dataset. Unsloth published benchmark results comparing accuracy and throughput against NVIDIA NVFP4, FP8, and BF16 quantizations and provided recommended inference backends and environment settings for optimal performance.
Local LLMs Reach Practical Usability
A developer revisits running large language models locally and reports that the landscape has shifted: newer Qwen models (dense and MoE variants) now run acceptably on consumer-class hardware with two RX6800 GPUs and 64 GB RAM. The author highlights Qwen3.6-27B (dense) for accuracy, Qwen3.6-35B-A3B (MoE) for speed, and Qwen-Coder-Next-80B (MoE) for coding tasks. Infrastructure improvements include llama.cpp's experimental router mode, ongoing work to persist attention checkpoints and context slots, and a personal fork that adds slot save/restore. The piece also compares harnesses (Hermes, Pi) and argues that capable local inference enables offline, libre-software experimentation without relying on commercial inference providers.
Receive all future signals for vLLM & competitors in your briefing
Never miss material M&A deals, leadership changes, account wins, or product rollouts. Polaris7 monitors 20,000+ primary sources 24/7, delivering verified telemetry directly into your executive briefing.