Milvus vs vLLM

Milvus External Collection: Index and Retrieve Lake-Resident Data Without Moving It

Engineering Aug 24, 2026 - Milvus External Collection: Index and Retrieve Lake-Resident Data Without Moving It

Vector Strike: Vector Database Semantic Search Demo

A developer published an educational retro-style arcade game called "Vector Strike" that visualizes how vector databases and embeddings work. The interactive demo maps semantic concepts to dense vectors and exposes core production mechanics — adjustable embedding dimensionality (2D/8D/32D), cosine similarity thresholds, and index types (flat scan vs HNSW graph traversal). The article explains the underlying ML concepts, shows JavaScript code for sliced cosine-similarity computation and greedy HNSW path traversal, and references real-world vector database technologies such as Pinecone, Milvus, Qdrant and pgvector. A live demo is available online and the post notes AI assistance was used for parts of the project and for the cover image. Publication date on the page is 2026-07-07.

• Author built "Vector Strike", an interactive retro-graphics game that visualizes vector database mechanics and semantic search.
• The demo lets users adjust embedding dimensionality (2D, 8D, 32D), cosine similarity threshold (τ), and choose index type (Flat Scan or HNSW).
• The article includes JavaScript implementations for sliced cosine-similarity calculation and a greedy HNSW graph traversal path generator.

MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3

How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.

Qwen3-8B inference benchmark and FP8 on Blackwell

Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.

• GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
• Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
• BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.

Tokens-per-Second Benchmarks Explained

This technical guide explains what "tokens per second" (tok/s) actually measures for local LLM inference, why single-user tok/s numbers can be misleading, and how concurrency, batching, and prompt processing change the observed speed. It contrasts single-user latency with server throughput, highlights vLLM's continuous-batching advantage versus Ollama under high concurrency, defines related metrics (P99 latency, time to first token / TTFT), and provides practical measurement advice using tools like Ollama and vLLM and calculators from notAcalculator. The article also gives realistic tok/s expectations for different model sizes on consumer hardware and lists practical tips for reading and running benchmarks yourself.

• Tokens are the unit of both billing and speed for LLMs; tokenization affects cost and measured tok/s.
• Under a Red Hat benchmark on an A100 40GB with Llama 3.1 8B, vLLM peaked around 793 tok/s combined throughput versus about 41 tok/s for Ollama at high concurrency (~19x gap).
• vLLM's key innovation is continuous batching (plus PagedAttention), which increases total throughput under concurrency compared with single-request processing tools.