Observed Signal · Aug 19, 2026 · Technical Release · Source: DEV Community · Impact: 2/5 · Sentiment: Positive
Large Language Models (LLM) & AI Market: Tokens-per-Second Benchmarks Explained
This technical guide explains what "tokens per second" (tok/s) actually measures for local LLM inference, why single-user tok/s numbers can be misleading, and how concurrency, batching, and prompt processing change the observed speed. It contrasts single-user latency with server throughput, highlights vLLM's continuous-batching advantage versus Ollama under high concurrency, defines related metrics (P99 latency, time to first token / TTFT), and provides practical measurement advice using tools like Ollama and vLLM and calculators from notAcalculator. The article also gives realistic tok/s expectations for different model sizes on consumer hardware and lists practical tips for reading and running benchmarks yourself.
Practical technical guidance on measuring and interpreting tok/s, concurrency, and latency helps engineers plan local LLM deployments and capacity, but it is not a major platform policy or industry-shifting announcement.
Key Takeaways & Evidence Grounding
- Tokens are the unit of both billing and speed for LLMs; tokenization affects cost and measured tok/s.
- Under a Red Hat benchmark on an A100 40GB with Llama 3.1 8B, vLLM peaked around 793 tok/s combined throughput versus about 41 tok/s for Ollama at high concurrency (~19x gap).
- vLLM's key innovation is continuous batching (plus PagedAttention), which increases total throughput under concurrency compared with single-request processing tools.
- Important metrics differ by use case: tok/s (throughput), P99 latency (tail latency under load), and time to first token (TTFT) which is dominated by prompt processing.
- Practical thresholds: ~30-40 tok/s feels instant for interactive chat; larger models (e.g., 70B) are typically much slower than 7B models on the same hardware.
Connected Companies & Entities
2 Entities mappedvLLM
Open-source LLM inference and serving engine.
“This is why vLLM's key innovation — continuous batching plus PagedAttention — lets one GPU serve dramatically more total traffic than a tool...”
Ollama
Local and cloud infrastructure for open-model AI development.
“That is the single-user case, where Ollama and vLLM are genuinely close, with Ollama even slightly ahead on some hardware....”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.