Observed Signal · Aug 19, 2026 · Technical Release · Source: DEV Community · Impact: 2/5 · Sentiment: Positive
Tokens-per-Second Benchmarks Explained
This technical guide explains what "tokens per second" (tok/s) actually measures for local LLM inference, why single-user tok/s numbers can be misleading, and how concurrency, batching, and prompt processing change the observed speed. It contrasts single-user latency with server throughput, highlights vLLM's continuous-batching advantage versus Ollama under high concurrency, defines related metrics (P99 latency, time to first token / TTFT), and provides practical measurement advice using tools like Ollama and vLLM and calculators from notAcalculator. The article also gives realistic tok/s expectations for different model sizes on consumer hardware and lists practical tips for reading and running benchmarks yourself.
Practical technical guidance on measuring and interpreting tok/s, concurrency, and latency helps engineers plan local LLM deployments and capacity, but it is not a major platform policy or industry-shifting announcement.
Track vLLM Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your Polaris7 workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Tokens are the unit of both billing and speed for LLMs; tokenization affects cost and measured tok/s.
- Under a Red Hat benchmark on an A100 40GB with Llama 3.1 8B, vLLM peaked around 793 tok/s combined throughput versus about 41 tok/s for Ollama at high concurrency (~19x gap).
- vLLM's key innovation is continuous batching (plus PagedAttention), which increases total throughput under concurrency compared with single-request processing tools.
- Important metrics differ by use case: tok/s (throughput), P99 latency (tail latency under load), and time to first token (TTFT) which is dominated by prompt processing.
- Practical thresholds: ~30-40 tok/s feels instant for interactive chat; larger models (e.g., 70B) are typically much slower than 7B models on the same hardware.
Connected Companies & Entities
3 Entities mapped“This is why vLLM's key innovation, continuous batching plus PagedAttention, lets one GPU serve dramatically more total traffic than a tool t...”
“That is the single-user case, where Ollama and vLLM are genuinely close, with Ollama even slightly ahead on some hardware.”
“Under sustained load, the Red Hat benchmark on an A100 40GB with Llama 3.1 8B measured vLLM peaking around 793 tok/s of combined throughput ...”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Benchmarking LLMs with AWS Labs' LLMeter
This article is a practical guide to using AWS Labs' LLMeter, a Python-based benchmarking library for large language models. It explains the key performance metrics LLMeter captures—Time to First Token (TTFT), Tokens Per Second (TPS), Time to Last Token (TTL), and Cost Per Request—and shows how to configure experiments, endpoints, and cost models. LLMeter targets modern Python (3.10+), leverages asyncio for concurrent client simulations, and recommends streaming endpoints for accurate latency measurement. The guide covers multi-client load testing, Plotly-based interactive HTML visualizations, and a minimal live dashboard the author built for real-time monitoring. The article links to the LLMeter GitHub, a QAInsights dashboard script, and a video walkthrough for hands-on replication.
Nine local LLM interfaces tested on one GPU
A hands-on survey evaluated nine local-model interfaces on the same GPU over roughly two weeks, comparing reliability, offload behaviour, file I/O honesty, and context handling rather than just tokens/sec. Results showed large variance driven by the runtime/harness rather than model weights: Ollama was the default reliable harness (32.9 tokens/sec on a 30B MoE with 443 tokens/sec prefill); llama.cpp was faster when carefully tuned; LM Studio reliably extracted structured data to files; several tools exhibited silent failures or context-window bugs (Unsloth capped at 4096 tokens on Windows); and Claude Code could not connect reliably because local models did not parse its system-prompt format. The author concludes benchmarks must target specific real-world use cases because tool behavior, not model choice alone, determines practical outcomes.
Stop Measuring LLM Latency With One Number
The author explains that recording a single latency number for LLM requests hides multiple distinct causes of perceived slowness. They propose decomposing LLM-feature latency into separate measurements such as queue_ms, ttft_ms (time to first useful token), generation_ms, tool_ms, and end_to_end_ms (user action to visible result). The post includes a minimal Node.js example using the OpenAI SDK that logs these metrics (including stream_setup_ms and model_total_ms) and shows why separate model and feature dashboards are useful. The author also recommends grouping metrics by feature/model/provider/status and recording timing for failed requests so slow failures are not excluded from dashboards. The main operational metric recommended is end-to-end time from user action to completed visible result.
Receive all future signals for vLLM & competitors in your briefing
Never miss material M&A deals, leadership changes, account wins, or product rollouts. Polaris7 monitors 20,000+ primary sources 24/7, delivering verified telemetry directly into your executive briefing.