Langfuse vs vLLM

Langfuse CLI 1.0 and new evaluator features

Langfuse CLI 1.0 released, along with new evaluator template gallery and reusable evaluators for production evaluations.

Deploying Langfuse Open-Source LLM Observability

This technical guide explains how to deploy Langfuse, an open-source observability platform for LLM applications, using Docker Compose. The deployment uses PostgreSQL for metadata, ClickHouse for trace and metrics analytics, Redis for cache/queueing, and S3-compatible object storage for media/exports, with Traefik and Let's Encrypt providing TLS. The article includes required prerequisites (Linux server 4 vCPU / 16GB RAM, Docker + Docker Compose, domain A record), step-by-step environment and docker-compose configuration, first-run setup (create organization/project and API keys), and a test-trace example using the Langfuse SDK and an OpenAI-compatible client. Publication date: 2026-08-12.

• Langfuse is an open-source observability platform for LLM applications that traces prompts/responses, tracks token usage and cost, and provides debugging analytics.
• The guide deploys Langfuse via Docker Compose using Traefik (TLS), PostgreSQL (metadata), ClickHouse (trace/metrics analytics), Redis (cache/queue), and S3-compatible object storage.
• Container images and versions shown include traefik:v3.7.0, postgres:17, clickhouse/clickhouse-server:26.5.1-alpine, and redis:7-alpine; Langfuse images used are langfuse/langfuse:3 and langfuse/langfuse-worker:3.

Reliable AI Agents: FSMs and Hidden Costs

This technical article argues that building production-grade AI agents requires engineering discipline rather than relying solely on LLM capability. It identifies common failure modes in naive agentic workflows—hallucination loops, infinite recursion, and context-window exhaustion—and recommends embedding LLMs inside deterministic Finite State Machines (FSMs) using an Orchestrator pattern to enforce valid transitions and step limits. The piece also highlights operational "hidden costs" (token complexity/latency, cost of failure, and observability/debugging overhead) and lists production best practices including human-in-the-loop approvals, structured output/schema validation, idempotent tool design, and fallback mechanisms.

• Agentic workflows are systems that perceive, plan, act, and observe to achieve multi-step goals and differ from simple prompt-response chatbots.
• Common failure modes in naive agents include: hallucination loops, infinite recursion (unbounded tool-call loops), and context window exhaustion.
• Finite State Machines (FSMs) and the Orchestrator pattern are recommended to govern LLM-driven agents, enforce valid state transitions, and limit steps.

MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3

How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.

Qwen3-8B inference benchmark and FP8 on Blackwell

Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.

• GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
• Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
• BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.

Tokens-per-Second Benchmarks Explained

This technical guide explains what "tokens per second" (tok/s) actually measures for local LLM inference, why single-user tok/s numbers can be misleading, and how concurrency, batching, and prompt processing change the observed speed. It contrasts single-user latency with server throughput, highlights vLLM's continuous-batching advantage versus Ollama under high concurrency, defines related metrics (P99 latency, time to first token / TTFT), and provides practical measurement advice using tools like Ollama and vLLM and calculators from notAcalculator. The article also gives realistic tok/s expectations for different model sizes on consumer hardware and lists practical tips for reading and running benchmarks yourself.

• Tokens are the unit of both billing and speed for LLMs; tokenization affects cost and measured tok/s.
• Under a Red Hat benchmark on an A100 40GB with Llama 3.1 8B, vLLM peaked around 793 tok/s combined throughput versus about 41 tok/s for Ollama at high concurrency (~19x gap).
• vLLM's key innovation is continuous batching (plus PagedAttention), which increases total throughput under concurrency compared with single-request processing tools.