Ollama vs vLLM

DeepSeek Launches V4.1 Flash with Novel Encoder-Decoder Architecture

DeepSeek released DeepSeek-V4.1-Flash, a 763B-parameter mixture-of-experts model employing a novel causal encoder-decoder architecture with 8B active parameters for prefill and 16B for decode. It features native vision understanding, 1M token context, an MIT license, and extreme inference efficiency, claiming up to 1/8 KV cache footprint versus V4 Flash. Independent evals (Artificial Analysis Index 40, Vals Index #1 open-weight) show it surpasses V4 Pro at lower cost. API pricing is $0.30/1M input and $1.20/1M output tokens. DeepSeek has soft-retired V4 Pro, routing traffic to V4.1 Flash. The model supports SSD offload and local deployment, with Ollama and Baseten offering day-0 support. Technical discussions highlight the architecture's novelty and potential impact on long-context agents.

• DeepSeek launched V4.1-Flash with a causal encoder-decoder architecture, 763B total params (8B prefill/16B decode active).
• Artificial Analysis Index scores V4.1-Flash at 40, above V4 Pro and below GLM-5.3-Flash.
• API pricing: $0.30 per 1M input tokens, $1.20 per 1M output tokens, cached input $0.006 per 1M.

Ollama's transparent pricing

Ollama's Pro, Max, and Team plans now use industry-standard per-token pricing with usage included on every plan.

Developer Builds Autonomous AI Agent to Hunt Paid Bounties

A developer built an autonomous AI agent that scans hundreds of online gig/bounty listings, filters scams and human-only tasks, generates deliverables using live market data and a local LLM, and notifies a human for approval. The stack uses free tools (Python orchestration, Ollama with a local model, Chart.js, public crypto APIs, GitHub Pages, Windows Task Scheduler) resulting in $0/month infrastructure cost. In 48 hours the agent found many listings but only a handful were actionable due to geo-walls, ghost sponsors, and other filters; the author highlights the need for revenue tracking and human-in-the-loop oversight.

• Author built an autonomous AI agent that scans 232+ listings across multiple platforms to find paid work and generate deliverables.
• Stack used: Python orchestration, Ollama with qwen3:4b (local LLM), Chart.js, CoinGecko, DeFiLlama, Solana RPC, GitHub Pages, and Windows Task Scheduler.
• Total stated infrastructure cost: $0/month.

MiniMax H3 on vLLM-Omni: From System-Wide Optimization to Real-Time Serving with FastVideo’s FastH3

How vLLM-Omni optimizes and scales the complete MiniMax H3 stack, then integrates FastVideo’s four-step FastH3 for generation faster than playback.

Qwen3-8B inference benchmark and FP8 on Blackwell

Independent benchmarks compare Qwen3-8B inference on an RTX PRO 6000 Blackwell (96 GB) across three serving stacks (vLLM 0.27.1, SGLang 0.5.9, and llama.cpp CUDA). At concurrency 32 using BF16, vLLM achieved 1,725 aggregate tokens/s (TTFT p50 39 ms), SGLang 1,327 tok/s (TTFT p50 42 ms), and llama.cpp 428 tok/s (TTFT p50 316 ms). Applying an FP8 checkpoint to vLLM increased throughput by ~1.5x (aggregate 1,725 -> 2,597 tok/s; single-stream 86 -> 130 tok/s) with lower latency and no detected regressions on a fixed factual check. The author documents methodology, reproductions, and an sm_120-specific kernel workaround required to run FP8 on workstation Blackwell hardware.

• GPU used: RTX PRO 6000 Blackwell, 96 GB (workstation Blackwell, sm_120).
• Model benchmarked: Qwen3-8B across vLLM 0.27.1, SGLang 0.5.9, and llama.cpp (CUDA).
• BF16, concurrency 32 aggregate throughput: vLLM 1,725 tok/s; SGLang 1,327 tok/s; llama.cpp 428 tok/s.

Tokens-per-Second Benchmarks Explained

This technical guide explains what "tokens per second" (tok/s) actually measures for local LLM inference, why single-user tok/s numbers can be misleading, and how concurrency, batching, and prompt processing change the observed speed. It contrasts single-user latency with server throughput, highlights vLLM's continuous-batching advantage versus Ollama under high concurrency, defines related metrics (P99 latency, time to first token / TTFT), and provides practical measurement advice using tools like Ollama and vLLM and calculators from notAcalculator. The article also gives realistic tok/s expectations for different model sizes on consumer hardware and lists practical tips for reading and running benchmarks yourself.

• Tokens are the unit of both billing and speed for LLMs; tokenization affects cost and measured tok/s.
• Under a Red Hat benchmark on an A100 40GB with Llama 3.1 8B, vLLM peaked around 793 tok/s combined throughput versus about 41 tok/s for Ollama at high concurrency (~19x gap).
• vLLM's key innovation is continuous batching (plus PagedAttention), which increases total throughput under concurrency compared with single-request processing tools.