llama.app

Lessons from Shipping Three AI MVP Side Projects

A developer recounts lessons from three AI side projects that gained real users after an earlier failed attempt. Key learnings: build the smallest working MVP instead of owning full AI infrastructure; pick a single user and workflow (build for your own pain first); and avoid hosting models yourself—use pay-as-you-go APIs or aggregators to minimize cost and friction. Concrete examples include a one-file Python commit-message generator (built in an evening), a standup-bot that evolved from a personal script into a Slack bot with 23 active users, and cost comparisons showing API usage often far cheaper than self-hosting. The author recommends prioritizing speed to prototype, using a single API call where possible, and shipping quickly to real users.

Trust the Harness, Not the Model: Local Agent Guardrails

An engineer recounts a weekend of running a local 27B coding model inside LLMKube’s Foreman harness (versions 0.8.12–0.8.13). The piece argues that a deterministic harness with gates, review, and clean-room verification is what makes stochastic local models reliable in practice. An audit after a regression (a runtime key mismatch that broke a Mac agent) uncovered self-confirming tests and other blind spots; the harness then authored three new gates (a scope guard, a reviewer rubric, and a "bite check" that rejects tests that pass against pre-change code). The model ran on both an AMD Vulkan box and an Apple Silicon M5 Max over Metal, produced some flawed gates, and was repeatedly caught by the harness' reviewer and CI processes. The project is Apache-2.0, runs on Kubernetes, and the author reports none of the activity touched cloud APIs.

PagedAttention reduces KV-cache memory for LLM serving

The article explains the KV cache — the cached Key/Value tensors required for autoregressive decoding — and why its memory growth is the main operational bottleneck for GPU-based LLM serving. It shows a Llama 3.1 70B example where a single 4,096-token sequence uses ~1.3 GB of HBM and 256 concurrent such sequences would require ~336 GB. PagedAttention (Kwon et al., 2023) applies OS-style paging to the KV cache (fixed-size token pages, page table, on-demand allocation, copy-on-write sharing and fine-grained eviction), enabling vLLM to reduce memory waste and improve throughput (published vLLM benchmarks show ~2–4× gains on mixed workloads). The post lists practical defaults (16-token pages), tuning knobs (--max-num-seqs, --max-num-batched-tokens), implementation notes (FP8 KV-cache support in vLLM v0.23.0) and scenarios where paging is not beneficial.