Ollama

RAGnarok: Scoping an Enterprise RAG System

A developer-published walkthrough launching a public series called RAGnarok that outlines the scope and architecture for an enterprise Retrieval-Augmented Generation (RAG) knowledge assistant. Part 1 describes the problem (scattered internal documentation), a proposed tech stack (Sentence Transformers, ChromaDB, LangChain, OpenAI/Ollama), a project folder structure, and a four-phase build plan from ingestion to production hardening. The author notes Part 2 will cover the ingestion pipeline (extractor.py, chunker.py, embedder.py, loader.py) and says code and a repo link will follow once Phase 1 is implemented.

Local-First AI: On-Device Inference & Agent Harnesses

This technical deep dive argues for a shift from cloud-first to local-first AI architectures, focusing on engineering on-device inference and building custom agent harnesses. It outlines benefits of local inference—lower latency (token generation under 10ms with NPU acceleration), improved data sovereignty and privacy (GDPR/HIPAA/CCPA compliance), cost predictability, and offline capability. The article surveys the local inference stack (e.g., llama.cpp, Ollama, MLC LLM, ExLlamaV2, Candle), explains GGUF model format and quantization strategies (FP16, Q8_0, Q4_K_M, Q2_K), and provides Python examples using llama-cpp-python and a ReAct-style agent harness. It also covers performance optimizations (KV cache, model parallelism, kernel fusion) and security mitigations (strict tool definitions, sandboxing, JSON schema validation).

Prefill Performance Undermines the AI PC

This technical analysis benchmarks large-model inference on three consumer machines and compares them to a free-tier cloud model. It explains inference has two phases — prefill (compute-bound, benefits from GPU) and generation (memory-bandwidth-bound) — and shows prefill dominates latency for large prompts. Measured with an 18 GB Gemma 4 26B model, prefill rates varied ~18x across machines (360 to 20 tok/s) while generation varied <2x. Model load time depended on storage (NVMe ~8s vs SATA ~50s), creating long cold-call stalls if models are unloaded. An AMD-powered laptop marketed as an “AI PC” failed on large prompts because its NPU was not used by the runner, leaving slow CPU prefill. A free cloud model (Google Gemini 3 Flash) returned answers faster end-to-end than local GPUs in the tested scenario.