LogRocket

Production RAG at Scale: Lessons from 10,000+ Listings

A hands‑on account of running a production RAG pipeline that processes over 10,000 job listings daily. The author describes production decisions and tradeoffs around chunking strategy, embedding models (OpenAI vs self‑hosted Llama via Ollama), vector storage (Pinecone for prototyping, pgvector/PostgreSQL in production), LLM scoring cost controls (OpenAI Batch API, caching, model tiering, function calls), and observability (structured logging, correlation IDs, Sentry, LogRocket). The post emphasizes data normalization, designing chunkers to match document structure, and operational considerations that move RAG from demo to stable, affordable production service.

Introducing self-improving software: LogRocket finds issues, agents fix them

New from LogRocket: Galileo AI now spots your highest-impact bugs and dispatches them to AI agents in Cursor, Claude Code, or Codex to fix.

When AI Agents Fail Silently: Operational Patterns

A developer recounts shipping an AI agent that appeared flawless in demos but began producing empty or degraded responses in production without errors. He identifies three common silent failure modes—rate-limit-induced partial results, memory/context accumulation in long-running agents, and model drift between model variants—and explains instrumentation and architecture patterns to detect and mitigate them. Recommended practices include logging an AgentStepLog for every model call (model, tokens, latency, status, fallback), recording breadcrumbs to Sentry, storing detailed decision logs in PostgreSQL, and alerting on a rising fallback ratio (example: Slack alert if >10% fallbacks/hour). He also describes a required three-tier fallback stack (primary: GPT-4o/Claude 3.5 Sonnet; tier two: Groq; tier three: local Llama 3.1 via Ollama) and routing logic to preserve availability and control costs.