Observed Signal · Sep 22, 2026 · Technical Guide · Source: Lennys Newsletter · Impact: 3/5 · Sentiment: Positive
Advanced Evals Guide: Finding Hidden AI Failures
This article is a comprehensive guide on advanced evaluation (evals) practices for AI products, authored by Hamel Husain and Shreya Shankar. It emphasizes the importance of error discovery before writing metrics, contrasting it with product discovery. The authors outline a three-step process for effective error discovery using coding agents like Codex or Claude, highlighting the pitfalls of automation bias and criteria drift. They introduce an open-source 'evals skills' plugin that assists in trace review and clustering. The article cites real-world examples from companies like Shopify, Cursor, Ramp, and Harvey, demonstrating how evals have led to significant product improvements. The piece also discusses synthetic data generation and the importance of human-in-the-loop annotation, recommending a target of 100 traces for meaningful analysis.
The article provides a practical framework for AI evaluation, a critical component for developing AI-driven advertising and marketing technologies. It underscores the importance of evals in creating reliable AI systems, which is directly relevant to the AdTech industry's increasing reliance on AI.
Track Shopify Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your Polaris7 workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Shopify used evals to build an AI workflow builder that was 2.2x faster and 68% cheaper than the frontier-model system it replaced.
- Cursor developed its Auto Balance routing performance with evals, reducing costs by 41%.
- Ramp increased its automatic receipt collection product's accuracy from 35% to 83% after investing in evals.
- Harvey rebuilt its AI contract reviewer with evals, nearly doubling the product's internal quality score.
- The authors launched an open-source 'evals skills' plugin, available on GitHub, to assist with error discovery.
Connected Companies & Entities
9 Entities mapped“Shopify used evals to guide development of an AI workflow builder that was 2.2 times faster and 68% cheaper than the frontier-model system i...”
“Cursor developed its Auto Balance routing performance with evals, resulting in much higher user satisfaction while reducing costs by 41%.”
“Ramp increased its automatic receipt collection product’s accuracy from 35% to 83% after investing in evals.”
“Harvey rebuilt its AI contract reviewer with evals, nearly doubling the product’s internal quality score.”
“Mike Krieger, Anthropic’s former CPO and now head of Labs, has said that "if there’s one thing we can teach product people, it’s that writin...”
“Garry Tan, the CEO of Y Combinator, shared that "evals are emerging as the real moat for AI startups."”
“Glean has also shared how they’ve been using evals to systematically make their AI products better.”
“Robinhood has also shared how they’ve been using evals to systematically make their AI products better.”
“ElevenLabs has also shared how they’ve been using evals to systematically make their AI products better.”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Is Your AI Agent Eval Set Testing Anything?
The article argues that an evaluation (eval) set is the enduring product that defines whether an AI agent is production-ready. Eval sets remain meaningful across model swaps, prompt rewrites, tool additions, and provider changes because they encode the definition of "working" independently of implementation. The author recommends building eval sets from real production failures (making every incident a permanent test case), weighting tests toward disqualifying failure modes, matching the distribution of normal traffic, and asserting on behavior rather than exact output strings. The piece references open evaluation frameworks (e.g., OpenAI's evals) and highlights reliability challenges for nondeterministic agents.
AI Agents Produce Flawed Production Code: Evaluation Bottleneck
An engineer who spent months grading AI-agent-generated code reports a recurring failure pattern: agent outputs are often syntactically correct but blind to real-world failure modes (retries, timeouts, partial writes, IAM, concurrency, distributed state). The author argues this is an evaluation problem — not a pure model capability issue — and says job roles like "AI evaluator" and practices such as RL environment design and LLMOps are emerging to address it. They describe common failures (reward hacking, golden-path assumptions) and announce they are building an open fault-injection harness to stress-test agent-generated infrastructure code with deterministic pass/fail checks, combining chaos engineering with AI evaluation. The author will publish the project on their portfolio and GitHub and invites collaboration.
Build Your First AI Eval in Claude
This guide (published 2026-07-28) explains a practical method for building an offline, "cold-start" evaluation (eval) for AI features before any production data exists. Drawing on Daniel McKinnon (former PM on Llama at Meta), the article describes creating an eval project in Claude or ChatGPT, defining a one-sentence feature spec, constructing an answer-first set of cases (floor and ceiling), using AI to generate ~100 varied test cases, and grading outputs with a calibrated binary pass/fail judge across three criteria (substantive, format, scope). It recommends iterative slicing to set guardrails and targets for engineering, highlights that PM judgment is central to eval design, and includes sponsors and related resources and podcasts.
Receive all future signals for Shopify & competitors in your briefing
Never miss material M&A deals, leadership changes, account wins, or product rollouts. Polaris7 monitors 20,000+ primary sources 24/7, delivering verified telemetry directly into your executive briefing.