Observed Signal · Sep 22, 2026 · Technical Guide · Source: Lennys Newsletter · Impact: 3/5 · Sentiment: Positive
AI Evals Market: Advanced Evals Guide: Finding Hidden AI Failures
This article is a comprehensive guide on advanced evaluation (evals) practices for AI products, authored by Hamel Husain and Shreya Shankar. It emphasizes the importance of error discovery before writing metrics, contrasting it with product discovery. The authors outline a three-step process for effective error discovery using coding agents like Codex or Claude, highlighting the pitfalls of automation bias and criteria drift. They introduce an open-source 'evals skills' plugin that assists in trace review and clustering. The article cites real-world examples from companies like Shopify, Cursor, Ramp, and Harvey, demonstrating how evals have led to significant product improvements. The piece also discusses synthetic data generation and the importance of human-in-the-loop annotation, recommending a target of 100 traces for meaningful analysis.
The article provides a practical framework for AI evaluation, a critical component for developing AI-driven advertising and marketing technologies. It underscores the importance of evals in creating reliable AI systems, which is directly relevant to the AdTech industry's increasing reliance on AI.
Key Takeaways & Evidence Grounding
- Shopify used evals to build an AI workflow builder that was 2.2x faster and 68% cheaper than the frontier-model system it replaced.
- Cursor developed its Auto Balance routing performance with evals, reducing costs by 41%.
- Ramp increased its automatic receipt collection product's accuracy from 35% to 83% after investing in evals.
- Harvey rebuilt its AI contract reviewer with evals, nearly doubling the product's internal quality score.
- The authors launched an open-source 'evals skills' plugin, available on GitHub, to assist with error discovery.
Connected Companies & Entities
9 Entities mappedHarvey
Enterprise legal AI platform for law firms and legal departments.
“Harvey rebuilt its AI contract reviewer with evals, nearly doubling the product’s internal quality score....”
Robinhood
Consumer brokerage and fintech app for investing and crypto.
“Robinhood has also shared how they’ve been using evals to systematically make their AI products better....”
Anthropic
Foundation model company selling AI assistants and model APIs.
“Mike Krieger, Anthropic’s former CPO and now head of Labs, has said that “if there’s one thing we can teach product people, it’s that writin...”
Shopify
Commerce platform for merchants, payments, POS and ecosystem tools.
“Shopify used evals to guide development of an AI workflow builder that was 2.2 times faster and 68% cheaper than the frontier-model system i...”
Cursor
AI-native code editor for developers and enterprise engineering teams.
“Cursor developed its Auto Balance routing performance with evals, resulting in much higher user satisfaction while reducing costs by 41%....”
Y Combinator
Startup accelerator and early-stage venture investment firm.
“Garry Tan, the CEO of Y Combinator, shared that “evals are emerging as the real moat for AI startups.”...”
ElevenLabs
Voice AI platform for speech, agents, and audio creation.
“ElevenLabs has also shared how they’ve been using evals to systematically make their AI products better....”
Ramp
Finance automation platform for cards, spend, payments and procurement.
“Ramp increased its automatic receipt collection product’s accuracy from 35% to 83% after investing in evals....”
Glean
Enterprise AI platform for search, assistants, and workflow agents.
“Glean has also shared how they’ve been using evals to systematically make their AI products better....”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.