Observed Signal · Sep 22, 2026 · Technical Guide · Source: Lennys Newsletter · Impact: 3/5 · Sentiment: Positive
AI Evals Market: Advanced Evals Guide: Finding Hidden AI Failures
This article is a comprehensive guide on advanced evaluation (evals) practices for AI products, authored by Hamel Husain and Shreya Shankar. It emphasizes the importance of error discovery before writing metrics, contrasting it with product discovery. The authors outline a three-step process for effective error discovery using coding agents like Codex or Claude, highlighting the pitfalls of automation bias and criteria drift. They introduce an open-source 'evals skills' plugin that assists in trace review and clustering. The article cites real-world examples from companies like Shopify, Cursor, Ramp, and Harvey, demonstrating how evals have led to significant product improvements. The piece also discusses synthetic data generation and the importance of human-in-the-loop annotation, recommending a target of 100 traces for meaningful analysis.
The article provides a practical framework for AI evaluation, a critical component for developing AI-driven advertising and marketing technologies. It underscores the importance of evals in creating reliable AI systems, which is directly relevant to the AdTech industry's increasing reliance on AI.
Wichtigste Kernpunkte & Evidenz
- Shopify used evals to build an AI workflow builder that was 2.2x faster and 68% cheaper than the frontier-model system it replaced.
- Cursor developed its Auto Balance routing performance with evals, reducing costs by 41%.
- Ramp increased its automatic receipt collection product's accuracy from 35% to 83% after investing in evals.
- Harvey rebuilt its AI contract reviewer with evals, nearly doubling the product's internal quality score.
- The authors launched an open-source 'evals skills' plugin, available on GitHub, to assist with error discovery.
Verknüpfte Unternehmen
9 verknüpfte UnternehmenHarvey
Enterprise legal AI platform for law firms and legal departments.
“Harvey rebuilt its AI contract reviewer with evals, nearly doubling the product’s internal quality score....”
Robinhood
Führende Consumer-Brokerage- und Fintech-Plattform für barrierefreies Trading, Krypto-Assets und digitale Finanz-Infrastruktur.
“Robinhood has also shared how they’ve been using evals to systematically make their AI products better....”
Anthropic
Anbieter von KI-Basismodellen, der intelligente KI-Assistenten und Modell-APIs für Entwickler und Unternehmen bereitstellt.
“Mike Krieger, Anthropic’s former CPO and now head of Labs, has said that “if there’s one thing we can teach product people, it’s that writin...”
Shopify
Eine führende Commerce-Plattform für Händler, die Online-Shops, Zahlungsabwicklung, POS-Lösungen und ein umfassendes Ökosystem bereitstellt.
“Shopify used evals to guide development of an AI workflow builder that was 2.2 times faster and 68% cheaper than the frontier-model system i...”
Cursor
AI-native code editor for developers and enterprise engineering teams.
“Cursor developed its Auto Balance routing performance with evals, resulting in much higher user satisfaction while reducing costs by 41%....”
Y Combinator
Y Combinator ist ein Startup-Accelerator und eine Investmentgesellschaft für die Frühphase.
“Garry Tan, the CEO of Y Combinator, shared that “evals are emerging as the real moat for AI startups.”...”
ElevenLabs
Eine führende Voice-AI-Plattform für Sprachgenerierung, KI-Agenten und die Erstellung von Audio-Inhalten.
“ElevenLabs has also shared how they’ve been using evals to systematically make their AI products better....”
Ramp
Ramp ist eine integrierte B2B-Finanzautomatisierungsplattform für Corporate Cards, Spesenmanagement, Zahlungsverkehr und strategische Beschaffungsprozesse.
“Ramp increased its automatic receipt collection product’s accuracy from 35% to 83% after investing in evals....”
Glean
Enterprise-KI-Plattform für organisationsübergreifende Suche, KI-Assistenten und autonome Workflow-Agents.
“Glean has also shared how they’ve been using evals to systematically make their AI products better....”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.