Observed Signal · Nov 17, 2026 · Technical Release · Source: Linas Newsletter · Impact: 3/5 · Sentiment: Positive
Nubank Tests AI Agents on Simulated Customers
Nubank's AI team published a research paper detailing a method called 'Screen Before You Serve', where AI agents are tested on thousands of simulated customers before live deployment. The approach reduced release cycles from 21.2 to 4.4 days and improved key metrics. An open-weight Qwen model replaced GPT-5.2 in production after rigorous screening, boosting self-service rates and cutting latency. The article also highlights that the main constraint in AI agent development has shifted from model capability to the testing loop, and includes a warning about simulator design, as a simpler simulator failed to rank the best agent version correctly.
Nubank's method for testing AI agents on simulated customers and migrating to open-weight models demonstrates a significant shift in AI agent development practices, which is highly relevant for the broader AdTech/MarTech/AI industry.
MARKET SIGNALS
Track Nubank Signals & Strategic Shifts
Polaris7 connects regulatory filings, primary sources, leadership changes, and M&A activity into structured intelligence. Set up your Polaris7 workspace to monitor relevant companies in your executive briefing.
- Verified primary sources
- Automated briefings
- Instant watchlist setup
Key Takeaways & Evidence Grounding
- Nubank's AI team published a research paper on 'Screen Before You Serve' method.
- Release cycles for AI agents dropped from 21.2 days to 4.4 days.
- In a live A/B test, the agent scored 4.9 points higher on self-service rate and 36.7 points higher on transactional NPS.
- An open-weight Qwen model replaced GPT-5.2 in production after screening across 29 configurations and 16,000 simulated conversations.
- Self-service rose 8.8 points and p95 latency fell 25% after the model switch.
Connected Companies & Entities
“Nubank's AI team published a research paper detailing their 'Screen Before You Serve' method for testing AI agents.”
“The article references Revolut's earlier move to cut AI costs by moving 2,000 engineers onto a self-hosted open-weight model.”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
AI Agents Bottlenecked by 4‑Minute CI Pipeline
The newsletter argues that modern AI agents operate 10–50x faster than humans, but end-to-end performance gains are being lost to tooling and infrastructure designed for human pace. Citing Jeff Dean at GTC, the author notes that making models infinitely fast yields only a 2–3x end-to-end improvement because compilers, CI pipelines, file systems, authentication flows and other human‑centric tools absorb the remainder. The piece describes a “three‑layer rebuild” toward agent‑native primitives and infrastructure, documents evidence from the METR study and Jellyfish data that human roles are shifting from execution to judgment, and offers concrete steps for engineers, leaders and buyers. It also provides four practical prompts (an Amdahl ceiling calculator, an agent‑readiness audit, a trait self‑assessment, and a taste encoder) to help organisations measure and adapt to the tooling bottleneck.
Five-question filter for evaluating AI agent launches
Nate's Substack piece (Apr 29, 2026) presents a five-question filter he applies to every AI agent launch to separate infrastructure-relevant products from feature noise. He argues the market has shifted away from model-focused debates toward infrastructure: the teams that win enterprise adoption are those that enable data access, workflow integration, and agent stacking. Nate says license spend is often wasted on flashy demos that fail in real work and highlights four recent launches that passed his filter, including ChatGPT Workspace Agents and Salesforce Headless 360. The article promises a reusable filter, guidance on matching tools (Copilot, Perplexity, Claude direct, Salesforce) to specific work, a reframing of “should I switch,” and three practical prompts for audits and layering decisions.
How to Test an AI Agent Skill Before Keeping It
The article explains how installing an AI "agent skill" does not guarantee improved work and provides a practical method to evaluate skills. It describes how skills encode another author's decisions about tools, taste, and completion criteria, which can produce unsatisfactory or homogenized outputs. The author warns that some agent platforms (e.g., OpenAI and Anthropic-powered tools) limit how many skills the model considers, causing output to dull as libraries grow. The piece presents a repeatable "one-job test" with seven steps (name the job, run it, compare results) and a nine-line test record template to produce verifiable evidence. The article concludes each skill should be either kept, forked (customized), or deleted based on test results.
WATCHLIST & BRIEFINGS
Receive future signals for Nubank & competitors in your briefing
Stay ahead of material M&A deals, leadership changes, account wins, and product rollouts. Polaris7 connects primary filings, company newsrooms, and trade reporting directly into your executive briefing.
- Regular primary-source checks
- Structured Watchlist briefings
- Preserves current context
Market Signal Monitoring
Follow affected companies in Polaris7 with regular source checks and structured Watchlist briefings.
Monitor Nubank
Follow affected companies in Polaris7 with regular source checks and structured Watchlist briefings.
