Observed Signal · Oct 7, 2026 · Funding · Source: a16z · Impact: 4/5 · Sentiment: Positive
Preference Model Raises Funding for AI RL Environments
Preference Model, a startup founded by former Anthropic and DatologyAI engineers Jennifer Zhou and Ning Cao, announced it has built reinforcement learning (RL) environments for leading AI labs and is open-sourcing its production framework, Karotte. The company focuses on creating robust environments for AI research and ML engineering tasks, addressing challenges like reward hacking. The announcement, made on a16z's newsletter, indicates that a16z is partnering with and investing in the company. Karotte has been hardened through over a million evaluation runs and controlled red-teaming. The demand for RL environments has surged over the last 18 months as labs apply them to real-world tasks in coding, research, and computer use. Preference Model aims to accelerate AI progress by enabling models to assist in building better AI.
This signals significant investment and innovation in AI training infrastructure, addressing critical issues like reward hacking and model robustness, which is essential for advancing AI capabilities in advertising and other industries.
Track Anthropic Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Preference Model announced a partnership and investment from a16z.
- The company is open-sourcing its production framework, Karotte.
- Karotte has been used in over a million evaluation runs and red-teaming.
- Founded by Jennifer Zhou (ex-Anthropic) and Ning Cao (ex-DatologyAI).
- Preference Model builds RL environments for leading AI labs.
Connected Companies & Entities
1 Entity mapped“Anthropic found that a model rewarded for cheating on coding tasks became more deceptive and willing to sabotage in completely unrelated sit...”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
RL Environments and Data Foundries Accelerate AI Scaling
The article argues that scaling reinforcement learning (RL) compute and the emergence of specialized RL environments and data foundries are driving recent capability gains in frontier AI. OpenAI's improvements are cited as largely driven by post‑training RL on a stable base model, while other labs (Anthropic, Google, xAI) also invest in pretraining and post‑training. Startups and vendors are building 'UI gyms', coding environments, and domain‑specific environments (healthcare, finance, lab robotics) and contracting domain experts for task design and grading. High demand exists for coding environments and grading pipelines (e.g., PR mining, synthetic bug generation). Labs differ in procurement strategy: Anthropic actively buys from many vendors, OpenAI is building in‑house human data teams, and Google can leverage first‑party product telemetry. The piece highlights RL for scientific discovery and closed‑loop lab experiments, economic/technical constraints for physical experiments, and enterprise demand for RL-as-a-service.
Frontier models and the case for owned, custom AI
A What’s Hot newsletter highlights a busy week of model releases from major labs (Meta, OpenAI, SpaceXAI) and spotlights Mira Murati’s Thinking Machines Lab and its mission to build multimodal, collaborative AI that organizations can own and customize. The author and their VC firm (boldstart) emphasize investing in teams that build proprietary models and data flywheels rather than only renting frontier models. The piece also references several related developments: Meta’s Muse Spark 1.1, OpenAI’s ChatGPT Work (powered by Codex and GPT-5.6), SpaceXAI’s Grok 4.5, Topos Bio’s Topos‑1, Netpreme’s X‑Mem MPU claims, Cloudflare’s Monetization Gateway waitlist (stablecoin settlement via x402), and the case for more U.S. open-weight models. Discussion topics include cost/performance tradeoffs, RL gains, local runnable frontier models, micropayments, and memory bandwidth bottlenecks in inference.
Patronus AI raises $50M to test AI agents
Patronus AI, a San Francisco startup founded in 2023 by former Meta AI researchers Anand Kannappan and Rebecca Qian, raised $50 million in a Series B round led by Greenfield Partners. The company builds simulated "digital world models" that replicate websites and internal systems to stress-test AI agents using reinforcement learning, aiming to reveal shortcuts and failure modes that benchmarks miss. Patronus says its revenue grew 15-fold over the past year and that many frontier AI labs and startups are customers. The Series B included participation from Notable Capital, Lightspeed, Datadog, and Samsung and brings Patronus’s total funding to $70 million. Current product focuses include engineering and finance simulations; the company positions itself against in‑house evaluation teams and human-data firms that assist with reinforcement learning.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.