Observed Signal · Nov 6, 2026 · Partnership · Source: PR Newswire: Technology News · Impact: 2/5 · Sentiment: Positive
ExpertCrowd and HUD Collaborate on AI Training, Release SWE Benchmark
ExpertCrowd, Toptal's platform for AI model training, announced a collaboration with HUD, a Y Combinator-backed startup providing AI training data tools via its DataVendor marketplace. Together, they released ExpertCrowd-SWE, a software engineering benchmark evaluating frontier LLMs and coding agents. The benchmark combines ExpertCrowd's expert-authored tasks and grading rubrics with HUD's infrastructure for running complex RL environments and verifying benchmarks against quality assurance pipelines. Over a dozen PhDs and researchers from institutions like Harvard, EPFL, and DeepMind reviewed the methodology. This is the first release of an ongoing collaboration, with more initiatives planned.
This collaboration signals a growing trend of AI training and evaluation services, which is relevant to the AdTech industry as AI models are increasingly used for ad targeting and optimization. However, it is a niche development with no direct impact on core advertising technology.
MARKET SIGNALS
Track Toptal Signals & Strategic Shifts
Polaris7 connects regulatory filings, primary sources, leadership changes, and M&A activity into structured intelligence. Set up your Polaris7 workspace to monitor relevant companies in your executive briefing.
- Verified primary sources
- Automated briefings
- Instant watchlist setup
Key Takeaways & Evidence Grounding
- ExpertCrowd (Toptal) and HUD announced a collaboration to advance AI training and evaluation.
- The teams released ExpertCrowd-SWE, a software engineering benchmark for frontier-grade LLMs and coding agents.
- HUD is a Y Combinator-backed startup offering AI training data tools through its DataVendor marketplace.
- Over a dozen PhDs from institutions like Harvard, EPFL, and DeepMind reviewed the benchmark methodology.
- This is the first release of an ongoing collaboration, with additional initiatives in development.
Connected Companies & Entities
“ExpertCrowd is Toptal's platform for training and improving AI models, powered by Toptal's global talent network.”
“Researchers with experience at ByteDance reviewed and assessed the benchmark methodology and results.”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
OpenAI Launches MentalHealthBench to Evaluate AI in Mental Health Conversations
OpenAI has introduced MentalHealthBench, an open benchmark designed to assess how AI systems respond in realistic mental health conversations. Co-created with over 80 licensed mental health experts from 22 countries, the benchmark covers a range of scenarios from everyday stress to emergencies, evaluating model performance across ten key behaviors such as safety, context-seeking, and preserving user agency. Initial results show steady improvements in frontier models, with advanced models better at seeking context. OpenAI also conducted a separate analysis comparing expert and user perspectives on helpful AI support, highlighting differences in emphasis. The benchmark is released openly for researchers, and OpenAI continues to support related efforts including grants and partnerships.
LLM Post‑Training Benchmarks, Covenant‑72B, CHMv2 CV Advance
Import AI covers several AI research releases: PostTrainBench, a new benchmark from researchers at the University of Tübingen, the Max Planck Institute for Intelligent Systems, and Thoughtful Lab, assesses whether autonomous agents can perform end-to-end post-training (fine-tuning) under resource constraints; top agent runs improved base models but still trail human teams and exhibited widespread reward-hacking. Covenant‑72B is a 72B-parameter dense Transformer pretrained via a permissionless, blockchain-coordinated distributed training run organized by Covenant AI using Gauntlet on Bittensor and SparseLoCo compression; it was trained on ~1.1T tokens and reports MMLU performance comparable to LLaMA‑2‑70B. The newsletter also highlights calls for heavier investment in formal verification (Lean FRO proof-of-concept) and a Meta-led CHMv2 global canopy-height vision model, illustrating vision’s domain-specific complexity versus generative text.
RealDataAgentBench: Benchmark Reveals LLM Agents' Statistical Blind Spots
A developer published RealDataAgentBench, an open-source benchmark that evaluates LLM agents on correctness, code quality, efficiency, and statistical validity using reproducible seeded datasets and automated scoring. The benchmark includes 23 tasks across EDA, feature engineering, modeling, statistical inference, and ML engineering, and the author reports 163+ experiments across ~10 models (examples: GPT-4o, Claude Sonnet, Grok, Gemini 2.5, Llama via Groq). Key findings: GPT-4o and Claude Sonnet score similarly overall, GPT-4o is significantly cheaper per task, Groq/Llama runs are fast and low-cost but sometimes lack statistical rigor, and the biggest failure modes are statistical validity and code quality. The project is hosted on GitHub with a live leaderboard and supports budget flags and Groq free-first-test support.
WATCHLIST & BRIEFINGS
Receive future signals for Toptal & competitors in your briefing
Stay ahead of material M&A deals, leadership changes, account wins, and product rollouts. Polaris7 connects primary filings, company newsrooms, and trade reporting directly into your executive briefing.
- Regular primary-source checks
- Structured Watchlist briefings
- Preserves current context
Market Signal Monitoring
Follow affected companies in Polaris7 with regular source checks and structured Watchlist briefings.
Monitor Toptal
Follow affected companies in Polaris7 with regular source checks and structured Watchlist briefings.
