Observed Signal · Sep 30, 2026 · Market Signal · Source: Prebid · Impact: 2/5
The State of AI Traffic: What Publishers Are Seeing in Crawls, Clicks, and Costs
By Raluca Mitchell. First in a series from the Prebid LLM Taskforce. For about twenty years, publishers had a simple bargain with crawlers: a crawler scraped your page, and a …
Track Prebid Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Connected Companies & Entities
1 Entity mappedRelated Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Stealth Crawlers Face New Legal Scrutiny
Stealth crawlers are web crawlers that scrape publisher sites without identifying themselves or their purpose, often mimicking human browsing or bypassing robots.txt. Their prevalence is growing: Cloudflare reports more than half of web traffic is bot-based, and cybersecurity firm Human Security found AI scraper traffic surged in 2025. Publishers struggle to detect and attribute stealth crawler activity, complicating efforts to block or monetize scraped content. New York passed the Stealth Crawler Prohibition Act in June, and matching federal legislation was introduced in the U.S. House, requiring bots to disclose identity and purpose and enabling enforcement with civil penalties. Publishers and trade groups (including News/Media Alliance and publishers such as People Inc.) are adopting technical mitigations like expansive bot-block lists and advocating for legal remedies to recover value and deter bad actors.
Third‑party Scrapers Fuel Black Market for AI Content
Digiday’s Media Briefing reports growing alarm among publishers that third‑party web scrapers are harvesting publisher content and selling it into AI content marketplaces without publishers’ permission or compensation. A biannual closed‑door meeting convened by media analyst Matthew Scott Goldstein highlighted the issue; Goldstein’s recent report identified 21 vendors (including Firecrawl, Exa, Tavily, Brave, You.com, Perplexity Sonar and Bright Data) and more than 70 companies buying content from them (including BCG, IBM, Cohere, AWS, Salesforce, Apple, Dow Jones, Shopify and Alibaba). TollBit and other research found scrapers evading robots.txt, retrieving paywalled articles, charging up to $22 per 1,000 pages, and accelerating — with AI scraping showing an average quarterly growth rate of 24.4% from Q2 2025 through last year. Publishers say detection and blocking are difficult and that most scraped content monetization bypasses rights holders.
Publishers: AI Data Brokers Keep 100% of Content
Publishers are warning that a growing cohort of third‑party web scrapers and AI data brokers are extracting publisher content at scale without payment, effectively capturing the full value of that content, according to a Digiday article published May 4, 2026. Industry voices — including an anonymous publishing executive and Chris Dicker, CEO of Candr Media — say these firms are acting like ad‑tech middlemen but take 100% of the content and return nothing. Media analyst Matthew Scott Goldstein’s report (citing Mordor Intelligence) estimates the “scraper economy” at roughly $1 billion and identifies dozens of vendors; TollBit’s index lists nearly 40. The article describes tactics such as stealth crawlers that ignore no‑crawl directives, syndication challenges that reintroduce scraped stories across feeds, and a shift to marketing scrubbed scraping as “agentic infrastructure,” raising legal, licensing and monetization concerns for publishers and the broader ad ecosystem.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.