Observed Signal · Aug 14, 2026 · Research Report · Source: Digiday · Impact: 3/5 · Sentiment: Negative
European publishers face heavier AI bot scraping
TollBit's H1 2026 analysis of 3,906 publishers (456 European) using a >987 billion-visit dataset found a sharp rise in AI-powered scraping: more than 22 billion bot accesses, about 1.9 billion events that bypassed or ignored robots.txt, and scraping up ~20% from January to June. European sites endure roughly four times the median AI scrapes of North American peers and far fewer human referrals from AI apps; TollBit measured about 179 automated page fetches per human referral and the scrape-to-referral ratio climbed from 150:1 in Q1 to 227:1 in Q2. Only 0.05% of external referrals to the European cohort came from AI apps (Google supplied ~84%), and AI-driven referrals generate far fewer clicks. Tests of commercial scrapers retrieved content on nearly all tested sites. Cloudflare has started blocking training crawlers on new domains and plans finer bot classifications and paid access from September 2026.
Report highlights growing AI scraping that can reduce referrals and undermine publisher monetization and content control; multiple datasets conflict, so findings are notable but not yet industry-shifting.
Track TollBit Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your Polaris7 workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Scope: TollBit analyzed H1 2026 across 3,906 publishers (456 European) within a >987 billion-visit dataset and identified >22 billion AI-bot accesses.
- Europe disproportionately affected: median AI scrapes per site ~4x higher than North America; scraping rose ~20% Jan–Jun 2026; overall measured ratio ~179 bot fetches per human referral, rising from 150:1 in Q1 to 227:1 in Q2.
- Referrals: only 0.05% of external referrals to the European cohort came from AI applications; Google supplied ~84% of referrals; AI chatbot referrals generate far fewer click-throughs than classic search (~96% fewer reported).
- Robots.txt and scraper tests: ~1.9 billion events bypassed/ignored robots.txt; tests of 14 commercial scrapers on 20 large UK sites retrieved content on 19; notable robots.txt bypass rates reported (e.g., ChatGPT 54%, Bytespider 48%, Perplexitybot 42%).
- Industry response and corroboration: Cloudflare now blocks training crawlers on new domains and plans finer bot categorization and paid access from Sept 2026; other datasets (Cloudflare, DataDome, Reuters Institute, Chartbeat) show mixed but corroborating signals of search traffic declines and regional variation.
Connected Companies & Entities
16 Entities mapped“A recent report from TollBit found that European sites get scraped more by AI bots, receive fewer referrals, and have their robots.txt rules...”
“Claude users in the U.S. account for more than a fifth (21.6%) of total Claude usage, and English-speaking countries make up less than a thi...”
“Cloudflare data, by contrast, does not show European publishers getting hit by bots more than North American sites, according to Lai Yi Ohls...”
“Data from DataDome, a cybersecurity and bot blocking company, also doesn’t show the same consistent gap between European and North American ...”
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
“By Sara Guaglione•August 14, 2026•”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Publishers Hit by Rising AI Scrapers and Bot Activity
New data and industry reports show a growing third‑party scraper economy and a surge in AI-driven bot activity that is extracting publishers' content at scale without returning commensurate traffic or revenue. Analysts identified dozens of vendors reselling scraped content to enterprise buyers and estimated the market around $1 billion. Akamai reports a roughly 300% rise in AI bot activity in 2025, with media companies heavily targeted; Human Security and other vendors recorded large year‑over‑year increases in AI scraper traffic. The scraped content is being purchased by scores of enterprises, while publishers report declining referral traffic and limited monetization. Publishers face technical and operational challenges blocking these actors, and some are moving to stricter bot‑allow/deny approaches via CDNs and bespoke commercial agreements.
Third‑party Scrapers Fuel Black Market for AI Content
Digiday’s Media Briefing reports growing alarm among publishers that third‑party web scrapers are harvesting publisher content and selling it into AI content marketplaces without publishers’ permission or compensation. A biannual closed‑door meeting convened by media analyst Matthew Scott Goldstein highlighted the issue; Goldstein’s recent report identified 21 vendors (including Firecrawl, Exa, Tavily, Brave, You.com, Perplexity Sonar and Bright Data) and more than 70 companies buying content from them (including BCG, IBM, Cohere, AWS, Salesforce, Apple, Dow Jones, Shopify and Alibaba). TollBit and other research found scrapers evading robots.txt, retrieving paywalled articles, charging up to $22 per 1,000 pages, and accelerating — with AI scraping showing an average quarterly growth rate of 24.4% from Q2 2025 through last year. Publishers say detection and blocking are difficult and that most scraped content monetization bypasses rights holders.
Publishers Demand AI Accountability Amid Rising Bot Traffic
Publishers report a sharp decline in human traffic and a surge in automated scraping by AI bots feeding large models. At the IAB Annual Leadership Meeting in Palm Desert, IAB CEO David Cohen called the trend a crisis and urged that AI companies be legally required to pay publishers for scraped content. AdExchanger’s episode on the topic includes insights from Lynne D Johnson (AdMonsters vet and Programmatic.ai content director) about publisher priorities for 2026. The discussion notes the expanding definition of content creators to include both traditional publishers and the broader creator community, and highlights that supply-chain transparency is a publisher priority that received relatively little attention at the conference.
Receive all future signals for TollBit & competitors in your briefing
Never miss material M&A deals, leadership changes, account wins, or product rollouts. Polaris7 monitors 20,000+ primary sources 24/7, delivering verified telemetry directly into your executive briefing.