Observed Signal · Aug 14, 2026 · Research Report · Source: Digiday · Impact: 3/5 · Sentiment: Negative
Bot scraping affecting publishers Market: European publishers face heavier AI bot scraping
TollBit's H1 2026 analysis of 3,906 publishers (456 European) using a >987 billion-visit dataset found a sharp rise in AI-powered scraping: more than 22 billion bot accesses, about 1.9 billion events that bypassed or ignored robots.txt, and scraping up ~20% from January to June. European sites endure roughly four times the median AI scrapes of North American peers and far fewer human referrals from AI apps; TollBit measured about 179 automated page fetches per human referral and the scrape-to-referral ratio climbed from 150:1 in Q1 to 227:1 in Q2. Only 0.05% of external referrals to the European cohort came from AI apps (Google supplied ~84%), and AI-driven referrals generate far fewer clicks. Tests of commercial scrapers retrieved content on nearly all tested sites. Cloudflare has started blocking training crawlers on new domains and plans finer bot classifications and paid access from September 2026.
Report highlights growing AI scraping that can reduce referrals and undermine publisher monetization and content control; multiple datasets conflict, so findings are notable but not yet industry-shifting.
Key Takeaways & Evidence Grounding
- Scope: TollBit analyzed H1 2026 across 3,906 publishers (456 European) within a >987 billion-visit dataset and identified >22 billion AI-bot accesses.
- Europe disproportionately affected: median AI scrapes per site ~4x higher than North America; scraping rose ~20% Jan–Jun 2026; overall measured ratio ~179 bot fetches per human referral, rising from 150:1 in Q1 to 227:1 in Q2.
- Referrals: only 0.05% of external referrals to the European cohort came from AI applications; Google supplied ~84% of referrals; AI chatbot referrals generate far fewer click-throughs than classic search (~96% fewer reported).
- Robots.txt and scraper tests: ~1.9 billion events bypassed/ignored robots.txt; tests of 14 commercial scrapers on 20 large UK sites retrieved content on 19; notable robots.txt bypass rates reported (e.g., ChatGPT 54%, Bytespider 48%, Perplexitybot 42%).
- Industry response and corroboration: Cloudflare now blocks training crawlers on new domains and plans finer bot categorization and paid access from Sept 2026; other datasets (Cloudflare, DataDome, Reuters Institute, Chartbeat) show mixed but corroborating signals of search traffic declines and regional variation.
Connected Companies & Entities
17 Entities mappedDataDome
Enterprise software for bot, fraud and traffic abuse prevention.
“Data from DataDome, a cybersecurity and bot blocking company, also doesn’t show the same consistent gap between European and North American ...”
Content Bridge
B2B platform for compliant publisher content licensing to AI companies.
Anthropic
Foundation model company selling AI assistants and model APIs.
“Claude users in the U.S. account for more than a fifth (21.6%) of total Claude usage, and English-speaking countries make up less than a thi...”
STYRIA Media Group
Regional media group spanning publishing, classifieds and advertising services.
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
SPIEGEL-Gruppe
German premium publisher monetising journalism through subscriptions, advertising and licensing.
Perplexity
AI answer engine with subscriptions, enterprise plans, and developer API.
TollBit
Publisher infrastructure for monetising and controlling AI agent access.
“A recent report from TollBit found that European sites get scraped more by AI bots, receive fewer referrals, and have their robots.txt rules...”
MEEDIA
Marketplace for licensing creator-owned images and video content.
sueddeutsche.de
German digital news publisher with subscriptions and advertising sales.
Medienfachverlag Oberauer
DACH trade publisher for media, PR and printing professionals.
Telegraph / The Telegraph
UK subscription-first news publisher with premium advertising inventory.
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
OpenAI
Foundation model company selling AI software, APIs and subscriptions.
Cloudflare
Cloud platform for security, performance, and edge application delivery.
“Cloudflare data, by contrast, does not show European publishers getting hit by bots more than North American sites, according to Lai Yi Ohls...”
Chartbeat
Publisher analytics and optimisation software for digital media companies.
Ringier
Swiss media, marketplaces and publishing software group.
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
Search, video, adtech and cloud giant within Alphabet.
Digiday
B2B trade publisher for media, marketing and retail professionals.
“By Sara Guaglione•August 14, 2026•...”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.