Observed Signal · Aug 14, 2026 · Research Report · Source: Digiday · Impact: 3/5 · Sentiment: Negative
Bot scraping affecting publishers Market: European publishers face heavier AI bot scraping
TollBit's H1 2026 analysis of 3,906 publishers (456 European) using a >987 billion-visit dataset found a sharp rise in AI-powered scraping: more than 22 billion bot accesses, about 1.9 billion events that bypassed or ignored robots.txt, and scraping up ~20% from January to June. European sites endure roughly four times the median AI scrapes of North American peers and far fewer human referrals from AI apps; TollBit measured about 179 automated page fetches per human referral and the scrape-to-referral ratio climbed from 150:1 in Q1 to 227:1 in Q2. Only 0.05% of external referrals to the European cohort came from AI apps (Google supplied ~84%), and AI-driven referrals generate far fewer clicks. Tests of commercial scrapers retrieved content on nearly all tested sites. Cloudflare has started blocking training crawlers on new domains and plans finer bot classifications and paid access from September 2026.
Report highlights growing AI scraping that can reduce referrals and undermine publisher monetization and content control; multiple datasets conflict, so findings are notable but not yet industry-shifting.
Wichtigste Kernpunkte & Evidenz
- Scope: TollBit analyzed H1 2026 across 3,906 publishers (456 European) within a >987 billion-visit dataset and identified >22 billion AI-bot accesses.
- Europe disproportionately affected: median AI scrapes per site ~4x higher than North America; scraping rose ~20% Jan–Jun 2026; overall measured ratio ~179 bot fetches per human referral, rising from 150:1 in Q1 to 227:1 in Q2.
- Referrals: only 0.05% of external referrals to the European cohort came from AI applications; Google supplied ~84% of referrals; AI chatbot referrals generate far fewer click-throughs than classic search (~96% fewer reported).
- Robots.txt and scraper tests: ~1.9 billion events bypassed/ignored robots.txt; tests of 14 commercial scrapers on 20 large UK sites retrieved content on 19; notable robots.txt bypass rates reported (e.g., ChatGPT 54%, Bytespider 48%, Perplexitybot 42%).
- Industry response and corroboration: Cloudflare now blocks training crawlers on new domains and plans finer bot categorization and paid access from Sept 2026; other datasets (Cloudflare, DataDome, Reuters Institute, Chartbeat) show mixed but corroborating signals of search traffic declines and regional variation.
Verknüpfte Unternehmen
17 verknüpfte UnternehmenDataDome
Enterprise-Software zur Prävention von Bots, Betrug und Traffic-Missbrauch.
“Data from DataDome, a cybersecurity and bot blocking company, also doesn’t show the same consistent gap between European and North American ...”
Content Bridge
Eine hochspezialisierte B2B-Plattform für die rechtskonforme, API-basierte Lizenzierung von Publisher-Inhalten an Entwickler von KI- und LLM-Infrastrukturen.
Anthropic
Anbieter von KI-Basismodellen, der intelligente KI-Assistenten und Modell-APIs für Entwickler und Unternehmen bereitstellt.
“Claude users in the U.S. account for more than a fifth (21.6%) of total Claude usage, and English-speaking countries make up less than a thi...”
STYRIA Media Group
Regionale Mediengruppe mit einem Portfolio aus Verlagshäusern, digitalen Marktplätzen und Werbedienstleistungen.
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
SPIEGEL-Gruppe
Führender deutscher Medienverlag monetisiert Qualitätsjournalismus durch Abonnements, Werbung und Content-Lizenzierung.
Perplexity
KI-Antwort-Engine mit Premium-Abonnements, Enterprise-Lizenzen und Entwickler-APIs.
TollBit
Publisher-Infrastruktur zur Monetarisierung und Kontrolle von KI-Agenten-Zugriffen.
“A recent report from TollBit found that European sites get scraped more by AI bots, receive fewer referrals, and have their robots.txt rules...”
MEEDIA
Ein dezentralisierter B2B-Marktplatz für die direkte Lizenzierung und Distribution von Creator-Owned Bild- und Video-Content ohne intermediäre Friktion.
sueddeutsche.de
Führende deutsche Digitalpublikation mit einem hybriden Monetarisierungsmodell aus Premium-Digital-Abonnements und programmatischer B2B-Werbevermarktung.
Medienfachverlag Oberauer
Führender DACH-Fachverlag für Medien-, PR- und Printing-Professionals mit integrierter B2B-Plattform-Infrastruktur für hochpräzises Audience-Targeting.
Telegraph / The Telegraph
Führendes britisches Subscription-First-Medienhaus mit datengestütztem Premium-Advertising-Inventory und First-Party-Identity-Lösungen.
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
OpenAI
Anbieter von Foundation-Modellen, der KI-Software, APIs und Abonnements für Entwickler, Unternehmen und Endverbraucher vertreibt.
Cloudflare
Globale Cloud-Plattform für Sicherheit, Performance und Edge-Application-Delivery.
“Cloudflare data, by contrast, does not show European publishers getting hit by bots more than North American sites, according to Lai Yi Ohls...”
Chartbeat
Analytik- und Optimierungssoftware für Verlage und digitale Medienunternehmen zur Echtzeit-Reichweitenmessung.
Ringier
Schweizer Medien-, Marktplatz- und Publishing-Software-Gruppe.
“Of the nearly 4,000 publishers included in TollBit’s analysis, 456 of which were European publishers, ranging from the Telegraph to Ringier ...”
Suchmaschinen-, Video-, AdTech- und Cloud-Gigant innerhalb des Alphabet-Konzerns.
Digiday
B2B-Fachmedium und Informationsplattform für Experten aus den Bereichen Medien, Marketing und Einzelhandel.
“By Sara Guaglione•August 14, 2026•...”
Ontology Mapping & Concepts
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.