DataDome

European publishers face heavier AI bot scraping

TollBit's H1 2026 analysis of 3,906 publishers (456 European) using a >987 billion-visit dataset found a sharp rise in AI-powered scraping: more than 22 billion bot accesses, about 1.9 billion events that bypassed or ignored robots.txt, and scraping up ~20% from January to June. European sites endure roughly four times the median AI scrapes of North American peers and far fewer human referrals from AI apps; TollBit measured about 179 automated page fetches per human referral and the scrape-to-referral ratio climbed from 150:1 in Q1 to 227:1 in Q2. Only 0.05% of external referrals to the European cohort came from AI apps (Google supplied ~84%), and AI-driven referrals generate far fewer clicks. Tests of commercial scrapers retrieved content on nearly all tested sites. Cloudflare has started blocking training crawlers on new domains and plans finer bot classifications and paid access from September 2026.

Scraping Sites Protected by Cloudflare, DataDome, PerimeterX

This technical guide explains how modern anti-bot systems block web scrapers and describes practical, probabilistic strategies to collect public data reliably. It outlines four independent detection layers—IP reputation, TLS/HTTP fingerprint, a JavaScript sensor, and behavioral signals—and explains why simple header spoofing fails. The article compares vendor behaviours (Cloudflare, DataDome, PerimeterX/HUMAN, Akamai, Kasada), shows how clearance cookies are IP-bound, and recommends an escalation pattern: Chrome-impersonated HTTP, hardened stealth browsers, and racing fresh IPs with cookie reuse. The guide also contrasts IP tiers (datacenter, residential, mobile), warns that success is never 100% and stresses counting only real pages as successes. It positions Crawlora's Web Scraping API as an example service implementing these techniques.

AI visibility shifts from referrals to agentic distribution

Publishers are shifting their focus from expecting referral traffic from AI answer engines toward treating AI agents as a distribution layer they must control and monetize. Multiple industry reports show rapid growth in agentic AI traffic (DataDome, Decodo/Cloudflare) and rising adoption of agent-readable web standards like LLMs.txt (Originality.ai), but usage remains low. Webflow analysis finds median sites appear in a minority of AI answers and receive few citation links. Many publishers are blocking or whitelisting bots (HasData; Reuters and Time examples), but technical limits mean blocking is imperfect. Industry voices urge publishers to build nuanced crawling, indexing, and monetization policies based on agent identity, purpose, and business value rather than blanket allow/block rules.