AdTector vs DataDome

Introducing DataDome’s Hosted MCP Server for Direct AI Agent Access to Trend Reports

DataDome announced the launch of its Hosted MCP Server, enabling direct AI agent access to trend reports. This product update expands DataDome's AI agent trust and management capabilities.

Scraping Sites Protected by Cloudflare, DataDome, PerimeterX

This technical guide explains how modern anti-bot systems block web scrapers and describes practical, probabilistic strategies to collect public data reliably. It outlines four independent detection layers—IP reputation, TLS/HTTP fingerprint, a JavaScript sensor, and behavioral signals—and explains why simple header spoofing fails. The article compares vendor behaviours (Cloudflare, DataDome, PerimeterX/HUMAN, Akamai, Kasada), shows how clearance cookies are IP-bound, and recommends an escalation pattern: Chrome-impersonated HTTP, hardened stealth browsers, and racing fresh IPs with cookie reuse. The guide also contrasts IP tiers (datacenter, residential, mobile), warns that success is never 100% and stresses counting only real pages as successes. It positions Crawlora's Web Scraping API as an example service implementing these techniques.

• Modern anti-bot systems evaluate four independent layers: IP reputation, TLS/HTTP fingerprint, a JavaScript sensor, and behaviour over time.
• Cloudflare issues a cf_clearance cookie bound to IP and User-Agent after a managed challenge; changing IP voids the cookie.
• DataDome scores requests in real time, sets a datadome cookie, and is aggressive about datacenter IP ranges and fingerprint replay.

AI visibility shifts from referrals to agentic distribution

Publishers are shifting their focus from expecting referral traffic from AI answer engines toward treating AI agents as a distribution layer they must control and monetize. Multiple industry reports show rapid growth in agentic AI traffic (DataDome, Decodo/Cloudflare) and rising adoption of agent-readable web standards like LLMs.txt (Originality.ai), but usage remains low. Webflow analysis finds median sites appear in a minority of AI answers and receive few citation links. Many publishers are blocking or whitelisting bots (HasData; Reuters and Time examples), but technical limits mean blocking is imperfect. Industry voices urge publishers to build nuanced crawling, indexing, and monetization policies based on agent identity, purpose, and business value rather than blanket allow/block rules.

• DataDome reported 17.7 billion AI agent requests in April–June 2026, a 45% increase from Q1 2026 (12.2 billion); June 2026 alone had 6.6 billion requests.
• Meta generated the majority of AI agent traffic on DataDome’s network; Meta’s training crawler grew 74% Q1→Q2 2026 and its RAG crawler grew 163% in the same period.
• A July Decodo report analyzing Cloudflare data found AI-driven traffic grew ~187% in 2025 and that automated systems generated 57.4% of web requests versus 42.6% from humans.