Observed Signal · Jul 1, 2026 · Policy Update · Source: techcrunch · Impact: 4/5 · Sentiment: Positive

Cloudflare blocks mixed-use crawlers on ad pages

Executive Signal Summary

Cloudflare announced a policy change that, starting September 15, 2026, will by default block “mixed-use” web crawlers (those that combine search, agent use, and training) from crawling pages that host ads. The new defaults apply to new Cloudflare customers, new sites of existing customers, and all existing free customers unless site owners change their settings. Cloudflare says the change encourages AI companies to separate search crawlers from agents and training crawlers and enables new commercial opportunities for publishers, evolving its Pay Per Crawl marketplace into a Pay Per Use model. Initial partners for publisher payments are Ceramic.ai and You.com. Cloudflare also cited internal data showing over 50% of AI crawler traffic re-fetches unchanged pages, and framed the move as protecting publishers’ IP and bandwidth while reshaping access for AI model providers.

Polaris7 AgentPolaris7 Strategic Assessment
High Confidence

A major CDN changed default crawler policy that affects publisher monetization, data access for AI model training, and crawler behavior industry-wide; it could materially alter how AI providers fetch and pay for web content.

MARKET SIGNALS

Track Cloudflare Signals & Strategic Shifts

Polaris7 connects regulatory filings, primary sources, leadership changes, and M&A activity into structured intelligence. Set up your Polaris7 workspace to monitor relevant companies in your executive briefing.

  • Verified primary sources
  • Automated briefings
  • Instant watchlist setup

Key Takeaways & Evidence Grounding

  • Effective September 15, 2026, Cloudflare will by default block mixed-use crawlers from pages that host ads.
  • The default change applies to new Cloudflare customers, new sites of existing customers, and all existing free customers unless site owners adjust settings.
  • Cloudflare is evolving its Pay Per Crawl marketplace into a Pay Per Use model to let publishers charge AI companies when content generates value.
  • Cloudflare is initially working with partners Ceramic.ai and You.com to pay publishers when content is used or appears in AI search results.
  • Cloudflare data suggested over 50% of crawl traffic from AI crawlers is spent re-fetching unchanged pages.

Connected Companies & Entities

“Cloudflare has just issued the AI industry a new deadline to separate the web crawlers used for traditional search purposes, like Google Sea...”

“Cloudflare specifically calls out the world largest search engine as having access to about twice more information than other AI companies b...”

“To put this into action, Cloudflare is initially working with two partners, Ceramic.ai and You.com.”

Ontology Mapping & Concepts

#mixed-use crawlers#Pay Per Use#Pay Per Crawl
Primary Source Grounding & Direct Attribution
Direct Origin Attribution
Primary Reporting: techcrunch•Published: Jul 1, 2026
Original Coverage Title: “Cloudflare’s new policy pushes AI companies to pay for publishers’ content”

Related Market Signals & Shifts

Recent verified developments and strategic activity across this market segment.

InfrastructureSep 16, 2026

Cloudflare introduces new controls for search crawlers, AI training, and agents.

Cloudflare is introducing new controls that let website operators separately decide whether their content can be used for traditional search, AI training, or AI agents. This aims to resolve the trade-off between being discoverable in search and protecting content from being used to train AI models. The company is replacing its 'Block AI Bots' toggle with three independent settings, including a 'Disallow AI Training' option. It is also introducing an 'Accountable' status for AI crawlers, which requires them to support independent opt-outs for AI training and search. Apple, Google, and Microsoft are already compliant or have committed to a timeline. Additionally, Cloudflare is replacing 'Managed Robots.txt' with 'Bot Preference Sync' for centralized control and is working on an open standard called 'ai-prefs' for expressing site preferences.

Read assessment
Publisher Monetization, AIOct 13, 2026

Cloudflare Launches Pay-Per-Use for AI Content Usage

Cloudflare, which routes about 20% of global internet traffic, is addressing the imbalance between AI bots scraping publisher content and the traffic they return. The company has introduced dashboards for publishers to monitor AI bot activity (e.g., ChatGPT, Claude) and is now beta-testing a 'pay per use' feature. This new model replaces the earlier 'pay per crawl' plan, allowing publishers to charge AI companies based on the actual value derived from their content, such as when an LLM uses content to answer a query. Publishers can choose to block bots or accept offers from AI firms. Lara Cohen, VP of Strategic Partnerships for Media, Creators & AI at Cloudflare, emphasizes the need for honest crawler declaration. The initiative aims to help publishers monetize AI usage without replacing ad revenue entirely.

Read assessment
InfrastructureMar 19, 2026

Cloudflare launches compliant crawler, sparking publisher tension

Cloudflare released a Crawl API (a crawl endpoint within its browser rendering API) that can scrape an entire website with one request and return content in HTML, Markdown, or structured JSON. The launch prompted publisher backlash after some sites reported they could not initially block Cloudflare’s crawler; Cloudflare product lead James Smith acknowledged messaging and implementation issues and said they have been fixed. The product is positioned as a compliant intermediary between publishers and AI builders, intended to respect publisher controls, reduce inefficient mass crawling, and create monetization options (following a prior pay-per-crawl offering). Publishers welcome tools that reduce server strain and preserve page performance, while some remain wary that intermediaries concentrating crawl control could shift power dynamics. Cloudflare says the goal is to establish best practices and support both supply (publishers) and demand (AI companies) sides of an emerging licensed AI content market.

Read assessment

WATCHLIST & BRIEFINGS

Receive future signals for Cloudflare & competitors in your briefing

Stay ahead of material M&A deals, leadership changes, account wins, and product rollouts. Polaris7 connects primary filings, company newsrooms, and trade reporting directly into your executive briefing.

  • Regular primary-source checks
  • Structured Watchlist briefings
  • Preserves current context