Observed Signal · Jun 9, 2026 · Policy Update · Source: Digiday · Impact: 4/5 · Sentiment: Positive
Reuters and Time adopt bot‑blocking whitelists
Reuters and Time have switched to blocking AI bots by default and are maintaining whitelists of approved crawlers to control access to their on-site content. Time allows roughly 70 approved bots and uses ScalePost to manage bot access; Reuters is enforcing a block‑all‑bots posture via robots.txt while combining monitoring tools, server‑level enforcement, commercial licensing and legal protections. Publishers say the change creates friction that raises the cost of scraping, helps identify unauthorized scrapers, supports licensing discussions, and can reduce hosting costs. Industry groups including the IAB Tech Lab and the SPUR Coalition are publishing guidance and technical standards to help publishers adopt similar measures.
Major publishers changing default bot policy and publishing whitelists raises the cost of unauthorized scraping, supports licensing negotiations, affects ad traffic and measurement, and sets a precedent for other publishers.
Track TIME Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Reuters and Time now block AI bots by default and maintain whitelists of approved bots to access content.
- Time allows about 70 bots to access onsite content and uses ScalePost to manage AI bots, according to COO Mark Howard.
- Reuters implements a block‑all‑bots approach using robots.txt and complements it with bot monitoring, server enforcement, commercial licensing and legal protections.
- A Tollbit report found that 30% of AI bot scrapes in Q4 2025 ignored robots.txt permissions; Reuters says adding latency and friction can deter scrapers.
- Reuters has AI licensing agreements with companies including Microsoft and Meta, and Reuters' robots.txt shows allowed bots from Amazon, Google, Bing and Yahoo.
Connected Companies & Entities
5 Entities mappedOntology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Publishers Lobby Congress for 'Bad Bots' Bill
Over 300 news publishing executives, including leaders from Condé Nast, Hearst Magazines, USA Today Co., and The Seattle Times, traveled to Washington D.C. to lobby Congress for the Stealth Bot Prohibition Act. The bill would require AI stealth crawlers to identify themselves, preventing them from disguising traffic and bypassing publishers' scraping blocks. Organized by News/Media Alliance, the event follows a similar New York state law and aims to address the growing problem of AI bots scraping content without permission. Executives met with lawmakers to emphasize the need for transparency, control, and fair compensation when their content is used for AI training. The initiative highlights the increasing urgency as AI bot traffic has surged significantly, with TollBit detecting over 22 billion AI bot scrapes in the first half of 2026.
Publishers Rebuild Web for AI Agents
Publishers including Time, The Economist and Le Monde are experimenting with agent‑readable versions of their sites and stricter bot controls to prepare for an agentic web where AI agents fetch and act on content. Time has converted pages into markdown, blocks unknown AI crawlers by default and whitelists approved bots that are redirected to markdown pages; it uses TollBit to perform HTML-to-markdown conversions. The Economist is piloting agent‑readable feeds for marketing and B2B material, while a third major publisher is testing the Web Model Context Protocol (WebMCP), a web standard co-developed by Google and Microsoft to share structured data with AI agents. Publishers cite benefits such as improved visibility in AI search, lower CDN/bot costs and reduced token consumption, while some consultants warn rebuilding for agents should be a deliberate, value-driven decision.
U.S. Bill Targets Stealth Web-Scraping Bots
A bipartisan bill introduced in the U.S. House would require so-called “stealth” or AI web crawlers to disclose their identity and purpose to site hosts. The Stealth Bot Prohibition Act, authored by the News/Media Alliance and sponsored by Reps. Valerie Foushee, Laurel Lee and Gus Bilirakis, would empower enforcement by the Federal Trade Commission and state attorneys general and impose fines of $53,000 per violation. Publishers and trade groups say clandestine bots scrape and resell editorial content, drain hosting resources, reduce publisher revenues and feed training data to AI systems. Media companies including Axel Springer and USA TODAY Co. back the measure, citing real costs (e.g., Politico dedicating ~25% of hosting costs to bot management) and national-security concerns tied to large-scale scraping activity.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.