TollBit

Publishers Rebuild Web for AI Agents

Publishers including Time, The Economist and Le Monde are experimenting with agent‑readable versions of their sites and stricter bot controls to prepare for an agentic web where AI agents fetch and act on content. Time has converted pages into markdown, blocks unknown AI crawlers by default and whitelists approved bots that are redirected to markdown pages; it uses TollBit to perform HTML-to-markdown conversions. The Economist is piloting agent‑readable feeds for marketing and B2B material, while a third major publisher is testing the Web Model Context Protocol (WebMCP), a web standard co-developed by Google and Microsoft to share structured data with AI agents. Publishers cite benefits such as improved visibility in AI search, lower CDN/bot costs and reduced token consumption, while some consultants warn rebuilding for agents should be a deliberate, value-driven decision.

Le Monde Blocks Bots, Plans Agent Identity for Subscribers

Le Monde has implemented a strict policy that blocks almost all non-human traffic by default and is now exploring how to recognize paying readers who access its journalism via AI agents rather than a browser. CTO Paul Laleu said the publisher is evaluating early technical standards — such as Model Context Protocol (MCP) and OAuth-style extensions — so agents can assert a user’s subscriber status and the site can apply paywall rules. Le Monde has licensing deals with OpenAI, Meta and Perplexity and cites Fastly research showing AI-driven traffic growing far faster than human visits, prompting publishers to treat some automated access as a legitimate distribution channel.

Two governments, one message: publishers need more control

For decades, the web ran on an honor system. A file called robots.txt sat at the root of your site and would try to deter bots that were 'disallowed'. Bots that respected it were welcome, but bots that ignored it faced no consequences.