Observed Signal · May 27, 2026 · Policy Update · Source: AdExchanger · Impact: 3/5 · Sentiment: Positive
IAB Tech Lab Urges Content Owners to Measure Bot Costs
IAB Tech Lab published draft guidance for bot and crawler management to help publishers, content owners and ad tech firms decide how to manage non-human traffic and AI systems that scrape or access content. The document, which complements the organisation’s CoMP API V1 framework, outlines multiple management approaches, their operational and commercial trade-offs, and warns that blanket bot blocking is increasingly impractical. The draft is aimed at business leaders rather than engineers and includes recommendations to quantify the costs and value exchange of crawlers. IAB Tech Lab opened the guidance for public comment through June 26, 2026, and said it will work with members and industry participants to support implementation. IAB executives Anthony Katsur (CEO) and Shailley Singh (EVP & COO) are quoted emphasizing clarity and practical adoption.
Guidance from IAB Tech Lab affects publisher revenue and programmatic validation practices by recommending measurable bot-cost reporting and selective crawler handling; relevant to publisher monetization and ad-quality operations but not a platform-level mandate.
Track SEMrush Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- IAB Tech Lab released draft guidance on bot and crawler management.
- The guidance is open for public comment through 2026-06-26.
- The document is intended for publishers, content owners, and ad tech companies and focuses on operational, financial, and strategic trade-offs in managing non-human traffic.
- The guidance complements IAB Tech Lab’s CoMP API V1 framework for communication and permissions between AI systems and publishers.
- IAB Tech Lab said it will work with members and industry participants to support implementation of the guidance and CoMP API.
Connected Companies & Entities
4 Entities mappedOntology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
New Protocols Aim to Monetize AI Content Usage
IAB Tech Lab has published version 1.0 of its Content Monetization Protocols (CoMP) specification for public comment, proposing a standardized way for publishers to signal permissions and commercial terms to AI systems before content is crawled or used. CoMP aims to create interoperable payment and attribution rails so publishers can monetize AI access without relying on bespoke licensing deals, while assuming existing edge-level access controls (paywalls, robots.txt) remain in place. IAB Tech Lab CEO Anthony Katsur framed the protocol as a mechanism to support compensation, accountability, and long-term sustainability for high-quality content. Bertelsmann’s Achim Schlosser endorsed the need for scalable compensation and visibility for content usage. The proposal is positioned as important to publishers, AI companies, and marketers; the public comment period runs through April 2, 2026.
U.S. Bill Targets Stealth Web-Scraping Bots
A bipartisan bill introduced in the U.S. House would require so-called “stealth” or AI web crawlers to disclose their identity and purpose to site hosts. The Stealth Bot Prohibition Act, authored by the News/Media Alliance and sponsored by Reps. Valerie Foushee, Laurel Lee and Gus Bilirakis, would empower enforcement by the Federal Trade Commission and state attorneys general and impose fines of $53,000 per violation. Publishers and trade groups say clandestine bots scrape and resell editorial content, drain hosting resources, reduce publisher revenues and feed training data to AI systems. Media companies including Axel Springer and USA TODAY Co. back the measure, citing real costs (e.g., Politico dedicating ~25% of hosting costs to bot management) and national-security concerns tied to large-scale scraping activity.
Search Engines Outpace AI Chatbots by 20x
An analysis by data provider OneLittleWeb of traffic across 10,171 websites in June 2026 shows that search engines still dominate website visits, drawing 162.19 billion visits (57.2% of tracked traffic) compared to just 8.26 billion for AI chatbots (2.9%). This makes search engines nearly 20 times more popular than all AI chatbots combined. Google alone recorded 98.19 billion visits, while ChatGPT had 5.32 billion, holding 64% of the chatbot segment. Both categories saw declines from the previous month, with chatbots dropping faster. The study, based on browser visits estimated from Semrush and Ahrefs, excludes app usage, which may undercount AI assistant usage. As Google integrates AI features into search, the line between search and chat continues to blur, but search engines remain the primary gateway to the web.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.