Observed Signal · May 26, 2026 · Policy Update · Source: AdExchanger · Impact: 4/5 · Sentiment: Positive
Microsoft Urges Publishers Not to Block AI Crawlers
At AdExchanger’s Programmatic AI event in Las Vegas, Nikhil Kolar, VP of publisher product at Microsoft AI, advised publishers and retailers to allow AI bots to crawl their sites and to update robots.txt to avoid hiding content from agentic AI tools. Kolar said four out of five websites currently block AI crawlers, which prevents discovery and recommendations. Microsoft positions its Publisher Content Marketplace — announced in February and now open to other AI firms — as a licensing vehicle to ensure publishers receive payment when their content is used for model grounding. Jonathan Roberts of People Inc. described a more restrictive, permissioned approach: People Inc. permits around 38 crawlers while blocking tens of thousands daily as leverage in licensing negotiations. Microsoft said real-time inferences using publisher data will run on Azure.
Guidance and licensing from a major platform (Microsoft) affects publisher discovery, content monetization, AI grounding practices, and could set commercial and technical precedents across the open web.
Track Microsoft Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Nikhil Kolar, VP of publisher product at Microsoft AI, told publishers at Programmatic AI 2026 to allow AI bots to crawl sites and to update robots.txt.
- Kolar said four out of five websites currently block traffic from AI bots and crawlers.
- Microsoft announced its Publisher Content Marketplace in February to facilitate licensing between publishers and AI developers and has opened the marketplace to other AI firms.
- People Inc. (Jonathan Roberts, Chief Innovation Officer) said it allows access to about 38 crawlers and blocks between 30,000–35,000 crawlers per day while negotiating access.
- Microsoft distinguishes using publisher data for 'training' versus 'grounding' and said real-time AI inferences using licensed publisher data will run on Microsoft Azure.
Connected Companies & Entities
6 Entities mappedOntology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
Publishers Navigate AI Challenges Amid Declining Traffic
Publishers face challenges from generative AI as content discovery declines and traffic drops, with Cloudflare proposing to block AI bots by default and introduce a pay-per-crawl model to compensate publishers for access. While applauding the stance, Amanda Martin, Mediavine’s Chief Revenue Officer, cautions that Cloudflare’s approach may not be commercially viable in the long term. Mediavine, which provides ad management services to help creators monetize content, does not prescribe site rules but advises publishers to weigh AI usage carefully and pursue responsible practices. The AdExchanger Talks episode also covers misconceptions between advertisers and publishers and reasons why the industry is rethinking made-for-advertising sites. Overall, the discussion highlights AI’s mixed impact on content discoverability and revenue, and underscores the need for publishers to navigate rapid policy and technology shifts.
Ad Spend Steady Amid Economic Woes; Microsoft Innovates AI Licensing
AdExchanger’s Daily News Roundup highlights macro pressure on ad spending, a new Microsoft initiative to license publisher content for AI, and security problems at a bot-focused social site. The Economist piece questions Big Tech’s claim that ad revenue is decoupled from GDP, noting smaller advertisers remain vulnerable to cuts. Microsoft Advertising announced the Publisher Content Marketplace (PCM) to connect AI developers with rights-cleared content from opted-in publishers; Microsoft worked with Condé Nast, Hearst and Vox Media and built PCM with usage-based reporting and publisher-defined licensing. Separately, Moltbook — a social site for AI bots — exposed an API key vulnerability that reportedly revealed about 1.5 million bot passwords and other private data, and its OpenClaw software became a malware vector. Briefs cover Fubo’s $19.1M loss, NBCUniversal AI ad tests, SpaceX’s xAI acquisition valuation, and executive moves at Disney and Mediaocean.
Publishers Pull Back from Google AI Search
Publishers and creators are reacting differently to the rise of AI-driven search: several publishers and publisher-facing services are preparing to block or delist crawlers used for search indexing and AI training, while some creators are leaning into AI search exposure. Beginning September 15, Cloudflare’s site-security software will default to blocking such crawlers for new publishers and free-tier users; publishers using Cloudflare include Financial Times, Condé Nast and The Atlantic. USA Today and creator network Beehiiv told Adweek they were preparing to delist Google. Google has been increasing AI investment even as its search and ads arm still grows; the company reported roughly $5.9 billion negative free cash flow. Studies cited show publishers receive a small share of AI-referral traffic, and companies including OpenAI and Microsoft are licensing publisher content or building publisher marketplaces.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.