Observed Signal · Aug 8, 2026 · Technical Release · Source: DEV Community · Impact: 3/5 · Sentiment: Neutral

Firewalls determine AI crawler access — 18-site audit

Executive Signal Summary

An author built an open-source tool, geo-crawl-audit, to probe how major websites treat AI crawler user-agents and how much readable content exists in raw HTML before JavaScript runs. The probe (against 18 sites on August 7, 2026) found that many sites' firewall and bot-management rules — not robots.txt alone — determine which AI crawlers can fetch pages, that several prominent crawlers (e.g., GPTBot, ClaudeBot, PerplexityBot) do not execute JavaScript, and that some well-known sites either deliberately or inadvertently present almost-empty raw HTML to most AI crawlers. Five sites blocked the probe's baseline requests entirely, highlighting the difficulty of measuring crawler access from arbitrary networks. The author published the tool and a public scanner to help operators check AI readability of their domains.

Polaris7 AgentPolaris7 Strategic Assessment
High Confidence

The audit and tool highlight concrete, operational gates (WAF/bot management, TTFB, server-rendering, robots.txt) that determine AI visibility of web content — relevant to publishers, SEO/GEO practitioners, and any organization aiming to control or enable LLM access, but not an industry-shifting platform policy change.

MARKET SIGNALS

Track The Guardian Signals & Strategic Shifts

Polaris7 connects regulatory filings, primary sources, leadership changes, and M&A activity into structured intelligence. Set up your Polaris7 workspace to monitor relevant companies in your executive briefing.

  • Verified primary sources
  • Automated briefings
  • Instant watchlist setup

Key Takeaways & Evidence Grounding

  • The author published an open-source tool, geo-crawl-audit, that probes sites using the user-agents of major AI crawlers and measures raw-HTML word counts.
  • The tool was run against 18 major sites on August 7, 2026, producing a per-crawler status matrix and timing data.
  • Many AI crawlers (e.g., GPTBot, ClaudeBot, PerplexityBot) do not execute JavaScript; Googlebot and Applebot are among the few AI-adjacent crawlers that render JS.
  • Observed HTTP status patterns often aligned with public business relationships: e.g., The Guardian served GPTBot and related UAs a 200 while disallowing others in robots.txt; The New York Times returned 403s to most crawlers but permitted bingbot and Amazonbot.
  • Five sites (Quora, OpenAI, Perplexity, Bloomberg, GitHub) challenged or filtered the probe's baseline browser request, causing the tool to flag them as BASELINE_ANOMALY.

Connected Companies & Entities

“The Guardian — which has a content deal with OpenAI — serves my simulated GPTBot, OAI-SearchBot, and ChatGPT-User a clean 200.”

“The Guardian — which has a content deal with OpenAI — serves my simulated GPTBot, OAI-SearchBot, and ChatGPT-User a clean 200.”

“The New York Times — in litigation with OpenAI — 403s nearly everyone: GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot, Common Crawl, meta-e...”

“Reddit's robots.txt blocks every AI bot in my list — fourteen tokens, no exceptions.”

“Figma runs the inverse configuration: serves a 200 to every bot UA in the list, but robots-disallows GPTBot, OAI-SearchBot, ChatGPT-User, Cl...”

“Airbnb returns 403 to nine of the twelve AI user-agents I probe — every OpenAI, Perplexity, Microsoft, Amazon, Meta, and Common Crawl token ...”

“Airbnb returns 403 to nine of the twelve AI user-agents I probe ... while, curiously, all three Anthropic user-agents get a 200.”

“LinkedIn served my probe a 23-word bot-check interstitial — whatever a verified crawler negotiates, the raw HTML a stranger gets is effectiv...”

“Quora, OpenAI, Perplexity, Bloomberg, and GitHub challenged even my baseline browser request from my network.”

“Stripe (1,957 raw-HTML words, a 200 to every UA in my list) ... — full server-rendered HTML, fast, no bot differentials against my probe.”

“OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, and meta-externalagent all received a 200 from the same IP, seconds apart.”

“OAI-SearchBot, Claude-SearchBot, PerplexityBot, Amazonbot, and meta-externalagent all received a 200 from the same IP, seconds apart.”

“Quora, OpenAI, Perplexity, Bloomberg, and GitHub challenged even my baseline browser request from my network.”

“Stripe (1,957 raw-HTML words, a 200 to every UA in my list), Anthropic (fastest warm response in the test at 0.102s), Vercel, MDN, Shopify —...”

“Stripe (1,957 raw-HTML words, a 200 to every UA in my list), Anthropic (fastest warm response in the test at 0.102s), Vercel, MDN, Shopify —...”

“Quora, OpenAI, Perplexity, Bloomberg, and GitHub challenged even my baseline browser request from my network.”

“Quora, OpenAI, Perplexity, Bloomberg, and GitHub challenged even my baseline browser request from my network.”

Primary Source Grounding & Direct Attribution
Direct Origin Attribution
Primary Reporting: DEV Community•Published: Aug 8, 2026
Original Coverage Title: “Your firewall is your AI policy — I probed 18 major sites to read it”

Related Market Signals & Shifts

Recent verified developments and strategic activity across this market segment.

SEO, GEO & SEM PlatformJun 15, 2026

One-Call API to Check AI Search Visibility

A developer (Daniel Igel) published a small audit engine and web tool that scores whether a website can be crawled, parsed and cited by AI search assistants (e.g., ChatGPT Search, Perplexity, Google AI Overviews). The audit evaluates robots.txt access for AI crawlers (GPTBot, OAI-SearchBot, PerplexityBot, ClaudeBot), presence of llms.txt, JSON-LD structured data, and passage citability, producing a 0–100 score with per-category pass/warn/fail results and concrete recommendations. The API is keyless and callable via a POST to https://citeready-api.sprytools.com/v1/audit; a web UI at https://citeready.sprytools.com offers a free tier (three audits/day) without signup.

Read assessment
SEO & GEO PlatformApr 9, 2026

AI Can't Read Many Websites — How to Fix It

Many modern websites are effectively invisible to AI-powered crawlers because content is rendered client-side, lacks semantic HTML, or buries facts in heavy markup. The article explains how AI systems consume the DOM and recommends practical fixes: enable server-side rendering (SSR) or static generation, simplify the DOM for higher information density, use semantic HTML5, and publish machine-readable assets such as llms.txt, agents.json, and JSON-LD structured data (Schema.org). It provides a checklist—robots.txt, SSR, JSON-LD Organization/Service schemas, llms.txt, and active indexing (sitemaps, IndexNow)—to improve discovery by LLM-based services like ChatGPT, Perplexity, Claude, and Gemini.

Read assessment
SEO, GEO & SEM PlatformAug 13, 2026

GEO Is the New SEO — Most Sites Fail

The author built a Generative Engine Optimization (GEO) agent inside WebScore to measure how discoverable websites are to AI engines (e.g., ChatGPT, Perplexity, Gemini, Claude). The article identifies key signals that influence AI visibility: llms.txt, E-E-A-T signals (author attribution, About/Contact pages, organization schema), FAQPage JSON-LD, outbound authority citations, a Wikidata entry, and allowing AI crawlers in robots.txt. Scanning real sites, the author reports widespread gaps — about 80% lack llms.txt, ~70% lack FAQ schema, and ~60% unintentionally block at least one major AI crawler — and claims sites with Wikidata entries are cited roughly three times more often. WebScore returns metrics like AI Mention Rate and Citation Rate and an ordered issue list for fixes.

Read assessment

WATCHLIST & BRIEFINGS

Receive future signals for The Guardian & competitors in your briefing

Stay ahead of material M&A deals, leadership changes, account wins, and product rollouts. Polaris7 connects primary filings, company newsrooms, and trade reporting directly into your executive briefing.

  • Regular primary-source checks
  • Structured Watchlist briefings
  • Preserves current context