Observed Signal · Jul 10, 2026 · Technical Guidance · Source: DEV Community · Impact: 3/5 · Sentiment: Positive
Let AI Crawlers In: AEO Not SEO in 2026
The article argues that 'AEO' (Answer Engine Optimization) is a distinct discipline from classic SEO and that publishers who want to be cited by AI assistants should allow AI crawlers to index their pages. It explains that AI crawlers are vendor- and task-specific (training, search-indexing, live fetch) and lists explicit user-agent names for OpenAI, Anthropic, Perplexity and Google. The piece reminds readers that robots.txt (RFC 9309, 2022) is a voluntary request, not an enforcement mechanism, and that hard blocks require server/CDN/WAF rules. Recommended practical steps include per-bot robots.txt entries, server-rendered HTML, structured data (schema.org), adopting the emerging llms.txt convention, and ensuring CDN bot‑management aligns with robots.txt.
Provides actionable, technical guidance for publishers and marketers about AI-driven discovery and citation (AEO), clarifies per-bot controls and robots.txt limits, and highlights emerging conventions (llms.txt) that affect content discoverability for AI assistants.
Track OpenAI Signals & Market Shifts in Real-Time
Polaris7 autonomous intelligence agents track regulatory filings, primary sources, executive changes, and deal flow 24/7. Create your free Explorer workspace to monitor these entities.
Key Takeaways & Evidence Grounding
- Ben Santora published the article on 2026-07-10.
- The author defines AEO (Answer Engine Optimization) as distinct from classic SEO and argues sites should allow AI crawlers if they want to be cited by AI assistants.
- The article lists specific AI crawler user-agents (e.g., GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User, Google-Extended).
- robots.txt is voluntary (RFC 9309, 2022) and does not technically block access; enforcement requires server/CDN/WAF-level rules.
- llms.txt is an emerging, non-standard convention proposed to surface important site content to LLMs and had around 10% domain adoption as of mid-2026.
Connected Companies & Entities
5 Entities mapped“OpenAI: `GPTBot` (training), `OAI-SearchBot` (powers ChatGPT Search results), `ChatGPT-User` (live fetch when a user clicks a citation)...”
“Anthropic: `ClaudeBot` (training), `Claude-SearchBot` (search indexing), `Claude-User` (fetches a page at a user's direct request)...”
“Perplexity: `PerplexityBot` (its search index crawler), `Perplexity-User` (user-triggered fetches)...”
“Google: `Google-Extended` (opts you in/out of Gemini training specifically — separate from `Googlebot`, which handles classic Search and isn...”
“Perplexity has been caught running undeclared crawlers that rotate user-agents, IPs, and ASNs specifically to evade no-crawl directives, and...”
Ontology Mapping & Concepts
Related Market Signals & Shifts
Recent verified developments and strategic activity across this market segment.
AI Citations (AEO) Required for Traffic in 2026
The article introduces Answer Engine Optimization (AEO), a content strategy aimed at getting generative AI and chat-based answer engines to cite a publisher's content. Citing a SE Ranking study of 2.3 million pages, the piece reports that AI citation patterns diverge from traditional SEO signals: AI now cites fewer Google top-10 pages, favors high domain traffic and fresh, well-structured content, and weights backlinks differently across AI platforms (e.g., ChatGPT vs. Google AI Mode). The author lists practical AEO tactics — FAQ schema, paragraph-length control, frequent updates, multilingual content, and authoritative citations — and notes typical timing for visible impact (initial: 4–8 weeks; meaningful: 3–6 months). The article frames AEO as an evolution of SEO rather than a replacement.
Architecting Websites for the AI Web
The article argues that traditional SEO focused on ranking in ten blue links is no longer sufficient as users increasingly rely on LLM-powered search (ChatGPT, Claude, Perplexity) and autonomous agents. It proposes a new discoverability stack built around three pillars: CRO (Conversion Rate Optimization) for humans, GEO (Generative Engine Optimization) for AI search, and ASO (Agentic Search Optimization) for autonomous agents. Practical recommendations include semantic HTML, comprehensive JSON-LD structured data, explicit self-contained statements for LLM citation, machine-readable application state, ARIA and standard form attributes for predictable agent interaction, and verifiable metadata. The author notes that low-code AI tools make implementation easier and promotes a commercial audit platform, Greater Than Services, which analyzes sites against the three pillars. Publication date: 2026-06-22.
How to Make AI Bots Promote Your Site
A developer observed multiple AI crawlers (e.g., GPTBot, ClaudeBot, PerplexityBot) visiting his technical blog and argues publishers should optimise to be cited by AI systems rather than simply blocking them. He defines two crawler categories — training crawlers that collect data for model training and answer engines that fetch live content and cite sources — and coins the term GEO (Generative Engine Optimization) for structuring content to influence AI-generated answers. The article outlines four practical tactics: publish an llms.txt file to declare authorship and citation preferences, write posts that preserve the author’s identity so summaries include the name, own a narrowly defined microniche to become the default citation, and treat Perplexity as a distinct traffic channel by using clear headings, FAQ schema, and Article/Person JSON‑LD. The author gives a recommended priority order and timelines for implementing these tactics.
Track Real-Time Market Signals & Shifts
Set up custom watchlists to receive automated, evidence-grounded executive digests whenever material signals or shifts occur across your tracked landscape.