Is Your Website Invisible to AI Search Engines?
Enter any domain to instantly audit which AI crawlers can — and can’t — access your content. See your AEO Visibility Score in seconds.
Checks GPTBot, ClaudeBot, PerplexityBot, Google-Extended & 17 more AI crawlers
Fetching robots.txt from …
Could not reach site
Check the domain is correct and the site is live.
AEO Visibility Score
Analyzing AI crawler access…
AI Crawler Access Breakdown
Required Fixes
0 blockedHow the Audit Works
Three stages, under two seconds. Your robots.txt is fetched live — no stored data, no account required.
Live Fetch
Your domain’s robots.txt is retrieved in real time via a CORS proxy chain. If your server responds with a 404, the tool correctly treats all bots as allowed by default.
Block-Based Parse
A standards-compliant parser reads every User-agent block, applies wildcard inheritance, and classifies each of the 21 bots using exact path matching — not pattern guessing.
AEO Visibility Score
Score is calculated from the 12 critical AI answer engine bots only. Admin-path Disallow rules (such as /wp-admin/) are excluded and do not penalise your score.
Bot Tiers
The AI answer engines that power ChatGPT, Claude, Perplexity, Gemini, Apple Intelligence, and DuckDuckGo AI. Blocking any of these directly reduces AI citation eligibility and counts against the AEO Visibility Score.
GPTBot · OAI-SearchBot · ChatGPT-User · ClaudeBot · anthropic-ai · Claude-Web · PerplexityBot · Google-Extended · Google-CloudVertex · Applebot-Extended · DuckAssistBot · MistralAI-User
Secondary AI platforms and AI-powered discovery engines. These are audited and displayed but do not affect the AEO Visibility Score calculation.
Bytespider · DeepSeek · LinerBot · QualifiedBot · ICC-Crawler
Traditional search engine crawlers. Essential for SEO but not included in the AEO Visibility Score, which focuses specifically on AI answer engines.
Googlebot · bingbot · Applebot · YandexBot
Score Interpretation
| Score | Grade | What it means |
|---|---|---|
| 80–100 | ✅ AI-Friendly | All or nearly all critical AI engines can access your content. Maximum citation eligibility across ChatGPT, Claude, Perplexity, and Gemini. |
| 50–79 | ⚠️ Partially Visible | Some critical bots are blocked. AI citation rate is reduced. Check the Fix Panel for specific bots and one-click fix code. |
| 0–49 | ❌ AI-Invisible | Most or all critical crawlers are blocked. Near-zero AI search visibility. Immediate action required — use the Fix Panel above. |
Robots.txt and AI Search Visibility: What Marketers Need to Know in 2026
A single two-line configuration in your robots.txt can silently exclude your entire site from every AI-generated answer engine on the internet. In 2026, as ChatGPT, Perplexity, Claude, and Google’s AI Overviews collectively mediate a growing share of all content discovery, this is no longer a technical footnote — it is a core AEO and GEO concern. Yet the majority of site owners have never audited which AI crawlers their robots.txt actually permits.
Why AI Crawlers Behave Differently From Traditional Search Bots
Traditional search crawlers like Googlebot operate with a degree of resilience. They discover URLs through external backlinks and XML sitemaps, and a temporary robots.txt block may only delay indexing rather than permanently exclude content. Google’s index can partially recover from a misconfiguration because it has multiple discovery pathways.
AI answer engines work on a fundamentally different model. When GPTBot or PerplexityBot encounters a Disallow: / directive, they do not add your URLs to a retry queue or fall back to backlink discovery. They exclude your domain from their retrieval pool entirely. For systems like ChatGPT’s web search and Perplexity’s real-time indexing, crawl access is binary: your content is either retrievable or it does not exist.
Google can partially recover from a crawl block because it discovers URLs through external sources. AI retrieval systems have no equivalent fallback. A robots.txt block means your content is permanently absent from AI-generated answers — regardless of how well it ranks in traditional search.
The Nuclear Block: How One Directive Disqualifies Your Entire Site
The most common cause of total AI search invisibility is the nuclear block pattern — a combination that originated as a development environment safeguard and became one of the most consequential misconfigurations in AI-era SEO:
Disallow: /
When WordPress sites enter maintenance mode, when developers lock down staging environments, or when site owners follow outdated SEO guides that recommend blocking all bots to prevent duplicate content penalties, this pattern gets added. Developers mean to remove it before launch. They frequently do not.
The outcome in 2026 is severe: User-agent: * catches every bot not specifically allowed, and the AI crawlers that power ChatGPT, Claude, Perplexity, and Gemini all fall into that wildcard bucket unless they have explicit override rules. A site with this configuration has a score of 0/100 on this auditor — and effectively zero presence in AI-generated search results.
The fix is two lines per bot, added before the wildcard block. Specific User-agent rules always take precedence over the wildcard:
User-agent: GPTBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# Your existing wildcard block stays untouched
User-agent: *
Disallow: /
Why Blocking /wp-admin/ Is Not the Same as Blocking AI Crawlers
One of the most misunderstood aspects of robots.txt scoring is the treatment of admin and system paths. A site with this configuration:
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
is not blocking AI crawlers from its content. It is correctly protecting a WordPress admin area that no legitimate bot — AI or otherwise — has any reason to access. This auditor distinguishes between these two cases. The admin-path whitelist includes /wp-admin/, /wp-login.php, /admin/, /login/, /cgi-bin/, /phpmyadmin/, and a dozen other system-only paths. Disallow rules that exclusively target these paths are treated as correct security hygiene and do not reduce your AEO Visibility Score.
The distinction matters because many security-hardening guides recommend adding Disallow: /wp-admin/ as a standard step. Sites that follow this advice should not be penalised for it — and with this auditor, they are not.
The Correct robots.txt Structure for Maximum AI Visibility
A well-structured robots.txt for AI-era search opens access for AI crawlers explicitly, then restricts only the paths that genuinely need protection. The pattern that achieves this without disrupting existing SEO:
User-agent: GPTBot
Allow: /
User-agent: OAI-SearchBot
Allow: /
User-agent: ClaudeBot
Allow: /
User-agent: PerplexityBot
Allow: /
User-agent: Google-Extended
Allow: /
# ── Restrict only genuinely private paths ──
User-agent: *
Disallow: /wp-admin/
Disallow: /checkout/
Disallow: /account/
Allow: /wp-admin/admin-ajax.php
This structure scores 100/100 on this auditor, maintains all necessary access restrictions, and does not require any changes to your existing SEO configuration.
How to Use This Tool as Part of Your AEO Workflow
The robots.txt audit is one diagnostic in a broader AEO readiness picture. A site can have a perfect robots.txt score and still score poorly on AEO readiness if it lacks structured data, author attribution, or question-based content. The recommended workflow is:
A page that clears this robots.txt audit and implements the schema from the generator tools is as well-positioned for AI citation as it can be without changes to the content itself.
Frequently Asked Questions
What is a robots.txt AI bot auditor?
A robots.txt AI bot auditor checks whether your site’s robots.txt file is blocking AI crawlers such as GPTBot (ChatGPT), ClaudeBot (Anthropic/Claude), PerplexityBot, and Google-Extended (Gemini/AI Overviews). It parses your live robots.txt, classifies each of the 21 audited bots as Allowed, Partial, Blocked, or Not Set, and calculates an AEO Visibility Score showing how visible your site is to AI search engines.
How is the AEO Visibility Score calculated?
The score is calculated from the 12 critical AI answer engine bots only. The formula is: (number of critical bots Allowed or Not Set ÷ 12) × 100. Admin-path Disallow rules — such as Disallow: /wp-admin/ — are excluded from scoring. A site that blocks only admin paths but allows all AI crawlers at the root level will score 100/100.
What does “Not Set” mean for a bot’s status?
Not Set means the bot has no specific User-agent block in your robots.txt and there is no wildcard User-agent: * block either. Per the robots.txt specification, the absence of any directive means full access is granted by default. Not Set bots are therefore treated as Allowed and count toward a positive AEO Visibility Score.
Does blocking /wp-admin/ affect my AI visibility score?
No. This auditor uses an admin-path whitelist that includes /wp-admin/, /wp-login.php, /wp-includes/, /admin/, /login/, /dashboard/, /cgi-bin/, /phpmyadmin/, and other system-only paths. Disallow rules that exclusively target paths on this list are treated as correct security hygiene and do not reduce your score. A site with only Disallow: /wp-admin/ in its robots.txt will score 100/100.
What is a “nuclear block” and how do I fix it?
A nuclear block is the combination of User-agent: * and Disallow: / with no Allow: / override. It blocks every crawler — including all AI engines — from accessing any page on your site. It is the most common cause of a 0/100 score.
The fix is to add explicit Allow: / rules for each AI bot before your wildcard block. Specific User-agent rules always take precedence over the wildcard. Use the Fix Panel above to generate and copy the exact code for your blocked bots.
Will allowing AI crawlers affect my Google rankings?
No. Allowing AI crawlers such as GPTBot, ClaudeBot, or PerplexityBot has no impact on how Googlebot crawls or indexes your site. AI bots and search bots are distinct crawler identities that operate independently. Adding explicit Allow: / rules for AI bots in no way changes your Googlebot configuration or your Google rankings.
Why can’t the auditor fetch some domains?
The auditor fetches robots.txt via two public CORS proxy services. Sites protected by Cloudflare, Akamai, or similar bot-mitigation systems often block proxy IP ranges, returning a 403 Forbidden error. Sites on very slow servers that time out before the 8-second limit will also fail. If your domain cannot be fetched via proxy, check your robots.txt directly by visiting yourdomain.com/robots.txt in a browser.
Sources & References
- OpenAI — GPTBot documentation. Official specification for the GPTBot and OAI-SearchBot user-agents and how to allow or disallow them in robots.txt.
- Anthropic — ClaudeBot robots.txt guidance. Official documentation on the ClaudeBot and anthropic-ai user-agents and robots.txt compliance.
- Google Developers — Google crawlers overview. Documentation covering Google-Extended, Google-CloudVertex, and how they interact with robots.txt directives.
- RFC 9309 — Robots Exclusion Protocol. The official IETF specification for the robots.txt standard, including User-agent precedence and Disallow/Allow parsing rules.