Does Scanverra validate my whole robots.txt file?

Yes, in two layers. The Website Audit's SEO score includes a real robots.txt syntax check - unrecognized directives, a bad HTTP status when fetching the file, and similar malformation issues. On top of that, two more targeted checks run: whether it blanket-blocks known AI crawlers, and whether any Disallow rule exposes sensitive-looking paths (like an admin or backup directory) to anyone who reads the file. What it doesn't do is check whether your Sitemap: directive points to a real, working sitemap.

Which AI crawlers does the blocked-crawler check look for?

Known AI agents including GPTBot, ClaudeBot, CCBot, Google-Extended, and PerplexityBot, among others. The check specifically looks for a blanket Disallow: / or Disallow: /* aimed at these user-agents, not narrower, deliberate restrictions on specific paths.

Why would blocking AI crawlers hurt my site?

As more search and discovery moves through AI assistants that browse or cite the web on a user's behalf, a site that blanket-blocks those crawlers becomes invisible to that entire channel - even if it ranks fine in traditional search. It's a visibility cost most teams don't realize they've taken on until it's pointed out.

Isn't listing sensitive paths in robots.txt supposed to hide them?

No - this is a common misunderstanding. robots.txt is a public file anyone can request at yoursite.com/robots.txt. A Disallow rule doesn't restrict access, it just asks well-behaved crawlers not to index the path - so listing /admin-backup/ there hands anyone reading the file a map of exactly where to look.

How do I actually keep AI crawlers out of specific paths without blocking them entirely?

Target the specific user-agent and path rather than a blanket rule - see the code example below. That way you can, for instance, keep GPTBot out of an internal search results path without blocking it from the rest of your site.

Fix robots.txt Blocking AI Crawlers (GPTBot, ClaudeBot)

robots.txt is a plain-text file at your domain's root that tells crawlers which paths they're allowed to request. A single overly broad Disallow rule - often copied from a staging environment or written to stop one bad bot - can silently block every AI crawler from your entire site, cutting you out of AI-powered search and assistant results without any warning.

Why This Happens

A staging Disallow: / makes it to production - a blanket block meant to keep a pre-launch site out of search engines never gets removed after launch.
A rule meant for one aggressive scraper gets applied to everything. A team blocks all crawlers to stop one bad actor, sweeping in every legitimate AI agent along with it.
Old blocklists predate today's AI crawlers. A robots.txt written years ago against a fixed list of "bad bots" now unintentionally includes newer AI user-agents nobody thought to add exceptions for.
Sensitive paths get listed in robots.txt as if it were access control. A team adds Disallow: /admin-backup/ intending to hide the directory, not realizing the file itself is public and now advertises exactly where to look.

How to Identify the Problem

Scanverra checks your robots.txt three ways, from general syntax down to two specific, AI-era failure modes:

Syntax and reachability. The Website Audit's SEO score includes Lighthouse's standard robots.txt check - it confirms the file is actually reachable (not a 4xx/5xx response) and that every directive it contains is one crawlers recognize, flagging malformed rules as an SEO issue.
Blanket AI-crawler blocking. A separate, Scanverra-specific check detects a blanket Disallow: / or Disallow: /* aimed at known AI crawlers - GPTBot, ClaudeBot, CCBot, Google-Extended, PerplexityBot, and similar - and raises it as an issue in your Website Audit's Agent Readiness score.
Sensitive-path exposure. A third check flags <code>Disallow</code> rules that point at sensitive-looking paths, like an admin or backup directory, since the rule itself reveals the path's existence to anyone who reads the file.

One real gap: none of the three confirms your Sitemap: directive actually points to a live, working sitemap - only that the line itself is syntactically recognized.

How to Fix It

1. Replace a blanket AI block with targeted rules, if you block at all

Decide deliberately which AI crawlers, if any, you actually want to keep out - and only restrict the specific paths that matter, not your entire site:

public/robots.txtBefore: blocks every AI crawler from everything

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: CCBot
Disallow: /

public/robots.txtAfter: allows AI crawlers, keeps them out of one internal path

User-agent: *
Allow: /
Disallow: /internal-search/

Sitemap: https://example.com/sitemap.xml

2. Never use robots.txt to hide sensitive paths

If a path genuinely needs to stay private, protect it with authentication or a noindex meta tag on the page itself - not a public robots.txt rule that just announces where it lives. Remove any Disallow entries that point at admin, backup, or staging directories.

3. Confirm the fix took effect

Re-run a Website Audit after editing robots.txt - the Agent Readiness score's "AI crawlers not blocked" check will reflect the change immediately since it reads the live file on every scan.

GPTBot, ClaudeBot, PerplexityBot and OAI-SearchBot: Which to Allow

When robots.txt is blocking AI crawlers, the fix depends on which ones. AI companies run separate bots for search, live user requests and model training, and each reads its own robots.txt group:

Token	Company	Purpose	Blocking it means
`OAI-SearchBot`	OpenAI	ChatGPT search	Your pages cannot be cited in ChatGPT search
`ChatGPT-User`	OpenAI	Fetches a page when a user asks	ChatGPT cannot open your page on request
`GPTBot`	OpenAI	Model training	Content is not used for training; no effect on citations
`Claude-SearchBot` / `Claude-User`	Anthropic	Claude search and browsing	Claude cannot cite or open your pages
`ClaudeBot`	Anthropic	Model training	No effect on Claude citations
`PerplexityBot` / `Perplexity-User`	Perplexity	Perplexity answers and browsing	Your pages cannot appear as Perplexity sources
`Google-Extended`	Google	Gemini training and grounding	No effect on Google Search or AI Overviews

A common middle ground is to allow the search and user bots so AI assistants can cite you, and block the training bots:

public/robots.txtAllow AI search bots, block AI training bots

User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Google-Extended
Disallow: /

User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

User-agent: *
Allow: /

Remember that a bot obeys only the most specific group that names it, so a User-agent: * block does not apply to a bot that has its own group. Read structured data for AI search for what to do once AI crawlers can reach your pages.

How Scanverra Detects This

The Website Audit's SEO score runs a standard robots.txt syntax and reachability check on every scan. On top of that, a dedicated AI-crawler check reads the same live file and looks specifically for a blanket-style Disallow aimed at known AI user-agents, surfacing it as part of the Agent Readiness score; a separate check in the security scan flags sensitive-looking paths named in Disallow rules. Three checks, each looking for a different failure mode.

How to Fix robots.txt Blocking AI Crawlers