Is your site blocking AI crawlers?

How to use it

1. Type your domain and press check. Protocol and trailing paths are stripped for you, so https://yoursite.com/blog and yoursite.com do the same thing.
2. Read the verdict line first. It counts how many of the four major AI crawlers are shut out of your whole site. Zero is what you want.
3. Then read the cards. Each one names the engine, the crawler, and the exact rule in your file that produced the verdict.

It works on any public domain, including a competitor's. There is no signup, nothing is stored, and the result has a shareable link that re-runs the same check for whoever you send it to, which is the fastest way to hand a developer proof that a line needs changing.

How to read your result

Every crawler gets a badge and a one-line reason. The reason is the part that matters, because two crawlers can both be Allowed for completely different causes:

The verdict line at the top only counts the four major engines, because those are the ones that decide whether you can appear in a mainstream AI answer today. The four listed under “More AI crawlers” are shown with the same detail but do not affect the count.

The crawlers it checks

Eight crawler families, matched against the user-agent tokens each company publishes, case insensitively. The first four decide the verdict:

One caveat that catches almost everyone: Google AI Overviews are not controlled by Google-Extended. They are built on the ordinary Googlebot, so blocking Googlebot drops you out of AI Overviews as well as normal search, while Google-Extended only governs Gemini training and grounding. If your goal is AI Overviews specifically, the line to check is your Googlebot rule, and opting out of AI Overviews is a separate, narrower decision.

What this tool does not check

A clean result here means one specific thing: your robots.txt does not shut these crawlers out of your site. It does not mean nothing else can. These are the gaps, and we would rather you knew them than trusted a green badge too far:

• Firewall and bot-management rules are invisible to it. Plenty of sites allow a crawler in robots.txt and then block it at the network edge with a security or bot-protection rule. That block is real and this tool cannot see it. If your robots.txt is clean and you still never appear anywhere, check your server logs for the crawler names above and see whether they are getting a 403.
• It answers “is the whole site off limits”, not “is this one URL blocked”. Path-level rules are shown to you but not evaluated against a specific page. If you need to know whether one URL is reachable, read the listed paths yourself.
• It does not read page-level directives. A noindex tag, an X-Robots-Tag header, or a snippet control such as nosnippet lives in the page or the response, not in robots.txt, and none of them are covered here. Any of them can suppress you while robots.txt looks perfectly clean.
• Allowed does not mean visited. Nothing in a robots.txt causes a crawler to come. It only stops it being turned away.
• robots.txt is a request, not a lock. Well-behaved crawlers honour it. It has never been an enforcement mechanism, so a Blocked badge tells you what you have asked for, not what is guaranteed.
• The result can be up to ten minutes old (the response carries a ten-minute cache-control header, which you can check yourself). Responses are briefly cached, so if you have just edited your robots.txt and re-run the check immediately, you may see the previous version. Wait a few minutes and check again.

How to unblock AI crawlers in robots.txt

If the checker shows a block you did not intend, the fix is a few lines. Make sure no Disallow: / applies to the crawlers you want in:

User-agent: GPTBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

One thing to get right while you are in there. Naming a crawler explicitly means it stops following your catch-all group entirely, so any Disallow lines you keep under User-agent: * no longer apply to it. If you were blocking /admin or /checkout for everyone, repeat those lines inside each named group, or you have just opened folders you meant to keep shut.

Save the file at yoursite.com/robots.txt and re-run the check. Your live file updates immediately; how quickly each engine notices depends on when it next crawls you, usually within days. The longer walkthrough, with the full rule set and the mistakes we see most, is in allowing AI crawlers in robots.txt.

When to run this

• First, whenever a brand is missing from AI answers. It costs ten seconds and rules out the one cause that makes every other fix pointless. Do this before you rewrite a single page.
• After a migration, redesign or platform change. This is where blocks appear by accident. A staging site ships with Disallow: / still in place, or a new platform adds AI crawler blocks by default and nobody reads the changelog.
• After anyone touches robots.txt. Including plugins and security tools that write to it on your behalf.
• On your competitors. If a rival blocks the crawlers and you do not, that is a real and temporary advantage in your category, and it is worth knowing you have it.

Does allowing crawlers mean you get cited?

No, and that is the honest part. Letting the bots in is necessary, not sufficient. It removes a blocker; it does not earn a citation. What earns one is being genuinely worth quoting: a clear, self-contained answer near the top of the page, plus real mentions of your brand across sites the engines already trust. Blocking guarantees invisibility, but unblocking guarantees nothing. Once you are reachable, the next two moves are checking whether your best page is actually shaped to be quoted, with the AI Readiness Checker, and then improving your brand visibility in AI search.

Questions people ask

Does it really read my live file?
Yes. A small function on our side fetches yoursite.com/robots.txt and returns the text; your browser does the parsing. We fetch it server-side only because a browser cannot request another site's file directly. There is no model involved and no estimate anywhere in the result.

It could not fetch my file. What now?
Some servers refuse our request. When that happens the tool offers a box where you can paste your robots.txt, and the analysis runs identically on what you paste. It also links straight to your file so you can copy it.

Should I block any AI crawlers?
Only with a clear reason, such as selling your content and not wanting it in training sets. Note that the training and answering crawlers are increasingly separate: you can block training while staying quotable in live answers, if you set the right user-agents. For most brands that want to be discovered, allowing the major assistants is the right call.

Is blocking Google-Extended how I stay out of AI Overviews?
No. Google-Extended covers Gemini training and grounding. AI Overviews run on Googlebot, so the only robots.txt lever there also removes you from regular Google search, which is almost never a trade worth making.

I have no robots.txt at all. Is that bad?
Not for AI access. Missing means nothing is blocked. You may still want one for control over specific folders, but its absence will never be why you are invisible.

Does an llms.txt file do the same job?
No, and confusing the two is common. robots.txt controls access and is genuinely honoured. llms.txt is an unofficial suggestion about reading order that Google has said it ignores. We wrote up the evidence in our honest take on llms.txt.

Is anything stored?
No. The domain you type is passed to the fetch and nothing else. No account, no log, no database.

The other three tools

AI Readiness Checker scores one live page out of 100 on the seven signals that make it quotable. Prompt Generator builds the buyer questions to paste into ChatGPT and Perplexity so you can see where you are missing. llms.txt Generator builds a curated index of your best pages. All free, all on our tools page.