How to Allow AI Crawlers to Access Your Site (robots.txt Guide)

By Minel Gunesoglu, founder of Is My Brand in AI · Last updated August 1, 2026

TL;DR: To allow AI crawlers, add explicit Allow rules in robots.txt using each bot's exact user- agent: GPTBot and OAI-SearchBot (OpenAI), ClaudeBot (Anthropic), PerplexityBot, and so on. Case and spelling matter, and a CDN can still block a crawler robots.txt allows — verify what the bot actually sees.

Most guides on this topic teach you how to block AI crawlers. That made sense during the 2025 scraping panic, when half the web rushed to wall off GPTBot. But if you want your brand to show up when someone asks ChatGPT, Claude, or Perplexity a question, the goal is the opposite: you want to allow AI crawlers to reach your pages, read them cleanly, and carry your content into the answer. This guide is the allow-side manual. It covers which bots to let in, the exact robots.txt syntax that actually works, the one trap that silently undoes all of it, and how to confirm a crawler can really get through.

I run a small lab on this site. I built the free AI bot checker linked below, I run IndexNow live on ismybrandinai.com, and I keep my own robots.txt deliberately open to the engines I want citing me. Everything here is the order I work in on my own site, not theory from a deck. And I will be honest throughout about what robots.txt can and cannot do, because this is a topic where confident-sounding advice often gets the mechanics wrong.

Why Allowing AI Crawlers Matters Now

The plumbing of search changed underneath a lot of site owners without them touching a thing. AI answer engines like ChatGPT, Perplexity, and Google's AI surfaces increasingly read content through their own dedicated crawlers, then quote a passage straight into the answer a user reads. If your robots.txt turns one of those crawlers away, you are not just opting out of scraping. You are removing yourself from the pool of sources that engine can cite. No access, no citation.

Two things made this urgent. First, a wave of defensive blocking. When AI scraping became a headline in 2025, a lot of sites added blanket disallows, and many CDNs and hosting platforms shipped default rules or one-click "block AI bots" toggles. Plenty of those blocks are still in place, quietly, on sites whose owners now want AI visibility and do not realize they are walled off. Second, the engines split their crawlers into separate jobs, which means a single old block can hit the exact bot that feeds live answers while leaving the harmless one untouched. The result is a lot of sites that think they are open and are not.

The fix is not hard once you see the moving parts. The hard part is that the file you would read to check your own status can be telling you a comforting lie, which is the whole reason a live check beats eyeballing the text. I will come back to that. First, the distinction that everything else hangs on.

Training Crawlers vs Search Crawlers: The Core Distinction

This is the single idea that makes the rest of the page make sense, and it is the one most "block AI bots" guides flatten into mush. The major AI companies do not run one crawler. They run at least two, with different jobs, and you can treat them differently.

A training crawler collects content to help build and improve future versions of a model. Its visits do not directly produce a citation today. OpenAI's GPTBot and Anthropic's ClaudeBot fall here. Google-Extended is the same category for Google's generative models. Blocking a training crawler is a reasonable choice if your only concern is "I do not want my work used to train models," and it carries almost no cost to your live AI search visibility.

A search crawler (sometimes called a retrieval or grounding crawler) fetches pages so the engine can find, index, and cite them inside answers, right now. OpenAI's OAI-SearchBot and Perplexity's PerplexityBot fall here. These are the bots that decide whether you can appear when a user asks a live question. If AI search visibility is your goal, these are the crawlers you must let in.

There is a third, smaller category worth naming: the user-action fetcher. When a person pastes your URL into ChatGPT or clicks a Perplexity citation, the engine fetches that one page on demand. OpenAI's ChatGPT-User and Perplexity-User do this. They are not bulk crawlers, they act on a real user's request, and blocking them can break the experience of someone who deliberately handed your link to the assistant.

The practical upshot: "should I allow AI crawlers?" is not one question. You can block the training bots to keep your content out of model training and still allow the search and user bots so you stay citable in live answers. Or you can open everything. What you should not do is block by accident and lose the search bots without meaning to, which is exactly what a blanket disallow does.

The Full AI Crawler Reference Table

Here is every major AI crawler you need to know, what its operator uses it for, and whether to allow it if your goal is AI search visibility. The "user-agent token" column is the string you actually write in robots.txt. Get that token exactly right, because robots.txt matches it as written. Each operator publishes its own crawler list (OpenAI, Anthropic at its support and crawling docs, Perplexity, and Cloudflare's public bot directory), and those first-party pages are the source of truth as tokens change.

A few rows deserve a sentence. Google-Extended is the one people misread most: it controls whether Google can use your content to train its generative models, and it does not control whether you appear in Google's AI Overviews. Blocking it will not pull you out of those AI answers, a confusion I see constantly; the full breakdown of that mistake and the right tool for it lives in how to opt out of Google AI Overviews. Bingbot earns a place here because Microsoft Copilot grounds on the Bing index, so a Bing block quietly costs you Copilot citations; the deeper Copilot ranking story is in how to rank on Microsoft Copilot. And Bytespider is the row to watch, because its compliance history has been the least consistent of the group.

See Which AI Crawlers Your Site Actually Allows

It is the fastest way to see what your file says. A common find is a leftover disallow from a template or an old policy that nobody revisited. Knowing your starting point turns the rest of this guide from abstract into a two-minute edit.

The robots.txt Allow Template (Copy, Paste, Adjust)

Here is a clean robots.txt block that allows the AI search and user crawlers you most likely want, while leaving you free to decide on the training bots. Drop it into the robots.txt file at your domain root (yoursite.com/robots.txt), adjust to taste, and host it as plain text.

# Allow OpenAI's search + user crawlers (citation + on-demand fetch)
User-agent: OAI-SearchBot
Allow: /

User-agent: ChatGPT-User
Allow: /

# Allow OpenAI's training crawler too, only if you're fine with training use
User-agent: GPTBot
Allow: /

# Allow Anthropic's two visibility crawlers + (optionally) its training crawler
User-agent: Claude-User
Allow: /

User-agent: Claude-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

# Allow Perplexity's crawler + user fetch
User-agent: PerplexityBot
Allow: /

User-agent: Perplexity-User
Allow: /

# Allow Bing (feeds Microsoft Copilot)
User-agent: Bingbot
Allow: /

# Keep a catch-all so normal crawling isn't accidentally disallowed
User-agent: *
Allow: /

Sitemap: https://yoursite.com/sitemap.xml

Three things make or break this file, and they are the errors I see most:

The token must be exact. robots.txt matches the user-agent token as a string. GPTBot works; gptbot, GPT-Bot, GPTbot, or pasting the bot's full browser-style user-agent string into the User-agent: line does not. If you want to allow OpenAI's training crawler, the line is literally User-agent: GPTBot and nothing else. A wrong case or an extra hyphen means your rule silently matches nothing, and the bot falls through to whatever your catch-all says.

An Allow without a Disallow is the default anyway, so most "allow" work is really removing a block. If your file has no rule that disallows a bot, that bot is already allowed. The reason to write explicit Allow: / lines is clarity and to override a broader disallow. So the real fix on most sites is to find and delete the line that says User-agent: GPTBot followed by Disallow: / (or a blanket User-agent: * / Disallow: /), not to add anything.

Order and specificity matter. A more specific user-agent block wins over the catch-all for that bot. If you keep a restrictive User-agent: * rule, make sure each AI bot you want has its own explicit block, or it will inherit the catch-all's disallow.

If you also want to hand AI models a clean map of your best pages, that is a different file with a different job. robots.txt is the access gate; llms.txt is a content map, advisory, not an access control. Different files, different jobs. Our llms.txt explainer covers when that one is worth the effort, and our free llms.txt generator builds you a valid one in a couple of minutes.

Your robots.txt May Be Lying to You

This is the part that justifies a live checker over a careful read of your file, and it is the gap almost no competitor guide closes. Your robots.txt can say "come on in" while something above it slams the door, and you will never see it by reading the file.

Here is the mechanism. A request from an AI crawler hits your CDN or web application firewall before it ever reaches the robots.txt your server would serve. Many of those layers ship with bot-management rules, and several now include a one-click "block AI bots" or "block AI scrapers" control. Some platforms turned versions of this on by default. When that layer recognizes a crawler's signature, it can return a block, a challenge page, or a 403 directly, and your origin's robots.txt never gets a vote. The crawler experiences a closed door; your file, read by a human, looks wide open.

So you can have a perfectly permissive robots.txt and still be invisible to GPTBot or PerplexityBot because a firewall rule, a hosting default, or a security plugin is intercepting the request upstream. This is the single most common reason "but my robots.txt allows it" turns out to be wrong. Reading the file tells you your intent. It does not tell you what the bot actually receives.

That is exactly why our AI bot checker is built around what the request really resolves to rather than just parsing your robots.txt text, and why I do not trust a clean-looking file on its own. If you suspect an upstream block, the place to look after the checker is your CDN or firewall's bot-management settings, plus any security plugin on your CMS, where a "block AI bots" toggle may be doing the damage your robots.txt would never reveal.

Cloudflare Changed AI Crawler Defaults on September 15, 2026

The section above describes the mechanism in general. Here is the specific, dated instance of it, and it is the largest one this year. The date has passed, so read this as a check to run on your own zone rather than a deadline to meet.

Cloudflare now sorts AI traffic into three buckets rather than one: Search (crawlers that index you and can send referrals back), Agent (automation acting in real time for a person, like a chat fetch), and Training (crawlers taking content to train a model). On September 15, 2026, the defaults attached to those buckets changed.

In Cloudflare's own words, for new domains: "the defaults will be set to allow for search but block training and agent use for pages with ads." The reasoning is that an ad on a page signals a website owner meant a human to land there.

Three details decide whether this reaches you.

It is not only new domains. Cloudflare's announcement is explicit that the change also lands on "all existing free customers that have not changed their settings by September 15, 2026 in their dashboard," alongside new customers and new sites added by existing ones. Paid plans keep their current settings. If you are on Cloudflare's free tier and never opened those AI settings before that date, this one landed on you.

It only bites on pages that display ads. No ads, no default block. If you do not monetise with advertising, the new default leaves you alone — though the settings still exist and can still be switched on by hand or by someone else on your team.

The Googlebot trap is the part to actually worry about. Some crawlers do more than one job, and Cloudflare applies the most restrictive rule that matches. Its announcement names the consequence directly: "Multi-purpose crawlers such as Googlebot, Applebot, and BingBot will be blocked by customers who have selected to block Training." Cloudflare's own figure, from its 2026 bot report, is that mixed-use crawlers represent over 36% of crawler activity. So a decision that feels like "I don't want my content training a model" can, on an ad-carrying page, take Google's search crawler down with it. That is a search-traffic problem wearing an AI-policy costume.

What to do now that the date has passed

1. Check whether you are actually on the free tier, and whether any page you care about carries ads. Both conditions have to be true for the default to reach you.
2. Open your zone's Security settings and look at the AI traffic controls. Cloudflare said every customer could opt out of the new defaults at any point before September 15, and the settings stay yours to change afterwards. What matters now is what they read today, not what you remember choosing.
3. Decide Training separately from Search. If you want to refuse model training, fine — but understand that on ad pages the multi-purpose rule can extend that refusal to Googlebot and Bingbot. Blocking Training and keeping Google is not something you can assume; it is something you have to verify.
4. Verify at the request level, not in the dashboard. A setting that reads correctly can still resolve to a 403. Run the AI bot checker against your own domain and see what the crawlers actually get.

One first-hand note, because it is the reason I take this seriously: on another site I run, Cloudflare had already injected a managed robots.txt block that disallowed GPTBot and ClaudeBot and declared ai-train=no, sitting directly above the permissive rules I had written myself. Same file, two sets of instructions, opposite meanings — and I had not turned any of it on deliberately. Reading my own file would never have told me. That is the failure mode this whole page exists to catch, and the September 15 change makes it more common, not less.

The Honest Tradeoff: What Allowing and Blocking Really Do

I want to be straight about the limits, because the topic is full of overconfident claims in both directions.

Blocking a training bot does not remove you from ChatGPT. This is the big one. If you block GPTBot, you are opting out of OpenAI's training crawl going forward. You are not deleting yourself from ChatGPT, and you are not blocking the search crawler that handles live citations, which is a separate token (OAI-SearchBot). Content already learned during prior training does not un-learn itself because you added a line today. So "I blocked GPTBot, why does ChatGPT still know me?" has a clean answer: you blocked the wrong half, and training is not the same as live retrieval. The same logic holds for ClaudeBot and Claude's search crawler.

Allowing a crawler is necessary but not sufficient. Letting a search bot in puts you in the pool of pages it can cite. It does not guarantee a citation any more than letting Googlebot crawl you guarantees a number-one ranking. Access is the entry ticket; whether you actually get quoted depends on whether your page answers the question cleanly. The on-page craft of being citable, not just crawlable, is its own subject, covered in how to get cited by ChatGPT and across the broader picture in AI search visibility.

robots.txt is a request, not a wall, and not every bot honors it equally. The major operators state that their crawlers respect robots.txt, and the well-behaved ones do. But robots.txt is a convention, not an enforcement mechanism. Less scrupulous crawlers have historically ignored it, and even with reputable engines, the user-action fetchers (the ones acting on a real person's request) may not treat your rules as a hard block, since the human, not the bot, initiated the fetch. If you need to guarantee a bot cannot reach something, robots.txt is the wrong tool; you need firewall-level enforcement. robots.txt is excellent at telling cooperative crawlers your preference, and it is the right and standard way to allow the engines you want. Just do not mistake a preference for a lock.

Put those three together and the allow-side strategy is simple and honest: open the search and user crawlers so you are citable, decide on the training crawlers based on how you feel about model training (knowing it costs you nothing in live visibility), and verify the door is actually open at the request level, not just in the file.

How to Verify a Crawler Can Really Reach You

Reading your robots.txt is step zero, not the finish line. Here is how I confirm a bot can actually get through, in order of how much each tells you.

Start with the live checker. Run your URL through our AI bot checker first, because it resolves what an AI crawler's request actually returns, which catches both robots.txt rules and the upstream firewall blocks that reading the file alone hides. If it comes back clean, you have your answer in seconds. If it flags a block, it tells you which bot and gives you the line to fix.

Cross-check with a manual request. From a terminal, you can fetch your page while presenting a crawler's user-agent and watch the status code. A 200 with your real HTML means the door is open for that signature; a 403, a challenge page, or a redirect to a block page means something upstream is intercepting it. Test a couple of the AI tokens, not just Googlebot, since firewalls often single out AI crawlers specifically. This is the manual version of what the checker automates.

Mind the limits of Google's tooling. Google Search Console's robots.txt report and URL inspection are genuinely useful, but they speak for Googlebot. They will not tell you whether GPTBot, ClaudeBot, or PerplexityBot can reach you, because those are not Google's crawlers. Confirming your robots.txt is valid in Search Console is worth doing, but do not read a green check there as proof the AI crawlers are getting in. That is precisely the gap the checker fills.

The pattern: verify the AI crawlers specifically, at the request level, not by trusting a file you read or a tool that only speaks for Googlebot. A clean robots.txt is necessary. It is not sufficient.

Frequently Asked Questions

How do I allow GPTBot in robots.txt?

Add a block with the exact token: User-agent: GPTBot on one line and Allow: / on the next. If your file currently has User-agent: GPTBot followed by Disallow: /, the fix is to change that Disallow to Allow or remove the disallow entirely. Case and spelling are matched literally, so gptbot or GPT-Bot will not work. Remember that GPTBot is OpenAI's training crawler; to be citable in live ChatGPT answers you also want OAI-SearchBot and ChatGPT-User allowed.

Will blocking GPTBot remove me from ChatGPT?

No. Blocking GPTBot opts you out of OpenAI's training crawl going forward; it does not delete you from ChatGPT and does not block the separate search crawler (OAI-SearchBot) that handles live citations. If your goal is to stay visible in ChatGPT's answers, leave the search and user crawlers allowed regardless of what you decide about training.

What is the difference between a training crawler and a search crawler?

A training crawler (GPTBot, ClaudeBot, Google-Extended) collects content to help build future model versions; blocking it has little effect on whether you appear in today's AI answers. A search crawler (OAI-SearchBot, PerplexityBot, Claude-SearchBot) fetches pages so the engine can find and cite you in live answers right now. If AI visibility is the goal, the search crawlers are the ones you must allow.

My robots.txt allows the bots, so why am I still blocked?

Almost always because a layer above robots.txt is intercepting the request. CDNs, web application firewalls, and security plugins often ship "block AI bots" rules, sometimes on by default, that return a block before the crawler ever reaches your robots.txt. Reading the file shows your intent; it does not show what the bot receives. Check the request itself with our AI bot checker, then look at your CDN or firewall's bot-management settings.

Does robots.txt actually stop a crawler?

For cooperative, reputable engines, yes, they honor it as a stated preference. But robots.txt is a convention, not an enforcement mechanism. Some crawlers ignore it, and user-action fetchers acting on a real person's request may not treat it as a hard block. To allow the engines you want, robots.txt is exactly the right tool. To guarantee something stays out, you need firewall-level enforcement, not a robots.txt line.

Should I add an llms.txt file too?

That is a separate decision. llms.txt is an advisory content map, not an access control, so it does not allow or block anything; robots.txt is the gate. If you want to point AI models at your best pages, our llms.txt explainer covers whether it is worth it, and the generator builds one for you. But it does nothing to fix a crawler block, which is purely a robots.txt and firewall question.

---

Allowing AI crawlers comes down to one honest chain: know which bot does which job, write the exact tokens, delete the leftover blocks, and verify at the request level that the door is actually open, not just open in the file. The training-versus-search split is the idea that makes the rest fall into place, and the firewall-above-robots.txt trap is the reason a clean-looking file fools so many people. Start where it is fastest: run your site through the free AI bot checker, see which AI crawlers can reach you right now, fix any block in a line, and then go make the pages worth citing.

This guide is part of our series on AI search visibility, anchored by how to rank on ChatGPT. Written by Minel Gunesoglu, founder of Is My Brand in AI — more about us. I run the free AI bot checker and IndexNow live on this site, and where exact figures are not public I have described the mechanics honestly rather than inventing numbers. Reviewed June 14, 2026.