Every few months a headline tells small-business owners that AI companies are taking their content. The advice that follows is usually the same: block the bots.
It is well meant, and it is often the wrong move. We have seen businesses block a list of bot names copied from a forum post, then wonder months later why ChatGPT never mentions them.
The short answer: let the search and assistant crawlers in, because they are how AI
tools find and cite you. Decide about the training crawlers separately, on your own
terms. Then check your robots.txt file so that whatever you have chosen
is deliberate rather than inherited.
What is an AI crawler?
An AI crawler is a program an AI company uses to read pages on your website, the same way Googlebot reads them for Google Search. Each one identifies itself with a name, called a user agent.
You control them with robots.txt. That is a plain text file that sits at
the top of your site, at an address like yourbusiness.com/robots.txt. It
lists crawler names and tells each one which parts of your site it may read.
One honest limit: robots.txt is a request, not a lock. Well-behaved
crawlers from the major AI companies follow it. It is not a security control. If a
page must stay private, put it behind a password instead.
Why do AI crawlers come in three types?
Because they do three different jobs, and the difference decides what blocking actually costs you.
Training crawlers collect content that may be used to train future models. Blocking one has no effect on whether you appear in AI answers today.
Search crawlers build the index an assistant looks through when someone asks it a question. Blocking one takes you out of that assistant's answers.
User-initiated fetchers visit your page at the moment a real person asks the assistant something. This is the closest thing to a customer knocking on your door.
Mixing these three up is the expensive mistake. "Block the AI bots" sounds like one decision. It is three.
Which six crawlers should you know by name?
These are the ones that matter for a small business. Each company publishes its own list, and all of them can be controlled separately.
- GPTBot — OpenAI's training crawler. Blocking it keeps your pages out of model training.
- OAI-SearchBot — OpenAI's search crawler. OpenAI states that sites which opt out of this one will not be shown in ChatGPT search answers.
- ChatGPT-User — fetches your page when a person asks ChatGPT something. OpenAI says it is not used to decide whether your content appears in search.
- ClaudeBot — Anthropic's training crawler. Its search and user counterparts are named Claude-SearchBot and Claude-User.
- PerplexityBot — Perplexity's search crawler. Perplexity's documentation says it must be allowed for your site to be indexed, and that it does not train models.
- Google-Extended — controls whether Google may use your content for Gemini. Google states it does not affect your inclusion in Google Search and is not a ranking signal.
Note what is missing from that list: Googlebot. It is not an AI crawler, and blocking it removes you from Google Search entirely.
So should you block them?
For most small businesses selling a service, no. You want to be found, and these crawlers are how you get found.
There are three common situations, and the answer changes with each.
You want more enquiries. Allow everything. Your service pages, prices and opening hours are marketing. Being read is the point. This covers most trades, clinics, consultancies and local shops.
Your writing is the product. If you sell courses, research or photography, allow the search crawlers and disallow the training ones. You stay visible in AI answers without handing over the work itself.
You have genuinely private pages. Customer portals and draft quotes belong behind a login, not behind a line in a text file.
Are you blocking AI crawlers by accident?
This is the more common problem, and it is worth ten minutes. Four places to look.
A robots.txt you inherited. Files get copied between sites and
templates for years. A Disallow: / left over from a staging site blocks
everything, including Google.
Your security plugin or firewall. Many are set to block unknown or unverified bots. That setting does not know the difference between a scraper and OAI-SearchBot.
Your CDN settings. Cloudflare changed its defaults for new domains on 15 September 2026. For a site that does not show ads, search, training and agent crawlers are all allowed by default. For ad-supported sites, training is disallowed and agents are blocked on pages with ads. Most small-business sites fall in the first group, but it is worth opening the dashboard to confirm which applies to you.
A switch in your website builder. Some platforms offer a single "block AI crawlers" toggle. It is usually all or nothing, which means it takes out the search crawlers too.
How do you check your robots.txt in two minutes?
Type your web address followed by /robots.txt into a browser and read
what comes back.
Look for Disallow: / on its own under any user agent. That single line
blocks the whole site for that crawler. Then look for the six names above, and check
whether a search crawler has been swept up with a training one.
If you get a 404 page, that is fine. No file means nothing is blocked, and every crawler is welcome by default. You are only missing the line that points to your sitemap.
To see which crawlers actually reach you, check your server logs or your CDN dashboard. Our guide to tracking AI traffic covers the free reports that show this.
What should your robots.txt actually say?
For most small businesses, this is the whole file. It welcomes every crawler and points them at your sitemap.
User-agent: *
Allow: /
Sitemap: https://yourbusiness.com/sitemap.xmlIf you would rather stay out of model training while staying visible in AI answers, add the training crawlers by name and leave the search ones alone.
User-agent: GPTBot
Disallow: /
User-agent: ClaudeBot
Disallow: /
User-agent: Google-Extended
Disallow: /Do not add OAI-SearchBot, Claude-SearchBot, PerplexityBot or Googlebot to that second block. Those are the ones earning you mentions. If your website was built for you and you cannot edit this file, ask whoever hosts it — it is a two-minute change.
Does an llms.txt file help?
Not much yet, and we would rather say so plainly. An llms.txt file is a
proposed standard that gives AI tools a short, plain-text map of your important pages.
The idea is sensible. The adoption is not there. None of the crawler documentation published by OpenAI, Anthropic, Google or Perplexity mentions reading such a file, and the crawlers overwhelmingly fetch ordinary HTML pages instead.
We publish one at greenkauri.com/llms.txt because it takes ten minutes and
costs nothing if the idea catches on. That is our judgement, not a recommendation to
prioritise it. What decides whether AI tools can use your pages is still the words on
them and the structured data behind
them.
Which official sources support this guidance?
- OpenAI: bots and crawlers — what GPTBot, OAI-SearchBot and ChatGPT-User each do, and what opting out of each one means.
- Anthropic: crawling and how to block it — ClaudeBot, Claude-SearchBot and Claude-User, and controlling each separately.
- Google Search Central: common crawlers — states that Google-Extended does not affect inclusion in Google Search or ranking.
- Perplexity: PerplexityBot and Perplexity-User — what each agent does and why PerplexityBot must be allowed.
- Cloudflare: search, training and agent crawler categories — the default settings for new domains from 15 September 2026.
AI crawler FAQs
Should I block AI crawlers from my website?
Usually not. If you want customers to find you, the search and user crawlers are how AI assistants read and cite your pages. Blocking them removes you from those answers. Training crawlers are a separate decision, and you can block those by name without losing visibility.
Does blocking GPTBot remove me from ChatGPT?
No. GPTBot only collects content that may be used for model training. ChatGPT search results come from a different crawler called OAI-SearchBot, and page fetches during a conversation come from ChatGPT-User. Block GPTBot alone and you stay visible in ChatGPT answers.
Does blocking Google-Extended hurt my Google ranking?
No. Google states that Google-Extended does not affect a site's inclusion in Google Search and is not used as a ranking signal. It controls only whether your content helps train and ground Gemini. Googlebot is the separate crawler that handles Search itself.
Is robots.txt enough to stop AI using my content?
No. It is a request that well-behaved crawlers honour, not a technical barrier. Scrapers that ignore it exist, and content already collected is not withdrawn. For anything that must stay private, use a password or a login rather than relying on a text file.
What happens if I have no robots.txt file at all?
Nothing is blocked. Every crawler, including the AI ones, may read your whole site. That is a reasonable position for most small businesses. Adding a simple file is still worth it, because it lets you point crawlers at your sitemap and name exceptions later.
Should I add an llms.txt file to my website?
Only as a small, cheap bet. No major AI company currently documents reading llms.txt, and crawlers fetch ordinary pages instead. It takes about ten minutes to write, so there is little harm in having one, but clear pages and structured data matter far more.
Letting the crawlers in is the first step. What they find when they arrive decides whether you get recommended, which is the work behind an xEO optimisation. And once those enquiries start arriving, a custom AI agent can make sure each one gets a fast, useful reply while you are out on a job.