Is your website blocking AI crawlers without you knowing?

This takes two minutes and is worth doing before anything else on this topic.

How to check

Type your domain followed by /robots.txt into a browser — for example yourbusiness.co.uk/robots.txt. You will see a plain text file. Search it for these names:

• GPTBot — OpenAI's training crawler
• OAI-SearchBot — OpenAI's search crawler, separate from the above
• ClaudeBot — Anthropic
• PerplexityBot — Perplexity
• Google-Extended — controls Gemini and AI Overviews usage

If any appears followed by Disallow: /, that assistant has been told not to read your site.

Why this is often accidental

Through 2024 and 2025 a great deal of advice recommended blocking these crawlers, and many SEO plugins, hosting platforms and agency templates added the rules by default. Publishers had a genuine reason: their content is their product, and they were not being paid for it.

Most businesses are not publishers. If you sell roofing, or dentistry, or logistics software, your website content is not the product — it is the marketing for the product. Blocking the crawlers protects nothing and costs you the recommendation.

The trade-off, stated honestly

Allowing these crawlers means your content may be used in training and in generated answers, sometimes without a click through to your site. That is a real cost. The question is whether being named in an answer with no click is worth more than not being named at all.

For a service business, being named is worth considerably more. People who ask an assistant for a recommendation and receive three names then search those names directly. You want to be one of them.

For a publisher whose revenue is advertising against pageviews, the calculation genuinely goes the other way.

What to change

If you want to be discoverable, ensure your robots.txt does not disallow those user-agents. Being explicit is better than silence:

User-agent: GPTBot
Allow: /

User-agent: OAI-SearchBot
Allow: /

User-agent: ClaudeBot
Allow: /

User-agent: PerplexityBot
Allow: /

User-agent: Google-Extended
Allow: /

Access alone will not get you recommended — corroboration across sources and citation-ready content do that. But without access, none of the rest can work.