AI crawlers
Blocking every bot with AI in the name is the most common mistake here, because these agents do very different jobs. One trains a model on your text. Another fetches your page so it can be cited in an answer a user is reading right now. Blocking the first costs you nothing. Blocking the second removes you from the answer.
What each one is for
Decide per purpose, not per company. The column that matters most is the last one: what you actually lose.
| Crawler | User agent | Purpose | robots.txt | If you block it | Checked |
|---|---|---|---|---|---|
| GPTBot OpenAI | GPTBot | Trains models Collects pages used to train future models | honours it | Nothing today. Your pages stay out of future training runs and nothing about ChatGPT answers changes. | May 2026 |
| OAI-SearchBot OpenAI | OAI-SearchBot | Builds a search index Builds the index behind ChatGPT search | honours it | You stop appearing in ChatGPT search results. This is a real cost, not a theoretical one. | May 2026 |
| ChatGPT-User OpenAI | ChatGPT-User | Fetches on behalf of one user Fetches a page because a user asked ChatGPT to open it | honours it | A user who explicitly asks ChatGPT to read your page gets an error instead. | May 2026 |
| ClaudeBot Anthropic | ClaudeBot | Trains models Collects pages used to train future models | honours it | Nothing today. Training data only. | May 2026 |
| Claude-User Anthropic | Claude-User | Fetches on behalf of one user Fetches a page on behalf of a user in a conversation | honours it | Claude cannot open your page when a user asks it to. | May 2026 |
| Claude-SearchBot Anthropic | Claude-SearchBot | Builds a search index Indexes pages so they can be surfaced in answers | honours it | You drop out of results Claude surfaces from the web. | May 2026 |
| Google-Extended Google | Google-Extended | Trains models Controls use of your content for Gemini training and grounding | honours it | Does not affect Google Search ranking at all - it is a separate token. It does affect whether you can be cited in AI Overviews. | May 2026 |
| PerplexityBot Perplexity | PerplexityBot | Builds a search index Indexes pages for citation in answers | honours it | You lose the citation links Perplexity sends, which for some sites is meaningful referral traffic. | May 2026 |
| Bytespider ByteDance | Bytespider | Trains models Broad collection, aggressive crawl rate | ignores it | It has a reputation for ignoring the rule. If it is a load problem, block it at the server or CDN by user agent. | May 2026 |
| CCBot Common Crawl | CCBot | Trains models Public crawl archive that many models are trained from | honours it | Your pages stay out of the Common Crawl archive, which is an input to a large number of models rather than just one. | May 2026 |
| Applebot-Extended Apple | Applebot-Extended | Trains models Controls use of your content for Apple model training | honours it | Nothing in Siri or Spotlight changes - that is plain Applebot, a separate token. | May 2026 |
| meta-externalagent Meta | meta-externalagent | Trains models Collects pages for Meta model training | honours it | Training data only. | May 2026 |
| Amazonbot Amazon | Amazonbot | Builds a search index Indexes pages for Alexa and Amazon services | honours it | You drop out of answers Alexa gives from the web. | May 2026 |
| YouBot You.com | YouBot | Builds a search index Indexes pages for citation in answers | honours it | You lose citations in You.com answers. | May 2026 |
A user agent that ignores robots.txt is not stopped by a rule in it. That needs a block at the web server or the CDN, and the honest version of that sentence is that determined scraping is not solved in a text file.
Build your robots.txt
Tick what you want to keep out. Everything runs in your browser and nothing is sent anywhere. Paste the result at the top of the robots.txt you already have, then keep your existing rules below it.
# Tick a crawler on the left and the rules appear here.Being useful to AI answers
The label for this is generative engine optimisation, which mostly describes old habits with a new name. What a model quotes is what it can extract cleanly, so the work is closer to technical writing than to keyword tuning.
Answer in the first paragraph
A model reading your page for a citation takes the passage that answers the question on its own. If the answer arrives after four paragraphs of preamble, the extractable part is the preamble.
Keep facts in text, not in images
Prices, specifications and steps rendered inside a picture do not exist for a text crawler. The same goes for anything that appears only after JavaScript runs, since most of these agents do not run it.
Say when it was last checked
A visible date next to a number is a strong signal when a model is choosing between two sources that disagree. This is why every model and crawler row on this site carries one.
Let the grounding agents in
If you block the agents that fetch pages to answer live questions, you are not protecting anything - your competitor gets cited instead. Block training if you want to; that is a separate decision and the builder above keeps them apart.
Structured data still helps
Not because a model reads JSON-LD directly, but because it forces you to state the author, the date and the subject explicitly, and those end up in the extractable text as well.