
For most small businesses, freelancers and agencies, the best robots.txt for AI is one that keeps the AI search crawlers welcome, decides deliberately about the training crawlers, and never blocks Googlebot by accident. The file is a polite request, not a lock. It works on crawlers that choose to honor it, and each AI company runs several crawlers with different jobs. Treat it as one part of a visibility plan, not a switch that says “AI: yes or no.”
This guide explains what the file does, which AI crawlers you can address, and which rules to use for the three most common goals. It also covers how to check that your rules work and what robots.txt cannot do.
TL;DR
- Robots.txt is a plain text file at the root of your domain that tells crawlers which URLs they may fetch. It is a request, not access control.
- AI companies run separate crawlers for training, search indexing and user-triggered fetching. Blocking one does not block the others.
- If you want to be cited in AI answers, leave the search crawlers (OAI-SearchBot, PerplexityBot, Claude-SearchBot) allowed. Blocking them can cost you visibility.
- Robots.txt does not hide pages, remove content already used in training, or stop crawlers that ignore it. Use authentication for anything private.
- A correct file still does nothing if AI crawlers cannot read your pages, so check that your content is in the raw HTML.
Table of Contents
- What robots.txt does, and how to read yours
- Why AI crawlers need a separate decision
- The main AI crawlers and what they do
- Three setups that cover most sites
- What the choice does to your visibility
- What robots.txt cannot do
- How to check that your rules work
- A short worked example
- Where robots.txt fits in an AI visibility plan
- Let us handle the content side
- Our perspective
- FAQ
- Sources
What robots.txt does, and how to read yours
Robots.txt is a text file that sits at the top level of your site, for example https://yoursite.com/robots.txt. Google’s documentation says it tells crawlers which URLs they can access and is mainly used to avoid overloading a site. Google’s specification adds that the file must be UTF-8 plain text in the top-level directory, and that it supports four fields: user-agent, allow, disallow and sitemap.
To read your own file, type your domain followed by /robots.txt into a browser. If you see plain text, the file exists. If you get a 404 error, there is no file, and compliant crawlers treat that as permission to crawl everything. Each subdomain, such as blog.yoursite.com, needs its own file.
A file is built from groups. Each group starts with a User-agent line naming a crawler, followed by Disallow or Allow lines naming paths:
User-agent: GPTBot
Disallow: /
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
The first group tells GPTBot to stay out of the whole site. The second says every other crawler may fetch everything. The last line points to your sitemap, which helps crawlers find your pages.
If you run WordPress, Shopify, Wix or a similar platform, you usually edit this file through a setting or plugin rather than uploading it by hand. Whatever the route, the result must appear at that root URL.

Why AI crawlers need a separate decision
For years, robots.txt was mostly about Googlebot and Bingbot. AI adds more visitors, and they do different jobs. Cloudflare’s documentation explains that AI companies use crawlers to collect content for training models, generating search answers and other purposes.
The practical consequence is that one company can have several user-agent tokens (the names crawlers use to identify themselves in your rules). A technical note from Barkhausen states that disallowing the training crawler does not disallow the others, and that a broad rule can remove a site from AI-generated answers as a side effect.
There are three jobs to keep apart:
- Training: collecting content to train or fine-tune language models.
- Search indexing: building an index that an AI assistant uses to find and cite sources.
- User-triggered fetching: loading a page live because a person asked the assistant about it.
Your goals decide which jobs you want to permit.
The main AI crawlers and what they do
Search Engine Journal’s December 2025 list of AI user agents documents the following. Kitbase’s July 2026 template guide adds notes on which ones honor robots.txt.
| Token | Operator | Job | Honors robots.txt? |
|---|---|---|---|
| GPTBot | OpenAI | Training data for GPT models | Yes |
| OAI-SearchBot | OpenAI | ChatGPT search indexing, not training | Yes |
| ChatGPT-User | OpenAI | Real-time browsing when a user asks | May bypass |
| ClaudeBot | Anthropic | Claude training | Yes |
| Claude-SearchBot | Anthropic | Claude search indexing | Yes |
| PerplexityBot | Perplexity | Answer-engine search indexing | Yes |
| Perplexity-User | Perplexity | User-triggered fetch | Often bypasses |
| Google-Extended | Controls AI-training use of content Googlebot crawled | Opt-out token | |
| Applebot-Extended | Apple | AI-training opt-out token | Opt-out token |
| CCBot | Common Crawl | Open web archive used as training data by multiple AI companies | Yes |
| Bytespider | ByteDance | LLM training | Inconsistent |
| Amazonbot | Amazon | Alexa and other Amazon AI services | Listed, see SEJ |
Two details matter here. First, Google-Extended is not a crawler that visits your site. It is a token that tells Google whether content its normal crawler fetched may be used for AI training. Second, the user-triggered fetchers are different from bulk crawlers: when someone asks ChatGPT to read a specific page, that request often does not follow robots.txt rules.
Tokens change as companies add and rename crawlers, so review the list a couple of times a year against each operator’s own documentation.
Three setups that cover most sites
Pick the one that matches what you want, then copy and adjust. Rules are matched per user-agent, so place each group on its own lines with a blank line between groups.
Setup 1: Be visible everywhere (the default for most businesses)
If your goal is traffic and citations, you do not need to “unlock” anything. With no file, or with a file that has no rules for these bots, they are allowed. Google’s specification states that by default there are no restrictions on crawling for the designated crawlers. A minimal file looks like this:
User-agent: *
Allow: /
Sitemap: https://yoursite.com/sitemap.xml
This is the right choice for a small business that depends on being found. Your content can then appear in search results and be cited in assistants’ answers.
Setup 2: Stay in AI search, opt out of training
Many site owners want citations but do not want their pages used as training material. The approach is to disallow the training tokens and leave the search tokens alone. Kitbase’s template follows this logic: it disallows GPTBot, ClaudeBot, CCBot, Bytespider, Meta-ExternalAgent, Google-Extended and Applebot-Extended, while deliberately leaving OAI-SearchBot, PerplexityBot and Claude-SearchBot allowed, because blocking the retrieval bots is “almost always a mistake” when you want citations.
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: CCBot
User-agent: Bytespider
User-agent: Meta-ExternalAgent
User-agent: Google-Extended
User-agent: Applebot-Extended
Disallow: /
User-agent: OAI-SearchBot
User-agent: PerplexityBot
User-agent: Claude-SearchBot
Allow: /
User-agent: *
Allow: /
Playing it safe means one more thing: do not block Googlebot or Bingbot. Disallowing the wrong token can pull you out of regular search too.
Setup 3: Opt out of AI use broadly
Some publishers, such as news organizations, want to keep AI companies out altogether. The IPTC, an international standards body for news media, notes that there is no universal generative-AI opt-out in robots.txt and recommends blocking AI engines individually by user-agent. You would repeat the Disallow rule from Setup 2 for each token and also add the search tokens. Expect to maintain the list, because new crawlers appear regularly.
Cloudflare also describes Content Signals, machine-readable directives that categorize use into three groups: search, ai-input (feeding content into AI models for real-time answers) and ai-train. They give you a way to state preferences by purpose rather than by crawler name, though how widely crawlers respect them is up to each operator.
What the choice does to your visibility
Blocking has a cost for businesses that rely on discovery. Playwire’s publisher guide says that blocking AI crawlers may reduce visibility in AI-powered search, since ChatGPT Search and Perplexity drive meaningful and growing referral traffic. The IPTC also warns that SEO performance might be negatively affected if search engine crawlers are inadvertently blocked.
Crawlix frames “block AI crawlers” as a crude stand-in for four goals that often conflict: being cited, avoiding training use, protecting paywalled value and controlling bandwidth. Decide which of those matters most to you before you write a rule. For a freelancer or a small business selling services, being cited usually outweighs the others. If you want to understand what earns citations once crawlers can reach you, see our guide on how to get cited by ChatGPT and AI search engines. For a wider view of the channel, this overview of AI marketing for small business covers where AI search fits in a small team’s plan.
Access is not the same as visibility
Allowing a crawler does not mean it can read your page. Crawlix reports that GPTBot, ClaudeBot, PerplexityBot and CCBot do not execute JavaScript: they fetch raw HTML. A page that builds its text in the browser after loading can look empty to them, whatever robots.txt says. Only Googlebot, and Bingbot to a degree, renders pages in a headless browser.
A quick check is to view your page source (not the inspected page) and confirm your headline and body copy are in it. If they are not, ask your developer or theme provider about server-side rendering.
What robots.txt cannot do
This is where most mistakes come from. Robots.txt has hard limits, and the official sources agree on them.
It is a request. RFC 9309, the formal Robots Exclusion Protocol, says the file is not a substitute for valid security measures. Cloudflare notes that compliance is voluntary and some operators may ignore Disallow rules. Reputable crawlers from OpenAI, Anthropic, Google and Perplexity generally follow it, and Kinsta’s comparison adds that only bot protection such as Cloudflare AI Crawl Control technically enforces blocking.
It does not hide pages. Per Google, a page disallowed in robots.txt can still be indexed if other sites link to it, and the URL can appear in results without a description. To keep a page out of Google, use noindex or password protection.
It advertises what you list. The file is public. MDN’s security guide warns that adding sensitive paths to it shows their locations to anyone who looks, and that some bots, such as malware and email harvesters, ignore it entirely. Never list private campaign pages, customer areas or draft URLs as a way to protect them. Put them behind a login.
It is not retroactive. Blocking stops future crawls. It does not remove content from models already trained on earlier snapshots. Crawlix notes this for CCBot specifically: blocking it today does not pull a site out of datasets built from prior Common Crawl snapshots.
A missing or blocked file counts as open. RFC 9309 says that if the file’s status code shows it is unavailable, a crawler may access any resource. The IPTC therefore recommends that firewalls should not return a 4xx status for /robots.txt.
llms.txt is not a replacement. Kinsta reports that llms.txt adoption sits around 9 to 10% of sites and that major AI crawlers largely do not fetch it. Robots.txt remains the file that compliant crawlers actually read.

How to check that your rules work
Syntax mistakes fail silently. Kitbase points out that a typo in a token or a stray Allow rule means the bot keeps crawling with no error message. Run through this routine after every change:
- Open the live file. Visit
yoursite.com/robots.txtand confirm the text you intended is what appears, including on any subdomain. - Check spelling and grouping. Each token must match the operator’s published name. Put one blank line between groups.
- Look at Google Search Console. Its robots.txt report shows the version Google fetched and any parsing problems.
- Review your host or CDN settings. If you use a service such as Cloudflare, its bot settings can add or override rules, and the most restrictive configuration wins.
- Read your server logs. Search the access log for crawler names. GrowthGPT’s guide gives the example
grep -aiE 'GPTBot|OAI-SearchBot|ChatGPT-User' access.log, and cautions that user-agent strings are easy to spoof, so confirm suspicious requests against the providers’ published IP ranges.
When you read the logs, expect different patterns. Training crawls arrive in scheduled waves, while retrieval hits can spike after your content is cited in a popular answer. In Search Engine Journal’s own measurements, ChatGPT-User made far more requests per hour than GPTBot, which is a reminder that live user fetches are a different kind of traffic.
Cloudflare’s style guide reports a useful data point from its own docs: after adding a robots.txt disallow on preview sites, crawls on those sites dropped from 80% to 20%. That is a large reduction, and it also shows that a share of crawling continued, so use server-level controls if you need a guarantee.
A short worked example
Imagine a three-person marketing agency with a blog on WordPress. They want clients to find them through ChatGPT and Perplexity, but they do not want their original research reused for model training.
They start from Setup 2. They disallow GPTBot, ClaudeBot, CCBot, Bytespider, Google-Extended and Applebot-Extended, and leave OAI-SearchBot, PerplexityBot and Claude-SearchBot allowed. They add their sitemap line, then open /robots.txt to confirm it loaded. In Search Console they see the file fetched without errors. Finally they view the page source of a recent post and confirm the full article text is in the HTML. Four checks, about half an hour, and the decision is documented for the next person who edits the file.
Where robots.txt fits in an AI visibility plan
Opening the door is only the first step. Crawlers that can reach you still have to find content worth citing, understand what each page is about, and see your brand attached to useful answers. That means pages built around the questions your customers actually ask, clear structure, and steady publishing. Our guides on how to rank in AI Overviews and search engines and AI citation tracking cover the next steps.
Let us handle the content side
Once your robots.txt welcomes the right crawlers, the question becomes what they find. Tepine plans, writes and publishes targeted articles every day, based on competitor-informed keyword research, in a brand voice extracted from your existing site content. It publishes to common CMS platforms, adds structured schema for Google and AI assistants, and tracks AI visibility metrics so you can see how your site is recognized. A built-in review queue lets you approve what goes live. The Starter plan is $79 per month, and the pricing page shows what it includes.

If you want to see how your site looks to AI assistants first, the free AI visibility checker answers whether ChatGPT can see your website.
Our perspective
Most robots.txt advice online is about saying no. For the businesses we work with, freelancers, small companies and agencies, the bigger risk is the opposite: a site that is technically open but gives AI assistants very little to cite.
We think of robots.txt as the front door and content as everything behind it. The door should be open to search and answer crawlers, and you decide separately what to do about training. After that, the work is publishing pages that answer real questions in your own voice, consistently, with structure that machines can read.
That is why we built Tepine around daily publishing, a plan grounded in proven search demand, structured schema and AI visibility metrics. Being mentioned and ranked in AI answers is a result of showing up often with useful, well-structured pages, and a clean robots.txt is what lets that work be seen.
Tepine
FAQ
Is a robots.txt file required?
No, a site works without one, and compliant crawlers treat a missing file as permission to crawl everything. Google says the file is mainly used to avoid overloading a site, not to keep pages out of search. You need one if you want to set different rules for specific crawlers or point to your sitemap.
Does blocking GPTBot remove my site from ChatGPT?
Not necessarily. GPTBot is OpenAI’s training crawler, while OAI-SearchBot handles ChatGPT search indexing and ChatGPT-User fetches pages when a person asks. Disallowing GPTBot stops future training collection by that crawler but does not address the other two, and it does not remove content already collected.
Is robots.txt legal protection?
No. The IPTC states that robots.txt does not guarantee compliance in any jurisdiction, and RFC 9309 describes it as a protocol for crawler cooperation, not authorization. If you need legal or contractual protection for your content, speak to a lawyer who handles intellectual property, and use authentication to restrict access to private material.
Should I block AI crawlers to protect my content?
That depends on your goal. If you rely on being found, blocking search crawlers such as OAI-SearchBot or PerplexityBot can reduce your presence in AI answers. Opting out of training tokens while leaving search tokens open is the common middle path, and the setups above show the exact rules.
How often should I review my robots.txt?
Review it whenever you change your site structure or platform, and a couple of times a year for new AI user-agent tokens. Operators add and rename crawlers, so an old file can miss new ones. Open the live file after each edit to confirm it saved as intended.
Sources
- Robots.txt Introduction and Guide, Google Search Central
- How Google interprets the robots.txt specification
- Complete Crawler List For AI User-Agents, Search Engine Journal
- Managed robots.txt, Cloudflare documentation
- Control how AI crawls your docs, Cloudflare Style Guide
- IPTC Generative AI Opt-Out Best Practices
- RFC 9309: Robots Exclusion Protocol
- robots.txt configuration, MDN Web Docs
- What robots.txt does and doesn’t control for AI crawlers, Barkhausen
- robots.txt for AI Crawlers: Copy-Paste Templates, Kitbase
- What GPTBot, Google-Extended, and ClaudeBot Can Actually See, Crawlix
- llms.txt vs robots.txt, Kinsta
- The Complete Publisher’s Guide to AI Crawlers, Playwire
- How AI Crawlers Work, GrowthGPT
- AI Marketing for Small Business: What Works in 2026


