AI crawlers in robots.txt: a study of the top 10k sites (2026)

A robots.txt file is a list of groups. Each group starts with one or more User-agent lines and lists the rules for those crawlers. The standard, RFC 9309, has a crawler look for the group that names it. Only if there is none does it fall back:

“If no matching group exists, crawlers MUST obey the group with a user-agent line with the "*" value, if present.”

Google documents the same thing for its own crawlers: user agent specific groups and the global * group are not combined. So the moment a file adds a group for GPTBot — even one that only says Allow: / to welcome it — GPTBot stops reading every rule written under User-agent: *.

User-agent: * Disallow: /search Disallow: /account/ Disallow: /api/

User-agent: GPTBot Allow: /

What each crawler may fetch

Path Every other crawlere.g. Googlebot GPTBothas its own group /search?q=… Blocked Allowed /account/settings Blocked Allowed /api/v2/orders Blocked Allowed /blog/any-post Allowed Allowed

Highlighted rows: paths the file blocks for everyone except the crawler it names.

User-agent: GPTBot Disallow: /search Disallow: /account/ Disallow: /api/

Our free robots.txt checker finds this for you: paste a file and it lists every path a named crawler can reach that User-agent: * blocks.

How we count it: a file has a gap when a path is blocked under User-agent: * and a named AI crawler's own group has no rule for that path at all, or only a blanket Allow: /. If the group allows a specific path on purpose (for example Allow: /api/public/), we don't count it. Blocking the whole site under * while allowing named crawlers is a common deliberate pattern, so Disallow: / on its own doesn't count either.

None of these paths is secret — robots.txt is public, and it isn't access control. But they are the paths site owners block on purpose: search results that spawn endless near-duplicate URLs, account pages, and endpoints that cost money to serve.

Grouped by patterns in the blocked path (search, ?s=, login, account, /api/, admin, cart, checkout, feed and similar), so the types are approximate.

The most common shape is a group that only allows everything — a blanket Allow: / or an empty Disallow:: 182 of the 329 files (55.3%). The other 147 give the crawler rules of its own but leave out some of the paths blocked for everyone else.

Google-Extended is a permission token, not a crawler: Google fetches with Googlebot, which still follows its own group. For it, a gap is a gap in the rules, not necessarily in what gets fetched.

Share of the 5,539 robots.txt files we could parse whose rules block each crawler from the whole site — the verdict for /. That includes files that block every crawler under User-agent: *: 238 files (4.3%) do. The top 1,000 domains block more than the rest: 21.6% block GPTBot, against 14.1% for ranks 1,001–10,000.

Used for, from each vendor's documentation: training crawlers collect pages to build models; answer engines fetch pages to answer a question and can cite them; other covers ad checks and similar fetchers. Perplexity says Perplexity-User generally ignores robots.txt because a user asked for the page, so blocking it states a preference it won't enforce.

A common recommendation is to split the decision: refuse training crawlers such as GPTBot and CCBot, which give nothing back, and allow answer engines such as OAI-SearchBot and PerplexityBot, which can cite and link to you. Few sites do it. Of the 1,106 files that block at least one training crawler, 714 (64.6%) also block at least one answer engine. Only 392 (7.1% of all files) block training while leaving all 9 answer engines open.

8 training crawlers, 9 answer engines and 3 others. Blocking all 20 is almost never done by name: 193 of the 194 files that block all 20 do it by blocking every crawler.

llms.txt, proposed in September 2024, is a Markdown file at the root of a site that gives language models a summary and a list of key pages. Its one required element is a # title line. 878 of the 6,082 sites that answered our request (14.4%) serve a valid one: status 200, plain text rather than a web page, starting with a title line. The top 1,000 domains are a little ahead: 18.5%, against 14.0% for ranks 1,001–10,000.

Content Signals, published by Cloudflare in September 2025, add a Content-Signal line to robots.txt with three yes-or-no preferences: search, ai-input and ai-train. 150 of the 5,539 files (2.7%) carry one, and 90 of those say ai-train=no. The line states a preference; it is not part of RFC 9309, and crawlers are not required to read it.

We identified 396 WordPress sites from their homepage HTML. 41.3% of their robots.txt files still contain WordPress's default rules (Disallow: /wp-admin/ and Allow: /wp-admin/admin-ajax.php). They name AI crawlers less often than other sites and block GPTBot about as often.

WordPress: a generator tag, an api.w.org link, or both wp-content and wp-includes paths in the homepage HTML. The 22 sites with WordPress rules in robots.txt but no trace in the homepage are left out of both columns. WordPress is uncommon this high up the list (7.5% of readable homepages), so this is a small sample.

Paste your robots.txt into our free checker. It shows what Googlebot, Bingbot and 20 AI crawlers may fetch, and lists every path a named crawler can reach that User-agent: * blocks. It runs in your browser; nothing is uploaded.

Writing a new file? The generator on the same page copies your rules into every crawler group it adds, so none of them has a gap.

On WordPress? AI Crawler Check, our free plugin, checks your live robots.txt for these gaps from the dashboard and flags them in Site Health. It reads the file and never edits it.

Download the aggregate figures (CSV). Every number on this page comes from that file, with its denominator. We don't publish per-site results, as our crawler page promises.

While running the numbers we found and fixed four problems in our own robots.txt checker — two that reported gaps that weren't there, and two that made it hang on unusual files (one with 88 wildcards in a rule, one with 12,000 Disallow lines). Every figure here was recomputed from the stored files with the fixed code.

HitMind, “AI crawler access in the top 10,000 sites,” October 2026. https://hitmind.ai/research/ai-crawler-access-2026/

Questions about the data, or a breakdown we didn't publish: support@hitmind.ai.

Related: Robots.txt for AI crawlers: the 2026 checklist · How HitMindBot crawls