Link Plumbing

How Robots.txt Rules Are Matched

Most robots.txt mistakes are not disagreements about policy. They are matching errors: a rule that blocks more than its author expected, a group of rules that is silently never consulted, or an Allow that everyone assumed was overriding a Disallow and wasn’t.

The matching rules are small enough to hold in your head, and once you have them the file stops being guesswork.

The file’s scope is a host, a scheme, and a port

A robots.txt file governs exactly one origin. https://example.com/robots.txt says nothing about http://example.com, nothing about https://www.example.com, and nothing about https://example.com:8443. Each is a separate origin and each is expected to serve its own file at its own root.

This is the first thing to check on a site that consolidated hosts: if www redirects to the apex, then a request for https://www.example.com/robots.txt follows the redirect and the apex file applies — fine. If www is not redirected and has no file of its own, it is unrestricted. See consolidating www and HTTPS in a single hop for the redirect side of that.

The file must also live at the root path. A file at /subdir/robots.txt is a text file, not a directive.

Only one group applies

A robots.txt file is a sequence of groups. Each group is one or more User-agent lines followed by rules:

User-agent: *
Disallow: /search

User-agent: ExampleBot
Disallow: /internal

A crawler picks the single most specific group whose user-agent token matches its own name, and ignores every other group. Groups are not merged. So ExampleBot, reading the file above, is not blocked from /search — its own group exists, so the * group is irrelevant to it.

This is the most consequential rule in the file and the one most often violated by accident. A common inherited shape:

User-agent: *
Disallow: /cart
Disallow: /checkout
Disallow: /search

User-agent: Googlebot
Allow: /

Someone added the second group meaning “and definitely let Google in.” What it actually says is: Googlebot is subject to only Allow: /, so the three exclusions above no longer apply to it. Every generated surface the file was written to keep out of the crawl is now open to the crawler that matters most.

Matching of the user-agent token is case-insensitive and is a prefix match on the product token, not on the full User-Agent header string. If two groups declare the same token, well-behaved parsers combine them, but do not rely on it — write one group per agent.

Path matching is a prefix match on the path and query

A rule value is compared against the URL’s path plus query string. The comparison is a prefix match, and it is case-sensitive.

Disallow: /admin therefore matches all of:

  • /admin
  • /admin/
  • /admin/users
  • /administrator
  • /admin-notes.html
  • /admin?x=1

That fifth one surprises people. If you meant the directory, write the trailing slash: Disallow: /admin/. If you meant that exact URL and nothing beneath it, you need an end anchor.

Case-sensitivity matters on a site with mixed-case URLs. Disallow: /Search does not block /search. That is one more reason to normalise case at the URL layer, as covered in uppercase URLs and the duplicates they create.

Two wildcards are supported:

  • * matches any sequence of characters.
  • $ anchors the match to the end of the URL.

So Disallow: /*.pdf$ blocks every URL ending in .pdf, and Disallow: /*? blocks every URL containing a query string — a heavy-handed rule that is occasionally what you want and frequently not, because it also blocks legitimate parameterised pages.

Percent-encoding is compared literally, with one exception worth knowing: the path is normalised for the encoding of unreserved characters, but %2F is not treated as /. If your URLs carry encoded slashes, test the rule rather than reasoning about it.

Longest match wins, and ties go to Allow

When both an Allow and a Disallow match a URL, the rule with the longer value wins. Order in the file is irrelevant. If the two matching values are the same length, the less restrictive rule — Allow — wins.

The worked example:

User-agent: *
Disallow: /reports/
Allow: /reports/public/
URL Longest matching rule Result
/reports/2026-q1.pdf Disallow: /reports/ (10) blocked
/reports/public/index.html Allow: /reports/public/ (17) allowed
/reports/publications/ Allow: /reports/public/? no — the value is not a prefix of this path blocked

The third row is the one to read twice. /reports/public/ is not a prefix of /reports/publications/, because the trailing slash in the rule does not appear at that position in the URL. Drop the trailing slash from the Allow and it would match, and you would have opened a directory you did not mean to.

This is also why Allow exists at all. It has no meaning on its own — with no Disallow in the group, everything is already allowed. Its only job is to carve an exception out of a broader Disallow, and it only does so when it matches with a longer value.

What happens when the file itself fails

The status code of /robots.txt is a directive of its own, and the handling is documented rather than intuitive:

  • 2xx — parse and obey.
  • 404 or any other 4xx (except 429) — treated as “no restrictions.” The absence of a file is permission.
  • 5xx and 429 — treated as a full disallow, temporarily. A crawler that gets a server error for robots.txt may back off from the whole host rather than assume it is welcome.
  • 3xx — followed, within a small hop limit. A robots.txt that redirects into an HTML error page counts as a failed fetch, not as an empty file.

The 5xx case is the one that causes mysterious crawl stalls. If your robots.txt is generated by the application rather than served as a static file, it shares the application’s failure modes — and a deploy that 500s for ten minutes has told every crawler to stop asking for anything. Serve the file statically from the edge if you can. The related throttling behaviour is in what a 429 does to your crawl rate.

Also: the file must be UTF-8 plain text, parsers stop reading after a size limit (as of this writing, in the region of half a megabyte for Google), and everything after # on a line is a comment.

Lines that are not rules

Sitemap: is a global directive. It is not scoped to a group, it can appear anywhere in the file, and it takes an absolute URL. Indenting it inside a User-agent block does not attach it to that agent — see what an XML sitemap is actually for.

Crawl-delay: was never part of the standard. Some crawlers honour it; the major ones largely do not, and expressing rate limits with HTTP status codes is the mechanism that actually works.

Noindex: in robots.txt does nothing at all. There is no such directive, and a page you want out of results has to be crawlable to say so — the whole argument is in robots.txt cannot deindex a page.

Unrecognised lines are ignored, which means a typo — Dissalow, User agent — fails silently. There is no error to notice.

The check

Before shipping a change, evaluate the file the way a parser does, in this order:

  1. Which group applies to the agent you care about? If it has its own group, nothing outside that group counts.
  2. For the URL in question, list every matching rule — remembering prefix semantics.
  3. Take the longest one. Ties go to Allow.
  4. Confirm the file returns 200 from the exact origin the URLs live on.
# what does each origin actually serve?
for h in https://example.com http://example.com https://www.example.com; do
  printf '%s -> ' "$h"
  curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "$h/robots.txt"
done

Four steps, and they resolve nearly every “why is this being crawled” and “why is nothing being crawled” question the file can produce.

http