Link Plumbing

X-Robots-Tag and Files With No Head Element

X-Robots-Tag is an HTTP response header that carries the same directives as <meta name="robots">. It exists because the meta tag requires an HTML <head>, and plenty of indexable URLs do not have one: PDFs, spreadsheets, images, plain-text files, JSON feeds. For those, the header is the only way to express noindex.

It is also the more convenient option for HTML in one specific situation — when you want to apply a directive to a class of URLs by pattern rather than by editing templates.

The equivalence

The two mechanisms carry the same vocabulary and are treated equivalently where both are supported. These are the same instruction:

<meta name="robots" content="noindex, nofollow">
X-Robots-Tag: noindex, nofollow

The directives worth knowing:

  • noindex — do not show this URL in results. The URL must be crawlable for this to be read, which is the trap described in robots.txt cannot deindex a page.
  • nofollow — do not follow links from this page. Page-level, and distinct from a per-link rel="nofollow".
  • none — shorthand for noindex, nofollow.
  • noarchive — no cached copy.
  • nosnippet — no text snippet in results.
  • max-snippet:N, max-image-preview:none|standard|large, max-video-preview:N — limits on how much of the page can appear in a result.
  • noimageindex — do not index images on this page.
  • unavailable_after: <date> — stop showing the URL in results after a given RFC 850 or ISO 8601 date.

index and follow exist but are the defaults; setting them explicitly does nothing. A tag reading content="index, follow" is decoration, and it is worth removing from templates simply so that the presence of a robots tag means something.

Targeting a specific crawler

The header takes an optional user-agent prefix:

X-Robots-Tag: googlebot: noindex
X-Robots-Tag: bingbot: noindex, nosnippet

The same URL can carry several X-Robots-Tag headers, and a crawler applies the ones addressed to it plus the ones with no prefix. An unprefixed directive applies to everyone.

Be careful combining prefixed and unprefixed lines: a bare X-Robots-Tag: noindex alongside X-Robots-Tag: googlebot: nosnippet means Googlebot honours both — it is not indexing the page and would not snippet it. Directives accumulate; they do not override by specificity.

Where the header is the right tool

Non-HTML files. A PDF has no <head>. If you are hosting price lists, whitepapers, or internal documents that are reachable by URL and should not appear in search results, the header is the only mechanism.

# nginx: keep a documents directory out of results
location ^~ /documents/ {
    add_header X-Robots-Tag "noindex, nofollow" always;
}

The always matters. Without it, nginx omits add_header on non-2xx responses, so a 304 Not Modified served from a conditional request would carry no directive — and 304s are common for static files behind a cache.

Pattern-based application across many URLs. Anything you can express as a location or rewrite condition, you can apply the header to, without touching a template. Useful when the URLs are generated by something you do not control.

# Apache: noindex any URL ending in .pdf
<FilesMatch "\.pdf$">
    Header set X-Robots-Tag "noindex, nofollow"
</FilesMatch>

Non-HTML endpoints that get indexed by accident. An RSS feed, a .json API response, a sitemap.xml — these can appear in results if linked. Whether that matters is a judgement call, but the header is how you act on it.

Edge-level directives. A CDN or reverse proxy can attach the header without an origin deploy, which makes it the fastest way to act on a URL class you have just discovered in the index. The trade-off is the usual one for edge configuration: it is invisible to anyone reading the application code, so it needs to be documented somewhere developers will look. The same trade-off applies to redirects — see where to put a redirect.

When both are present on one URL

If a URL carries both a header and a meta tag, the directives combine rather than one winning outright. The restrictive interpretation applies: a header saying noindex and a tag saying nothing still results in noindex, and a header saying nothing with a tag saying noindex gives the same outcome.

Where they genuinely contradict — the header permits indexing, the tag forbids it — the more restrictive directive is applied, because there is no mechanism for one to cancel the other. There is no “index” instruction that undoes a noindex; the absence of a directive is the permissive state, and any present restriction sticks.

The operational consequence: you cannot fix an unwanted noindex by adding a permissive one elsewhere. You have to remove the restriction at its source, which means finding it. The commonest version of this is a noindex left on a page from a staging deploy, plus a later attempt to override it in a template, plus confusion about why the page is still absent.

Check both, always:

curl -sI https://example.com/page/ | grep -i x-robots-tag
curl -s  https://example.com/page/ | grep -io '<meta name="robots"[^>]*>'

The failure modes worth naming

A global header set at the server or CDN level and forgotten. The single most damaging version: X-Robots-Tag: noindex applied to an entire host during a pre-launch period, then never removed. The site launches, the templates are correct, and nothing is indexed. It is invisible from the HTML and invisible from the application code. Any launch checklist should include a curl -I against the production host looking for this header specifically.

A staging host that shares configuration with production. If the noindex header lives in a config file that both environments load, whether it applies is decided by an environment variable that someone can get wrong. Prefer HTTP authentication for staging — a 401 removes the question entirely.

Blocking the URL in robots.txt as well. The header is in the response; the crawler has to fetch the response. Belt-and-braces here means neither works.

Trusting the header on a cached response. An edge cache that was populated before the header was added will keep serving the old response. Purge, then verify from outside.

unavailable_after with a past date, set by accident. A date in the past means the directive is already in effect. It reads as a scheduling feature and behaves as a noindex.

Choosing between the two

For HTML pages where the directive is a property of the page’s content or type, put it in the template as a meta tag — it lives next to the thing it describes and is visible to anyone reading the page source.

For non-HTML files, and for directives that apply to a URL pattern rather than a content type, use the header.

For anything applied above the application, write down where it lives. A directive that nobody can find is a directive that stays in place through three redesigns.

httpcanonicals