Vary, Content Negotiation, and One URL Serving Two Pages

A URL is supposed to name one thing. Content negotiation is the mechanism by which it can legitimately name several: the same address returns different bytes depending on the requesting client’s headers — a mobile layout, a Spanish translation, an AVIF image instead of a JPEG.

The Vary response header is how the server declares which request headers changed the answer. Get it wrong and the wrong representation gets cached and handed to everyone downstream, including the crawler, which then indexes a version of the page no human visitor sees.

What Vary actually does

Vary is not a hint about content. It is an instruction to caches about the cache key.

Without it, a cache stores one response per URL. Vary: Accept-Language tells the cache to store one response per URL per distinct value of the request’s Accept-Language header. The URL alone no longer identifies the cached object.

HTTP/1.1 200 OK
Content-Type: text/html; charset=utf-8
Vary: Accept-Encoding, Accept-Language
Cache-Control: public, max-age=0, must-revalidate

Two consequences follow immediately, and they pull in opposite directions:

Omitting a Vary you needed causes wrong answers. The first requester’s representation is stored under the bare URL and served to everyone. A mobile visitor warms the cache and desktop visitors get the mobile page — or the crawler warms it and every visitor gets whatever the crawler was served.

Adding a Vary you did not need destroys cache efficiency. Vary: User-Agent fragments the cache across effectively unbounded header values, because user-agent strings are near-unique. Each variant is a separate object with its own miss.

So the rule is narrow: declare exactly the headers the response genuinely depends on, and no others.

Where it intersects with crawling

Three specific cases account for most of the trouble.

Dynamic serving on user agent. The same URL returns a mobile template to mobile clients and a desktop template to everyone else. This requires Vary: User-Agent, and it means the crawler’s own user agent determines which template gets indexed. As of this writing the dominant crawler fetches primarily as a mobile client, so the mobile representation is the one that counts — the desktop version can differ substantially and never be assessed. If content, headings, structured data or internal links differ between the two templates, the difference is invisible in your desktop testing and decisive for indexing.

Language and region negotiation. Returning Spanish to a request with Accept-Language: es on the same URL as the English page collapses both languages into one address. There is then no URL to declare in an hreflang annotation, because hreflang needs one distinct URL per language — see hreflang and the return-tag requirement. A crawler that fetches from one region with one language preference sees exactly one of your translations and has no way to discover the others.

The related and worse variant is a redirect based on inferred region: a request to /product/ gets a 302 to /es/product/ because the requesting IP geolocates to Spain. Crawling largely originates from a small set of locations, so one locale’s pages get discovered and the rest are behind a redirect nobody triggers. If you must steer users by region, do it with a dismissible banner or a suggestion, and keep every locale reachable at its own URL by direct request. The status-code side of that choice is in 301 versus 302 and what each does to link equity.

Cookies. Vary: Cookie on an HTML page is almost always a design smell. It means the page’s content depends on session state, which makes the page uncacheable in practice (every session has a distinct cookie) and means the crawler — which sends no cookies — sees the anonymous representation. That is fine if the anonymous representation is the real page. It is a problem if the real content only appears once a cookie is present, because then the indexed page is an empty shell.

Serving the crawler something different is a separate question

There is a hard line between negotiating a representation and detecting a crawler.

Content negotiation keys off a declared client preference — Accept, Accept-Language, Accept-Encoding — and applies the same logic to every client that expresses the same preference. Whatever a human with those headers gets, the crawler with those headers gets too.

Branching on whether the user agent is the crawler is different in kind. It produces content that no visitor can see, and it is treated as cloaking regardless of intent. The awkward part is that the two are easy to conflate in an implementation: a rule written as “if user agent matches a bot list, serve the pre-rendered version” is crawler detection even when the motive is purely technical. The safe framing is that the rendered output must be equivalent for a human sending the same headers.

Failure modes, in the order you will meet them

No Vary on a negotiated response. Symptom: intermittent reports of the wrong language or layout, impossible to reproduce, and correlated with cache warmth rather than with the user. Check whether a CDN sits in front of the origin and what its cache key is — many CDNs ignore Vary: User-Agent by default precisely because it is so destructive, which means declaring it is not enough on those platforms. You need a normalised cache key (for example, a device-class bucket) configured at the edge.

Vary: *. Legal, and it means “this response is not cacheable by a shared cache under any circumstances.” Occasionally intentional; usually pasted in by someone trying to fix a caching bug, and it removes edge caching for that URL entirely.

Vary on a redirect. A 301 that varies by language sends different clients to different destinations from the same URL. Now the URL’s canonical target depends on who asked, which is unresolvable — the crawler consolidates to whichever destination it was sent to.

Compression only. Vary: Accept-Encoding is the one nearly every server sets automatically and it is correct: gzip and Brotli bodies are genuinely different objects. It has no indexing implications at all. Do not let its presence convince you that negotiation is configured for anything else.

Image negotiation. Serving AVIF or WebP from the same image URL based on Accept is legitimate and needs Vary: Accept. This one rarely goes wrong, because the alternative — distinct URLs per format via <picture> — is also fine.

The diagnostic

Ask for the same URL twice with different request headers and compare. If the bodies differ, Vary must name the header you changed.

U=https://example.com/product/

# does the declared Vary match reality?
curl -sI "$U" | grep -i '^vary'

# same URL, different declared preference
curl -s -H 'Accept-Language: es' "$U" | md5sum
curl -s -H 'Accept-Language: en' "$U" | md5sum

Two different hashes with no Accept-Language in Vary is a bug you can fix today. Two identical hashes with Vary: Accept-Language declared is cache fragmentation you are paying for and not using.

What to prefer

For anything that affects indexing — language, region, substantive layout differences — use distinct URLs and annotate the relationship explicitly: hreflang between locales, a canonical between genuine duplicates. Distinct URLs are addressable, linkable, individually indexable, and they make the relationship visible in the response rather than dependent on the requester.

Reserve content negotiation for representations that are the same document in a different encoding: compression, image format, and — where you have to — a device-class template whose content is identical to its sibling.

The underlying reason is the one that runs through most canonicalisation work: a search engine consolidates signals onto a URL, and it can only do that if the URL means something stable. When the answer depends on who is asking, the declaration you made in a canonical tag is a hint, not a command is being evaluated against a moving target.