Link Plumbing

Keeping a Staging Host Out of the Index

A staging host is a complete copy of your site at a different address. That is the whole problem: every URL on it duplicates a URL on production, it usually has no reason to be crawled, and the two most common attempts to keep it hidden — a noindex tag and a Disallow: / — are respectively fragile and actively counterproductive.

The instrument that actually works is HTTP authentication, and the reason is that it answers the question one layer below where indexing directives operate.

How staging gets discovered

Nobody links to staging.example.com on purpose. It gets found anyway:

  • Absolute URLs in copied content. Staging is usually a database restore from production, and any absolute URL that was rewritten to the staging host on import — in body content, in a sitemap, in a canonical tag, in an RSS feed — is a link.
  • The reverse: staging URLs leaking into production. An editor writes a post on staging, someone copies the finished HTML to production, and now a live page links to staging. This is the single most common vector, and it is a real, followable, external-looking link.
  • Certificate transparency. Issuing a TLS certificate publishes the hostname to a public append-only log, from which anyone can enumerate a domain’s hostnames. staging. and dev. are the first two guesses anyway.
  • A pasted link. In a ticket, a public repository, a chat channel with a link unfurler.

Discovery is not the thing you can control. What the host does when something arrives is.

Why noindex alone is fragile

A noindex on staging works, in the sense that it is the correct directive and it will keep the host out of results as long as it is served on every response.

The fragility is in “every response.” A noindex injected by application middleware disappears the moment a response bypasses that middleware: a static file, a PDF, an error page rendered by the web server rather than the app, an endpoint returning JSON. And the flag is usually driven by an environment variable, so a mis-set variable on one deploy silently removes it — while a mis-set variable in the other direction ships noindex to production, the same bug with a much worse blast radius.

It also does nothing about the underlying issue: staging is serving unreleased content to anyone who asks. Indexing is a symptom, not the disease.

Why Disallow: / is worse

Disallow: / on staging feels like the strongest possible statement. It is the weakest useful one, because it prevents the crawler from reading anything — including a noindex.

The result, if the host is ever linked from anywhere, is a URL that is known, blocked, and therefore indexable-without-content: present in results, undescribed, and beyond your influence, because influencing it would require the crawler to fetch a response it is not permitted to fetch. The full mechanism is in robots.txt cannot deindex a page.

Belt-and-braces is the trap here. Disallow: / plus noindex is not two defences; it is one defence with the other one disabled.

401 settles it at the transport layer

HTTP authentication in front of the entire staging host — Basic auth at the web server or edge, or an identity-aware proxy — returns 401 Unauthorized to any unauthenticated request. There is no body worth indexing, no directive to interpret, and no content disclosure. A crawler receiving a 401 has nothing to index and no reason to retry aggressively.

What makes it categorically better than a directive:

  • It is not per-response logic. It sits in front of the application, so it covers static files, error pages, JSON endpoints, redirects, and anything else the app does not know it is serving.
  • It fails closed. A misconfiguration usually results in you being locked out, which you notice within a minute. A noindex misconfiguration results in nothing visible.
  • It solves the confidentiality problem too, which the indexing directives never touched.

Implementation notes worth stating precisely:

# nginx: whole-host auth, with a deliberate exception for health checks
server {
    server_name staging.example.com;

    auth_basic           "staging";
    auth_basic_user_file /etc/nginx/staging.htpasswd;

    location = /healthz {
        auth_basic off;
        return 200 "ok\n";
    }
}

403 Forbidden from an IP allowlist is equivalent for this purpose — same properties, no shared password, and the better choice when access is by network rather than by person. Either way, put the control at the edge or the web server, not inside the application, so a broken deploy cannot remove it.

Two things not to do: do not exempt /robots.txt from the auth (an accessible robots.txt on an otherwise-401 host is just an inventory of your paths), and do not exempt paths by user agent so a monitoring tool can get in — that turns a transport-layer control back into a user-agent check.

Cleaning up a host that is already indexed

If staging URLs are already in results, adding authentication first is the wrong order. A 401 does eventually lead to removal, but it gives the crawler no information beyond “you cannot see this,” and if the URLs were also Disallowed you have simply replaced one wall with another.

The sequence that resolves it:

  1. Remove any Disallow on staging. The crawler has to be able to fetch in order to be told anything. This step feels wrong every time.
  2. Decide what each staging URL should say. For a host that should never have been public, 410 Gone on everything is the cleanest statement — it is a definite claim rather than an ambiguity, per 404 versus 410 and how long a dead URL lingers. If any staging URL accumulated actual inbound links, 301 that specific URL to its production equivalent instead.
  3. Or serve noindex via header if you need the host to stay browsable for the team during the cleanup. X-Robots-Tag: noindex at the server level covers every response type, including the ones with no <head> to put a tag in — see X-Robots-Tag and files with no head element.
  4. Fix the links that caused it. Search production for the staging hostname in content, sitemaps, canonical tags and feeds. Until those are gone, the crawler has a standing invitation.
  5. Confirm removal. Watch the host’s coverage in Search Console, or inspect specific URLs, until they are reported as excluded rather than indexed.
  6. Then add the authentication. Once the URLs are out, close the door — and only then, because closing it earlier freezes whatever state they were in.

Step 5 before step 6 is the part people skip, and skipping it is why staging hosts sometimes sit in the index for months behind a 401.

The canonical question

A common suggestion is to point every staging page’s canonical at its production twin. As a temporary measure during cleanup it is harmless and mildly helpful — it tells the crawler which URL you consider authoritative. It is not a solution, for two reasons: a canonical is a hint, not a command, and it does nothing about the content being publicly readable.

There is also a mirror-image failure worth knowing about. Because staging is usually restored from a production database, staging pages frequently carry canonicals pointing at production by accident, which is the harmless direction. The dangerous direction is a production deploy that inherits staging’s configuration and emits canonicals pointing at staging — a host that is 401, or noindex, or blocked. Every affected production page then declares its authoritative version to be a URL the crawler cannot index. If canonical targets suddenly appear as unreachable in your reporting, check the environment configuration before anything else; the decision framework for which directive belongs where is in canonical, redirect, or noindex.

The check

One command per environment, run in CI if you can:

# staging must be 401 (or 403) at the root and on a deep path
for p in / /sitemap.xml /wp-content/uploads/x.pdf; do
  curl -s -o /dev/null -w "$p %{http_code}\n" "https://staging.example.com$p"
done

# production must NOT be canonicalising to staging
curl -s https://example.com/ | grep -i 'rel="canonical"'

Two assertions: nothing on staging returns 200 to an anonymous request, and no production response mentions the staging hostname. Both are cheap enough to run on every deploy, and between them they close the loop that keeps re-opening.

httpcanonicals