Link Plumbing

An Intermittent 404 Is Worse Than a Permanent One

A URL that reliably returns 404 is a clean signal: the resource is gone, the crawler drops it from the index after enough confirmations, and the story ends. A URL that returns 404 on one fetch and 200 on the next is a much worse position to be in, because every consumer of that URL — crawler, monitoring check, SEO tool, a person clicking a link — records whichever answer it happened to get, and none of them are wrong.

The usual shape of the report is “some pages 404 when we open twenty at once, then load fine on refresh.” That’s a concurrency or cold-start failure, not a routing failure, and the visible symptom is a status code that varies by request rather than by URL.

What a crawler stores

A crawl is a series of independent samples. There’s no notion of “the URL’s true status” — there’s the status returned at the moment of each fetch, plus whatever history has accumulated.

Consequences worth being precise about:

  • A single 404 doesn’t deindex a page, and it isn’t supposed to. Transient errors are common and crawlers are built to tolerate them; a URL generally has to keep answering 404 across multiple fetches before it’s dropped. That tolerance is what makes an intermittent 404 survivable at all.
  • But it does spend the fetch. The crawler used a request on that URL and got nothing indexable. Whatever recrawl interval it had for the page, it now has less recent content than it would have.
  • The sampling is invisible to you unless you log it. Analytics won’t show it, because a 404 to a crawler isn’t a page view. Only the server or edge log has the row.
  • Different clients disagree, legitimately. A page can be fine in your browser and broken in a tool five seconds later, for exactly this reason. See Why an SEO Tool Might Report a Different Status Code Than Your Browser for the other reasons that happens.

Compare that with the deliberate cases. 404 versus 410 is a choice about how quickly you want a dead URL to stop being retried. Serving 503 during planned downtime is a choice to say “not now, come back” instead of “not here.” Both are deterministic: the same URL gives the same answer to everyone, and the crawler can act on it. An intermittent 404 gives it nothing to act on.

Why 404 is the wrong code for a failure

This is the part worth fixing regardless of the underlying bug.

A 404 means the resource does not exist. If your application couldn’t resolve a route because a worker was cold, a database connection was exhausted, or twenty concurrent requests hit a process pool of four, the resource exists perfectly well — you failed to serve it. The honest codes for that are in the 5xx range: 503 Service Unavailable for “temporarily can’t”, 500 for “something broke”. Both tell a crawler to come back; 404 tells it there’s nothing to come back for.

The same applies to the flapping case people see in the other direction — a product page that answers 503 and then 200 on immediate retry. That one is at least telling the truth, which is why it’s the less damaging of the two.

If you can only make one change, make the failure path return 503 with a Retry-After:

HTTP/1.1 503 Service Unavailable
Retry-After: 120
Cache-Control: no-store

no-store matters. A cached error is how a two-second blip becomes a two-hour one, and a CDN caching a 404 is the single most common reason an intermittent failure appears to have persisted long after the underlying cause was fixed.

Finding it

Reproduce it with concurrency, because serial requests usually pass:

# 40 requests, 8 at a time, print only the status of each
xargs -P 8 -I{} curl -s -o /dev/null -w "%{http_code}\n" \
  https://example.com/some-page/ < <(seq 40)

A run that prints a mix of 200 and 404 has reproduced the bug in one command. Vary the URL set too — a failure that only appears when the twenty URLs are different points at a per-request resource (a connection, a lambda instance, a render worker) rather than at one broken route.

Then confirm which layer answered. Compare the response headers of a good and a bad fetch: a 404 carrying your application’s HTML error template came from the app; a 404 carrying an edge or platform error page came from the CDN or host, and the app may never have been reached. Cache headers (cf-cache-status, x-vercel-cache, age) tell you whether you’re looking at a fresh failure or a stored one.

Then read the logs, which is where the actual rate lives. Reading Log Files for Crawl Evidence has the general approach; for this specific problem you want status counts per URL over time, filtered to crawler user agents, so you can see whether the crawler’s sample of your site looks like your monitoring’s sample. A 2% error rate you’d never notice in aggregate can be a 40% error rate on the URLs a crawler hits, if those URLs are the ones nobody has warmed.

What to fix, in order

  1. Stop returning 404 for failures. Whatever the root cause, this is a two-line change in the error handler and it converts an unrecoverable signal into a recoverable one.
  2. Don’t cache errors. no-store on 4xx and 5xx responses, and check that your CDN honours it. Verify with a repeated request and a cache-status header.
  3. Fix the concurrency limit. Process pool, connection pool, cold-start budget, per-instance memory — whatever the twenty-simultaneous-requests test exhausts. This is application work and it’s the actual bug.
  4. Then re-test. The same xargs loop, and a clean sweep of 200s across 100 requests before you believe it.

The recurring theme in the plumbing on this site is that crawlers respond well to consistency and badly to ambiguity. A URL that is honestly and permanently gone is a solved problem. A URL that can’t decide is the one that quietly costs you indexed pages while every manual check you run comes back fine.

http