404 Versus 410 and How Long a Dead URL Lingers

A 404 means “not found.” A 410 means “gone, deliberately, and don’t expect it back.” In HTTP terms the difference is intent: 404 carries no claim about the future, 410 asserts permanence. Search engines treat both as a signal to drop the URL from the index, and the practical difference between them is smaller than most posts on the subject suggest.

What the choice actually affects is how quickly a crawler stops asking.

What the spec says versus what a crawler does

The HTTP specification defines 404 as the origin server not finding a current representation, explicitly noting that it does not say whether the absence is temporary or permanent. 410 is narrower: the resource is gone and the condition is expected to be permanent, and the spec suggests the code is intended for cases where the server owner wants remote links removed.

Search-engine handling is a separate question, and the two should not be blurred. Google has stated that it treats 404 and 410 nearly identically for indexing purposes, with 410 processed slightly faster because there is no need to hedge against the page reappearing. Both result in the URL dropping out of the index. Neither is a penalty, and neither harms the rest of the site — a 404 is a normal, expected response on any site that has existed for more than a year.

The recrawl behaviour is where you see a real difference, and it is a difference of degree. A crawler that receives a 404 will typically come back to check, sometimes for months, because the code is ambiguous about permanence. A 410 shortens that tail. Exactly how much is not published, so treat any specific number you read as inference rather than documentation.

When 410 is worth reaching for

The honest answer is: rarely, and only when you can express it cheaply.

Deliberate mass retirement. You are killing 40,000 auto-generated tag pages and you want them out of the index and out of the crawl queue without waiting through months of speculative revisits. A 410 states the intent precisely.

Content you are legally or editorially required to remove. The URL is not coming back and you want no ambiguity in the record.

Spam or hacked URLs you have cleaned up. Injected pages that never should have existed are the textbook 410 case — you want the crawler to stop probing them.

Everywhere else, the default 404 your platform already emits is fine. Rewriting an application’s error handling to return 410 for a handful of URLs is effort spent on a marginal scheduling difference.

What neither code does

Neither one moves link value. If a retired URL has inbound links, a 404 or 410 discards whatever those links were carrying. That is a decision, not an accident, and it should be made deliberately — see retiring a page that still has inbound links for the decision path.

Neither one is a redirect. A tempting shortcut is to redirect every dead URL to the homepage so nothing “404s.” That does not preserve anything; a redirect to an unrelated page is generally reclassified as a soft 404 and you have added latency for no benefit.

Neither one removes the URL from your sitemap. If a dead URL is still listed in sitemap.xml, you are actively telling the crawler to fetch something you have told it is gone. Clean both.

Serving them correctly

The commonest failure is not choosing the wrong code — it is serving the right words with the wrong code. A page that says “Sorry, that page doesn’t exist” over an HTTP 200 is invisible to a crawler as an error, because crawlers read the status line, not the copy.

Check what you actually emit:

curl -sI https://example.com/deleted-page | head -1

For nginx, returning 410 for a known-dead prefix:

# nginx: retire an entire path prefix
location ^~ /old-tags/ {
    return 410;
}

For Apache, via .htaccess:

# Apache: retire specific URLs
RedirectMatch gone ^/old-tags/.*$

Both are narrow by design. Be careful with broad prefix matches: a location ^~ /t/ rule intended for tag pages will also swallow any future URL under that prefix, and the failure is silent because a 410 looks intentional to everyone reading logs afterwards.

The error page itself

A useful 404 page is still a 404. Serve the status code, and serve something helpful in the body: site search, the top-level sections, and a link to the section the requested URL appeared to belong to. Parsing the requested path and offering the nearest surviving parent is a small amount of work that converts a dead end into a navigation step.

What not to do:

Do not redirect to a generic error page. /missing returning 200 with an error message means every dead URL on the site resolves to one indexable page. Serve the error at the requested URL.
Do not noindex your 404 page. It is unnecessary — the status code already keeps it out — and it means that if the template is ever served with a 200 by mistake, you have two problems instead of one.
Do not block the error path in robots.txt. Blocking the crawler from fetching a URL prevents it from seeing the 404 at all, which is the opposite of what you want. This is the same trap described in robots.txt cannot deindex a page.

Reading the aftermath

After a deliberate retirement, the URLs move through reporting in stages: crawled and returning the error, then dropped from the index, then eventually crawled less often. The middle stage is the one people panic about, because an “excluded” or “not found” count climbing into the thousands looks like damage. It is the intended outcome of the change you made.

The number worth watching instead is inbound requests to the retired paths, from logs. If real referrers are still sending traffic to a URL you killed, that is evidence you retired something with an audience — and the right response is a redirect to a genuine equivalent, not a better error page.

One rule that holds up: decide whether a URL is gone or moved before choosing a status code, and never let “gone” be the default because it required no configuration. Gone is the correct answer often enough to be honest about; it is just rarely the correct answer for a URL anyone ever linked to.