Robots.txt Cannot Deindex a Page

robots.txt controls whether a crawler fetches a URL. It does not control whether that URL appears in search results. Those are two separate systems, and conflating them produces the single most common self-inflicted indexing problem in technical SEO: a page blocked from crawling that stays in the index indefinitely, because the directive that would have removed it is inside a file the crawler is no longer allowed to read.

The mechanism is worth stating precisely, because the fix is counter-intuitive.

Two different questions

Crawling is fetching the URL. robots.txt governs this, and it governs it before the request happens — a Disallow match means the crawler does not send the request at all.

Indexing is deciding whether the URL is eligible to appear in results. noindex governs this, delivered either as a <meta name="robots"> tag in the HTML or as an X-Robots-Tag HTTP header.

A noindex directive lives in the response. To read it, the crawler has to make the request. If robots.txt blocks the request, the noindex is never seen. The two directives are not alternatives at different strengths; they operate at different stages, and one of them can prevent the other from being delivered.

Why a blocked URL can still rank

A search engine can know a URL exists without ever fetching it. Other sites link to it. Your own sitemap lists it. Your internal navigation references it. Any of those is enough to put the URL in the index as a known address.

What the engine lacks is the content, so the result appears without a useful description — historically with wording along the lines of no information being available for the page. As of this writing the exact presentation varies, but the durable point is: a blocked URL is an indexable URL with no content, not an absent URL.

That is worse than either alternative. It is in results, and you have no ability to influence what it says, because influencing that would require the crawler to read something.

The correct instrument for each intent

“Don’t waste crawl requests on this.” robots.txt Disallow. Correct for generated surfaces — internal search, session-parameterised URLs, infinite calendars, faceted combinations. You do not care whether they are indexed because nothing links to them; you care that they are not fetched. See faceted navigation and the crawl space it opens.

“Don’t show this in results.” noindex, and the URL must remain crawlable. Correct for thin-but-necessary pages, internal utility pages, staging hosts, and anything a human is meant to reach by link but not by search.

“This is a duplicate of another URL.” A canonical tag, and again the URL must stay crawlable, since the tag lives in the response. Its limits are covered in canonical tags are a hint, not a command.

“This URL is gone.” A 404 or 410. Also requires the fetch, for the same reason.

“This URL moved.” A 301. Same again — a blocked redirect is a redirect nobody reads.

Notice the pattern: every directive except Disallow requires the crawler to make the request. Disallow is the only tool that works by preventing communication, which is exactly why it cannot express anything except “don’t ask.”

Unwinding the mistake

You have URLs in the index with no description, and they are Disallowed. The sequence:

1. Remove the Disallow rule. This is the step that feels wrong and is required. You have to let the crawler in to tell it anything.
2. Serve the real directive on those URLs — noindex if they should not be in results, a 301 if they moved, a 410 if they are gone.
3. Wait for a recrawl. The URLs have to be fetched before the directive takes effect, and a URL that has been blocked for a long time may be crawled infrequently. A sitemap listing them, temporarily, speeds up discovery.
4. Confirm the directive was read — check the URL in Search Console’s inspection tool, or just verify the header and tag with a request.
5. Only then, if you also want to stop the fetches, re-add the Disallow — and only once the URLs have actually dropped out. Re-adding it too early freezes them in place again.

Step 5 is optional and usually unnecessary. Once a URL is de-indexed and nothing links to it, it will be crawled rarely enough that blocking it buys little.

Where the two get combined by accident

Blocking a directory that contains redirects. A Disallow: /old/ added during a migration means none of the 301s under /old/ are ever followed. The redirect map is correct and invisible. This is a routine way for a migration to lose links despite the redirects being in place — see preserving links through a site migration.

Blocking CSS or JavaScript. Not an indexing problem in itself, but it prevents the crawler from rendering the page as a user sees it, which affects how the content is assessed. There is rarely a reason to block assets.

Blocking a URL that carries a canonical to your preferred version. The consolidation you declared never happens, because the tag is unread. Both URLs stay known.

A staging host with both. A Disallow: / on staging plus a noindex looks belt-and-braces and is actually mutually exclusive: the block means the noindex is never read, so if the staging host ever gets linked from anywhere, it can appear in results as an undescribed URL. On a staging host the right configuration is HTTP authentication, which returns 401 and settles the question at the transport layer.

A minimal, honest robots.txt

User-agent: *
Disallow: /search
Disallow: /cart
Disallow: /*?sort=

Sitemap: https://example.com/sitemap-index.xml

Three blocks, each one a generated surface nobody links to, plus a sitemap reference. Everything else about indexing is expressed in responses, where it can be read.

Note that robots.txt has no Noindex directive. Some servers historically honoured one and it was never part of the standard; a Noindex: line in robots.txt today is a comment with extra steps. If you find one in an inherited file, it has been doing nothing.

The check

For any URL you want out of search results, ask: can the crawler fetch this? If the answer is no, nothing you have written in the response matters, and the URL’s presence in results is not under your control.

# does the response actually carry a noindex?
curl -sI https://example.com/private-page | grep -i x-robots-tag
curl -s https://example.com/private-page | grep -i 'name="robots"'

If neither returns anything, the page has no indexing directive at all — and if it is also Disallowed, you have the combination this post exists to describe. The header-based half of that check has its own uses, covered in X-Robots-Tag and files with no head element.