What a 429 Does to Your Crawl Rate

A 429 means the client has sent too many requests in a given window. Applied to a search engine crawler it works: the request rate drops. The cost is that a crawler which has learned your site refuses requests takes longer to come back up to speed than it took to slow down, and while it is throttled it is fetching your new and changed pages more slowly too.

That asymmetry is the whole decision. Rate limiting is a legitimate tool; it is just rarely the right first tool.

What the codes signal

429 is defined in the HTTP specification as too many requests from this client, optionally with Retry-After. It is client-specific by design: the server is not broken, this particular requester is being asked to slow down.

503 is a server-side temporary failure, described in serving 503 during planned downtime. It says nothing about who is asking.

Documented crawler behaviour is that both codes cause a reduction in crawl rate, and that the reduction persists while the responses continue. The distinction that matters operationally is what happens to the URL: neither code is treated as content, so a URL that returns 429 is not indexed with the rate-limit body, and existing index entries survive short episodes. What you lose is throughput, not coverage — as long as the episode is short.

A 429 is not a signal of low quality and does not carry a ranking consequence. It is a scheduling input.

How the throttle unwinds

The mechanism, as far as it is observable from the outside: a crawler maintains a per-host request rate that it adjusts based on response codes and response times. Errors and slow responses push the rate down. Clean, fast responses let it drift back up.

The drift up is gradual and not published. What you can observe in your own logs is the shape: request volume falls quickly after an episode of 429s begins, and recovers over a period measurably longer than the episode itself. Attributing an exact recovery curve to a specific site from log data alone is inference, so treat your own measurement as a description of your site rather than a general rule.

The practical consequence is that a rate limit you applied for an afternoon can leave the crawler cautious for longer than the afternoon. If your site publishes frequently, that shows up as slower discovery of new URLs.

When rate limiting a crawler is the right call

The crawler is genuinely the load problem and you can prove it. Not “traffic is high and some of it is bots” — you have grouped requests by verified user agent and reverse-DNS-confirmed source, and the search crawler is a top contributor to the requests that are hurting.

It is hitting an expensive surface. Almost always a generated one: faceted URLs, internal search, calendar pages, session-parameterised URLs. The fix here is not the rate limit, it is removing the surface — see faceted navigation and the crawl space it opens. The rate limit is what you do while you build the real fix.

You are mid-incident. The database is at capacity and shedding crawler load buys you the headroom to stay up for humans. Correct decision, and one you should undo the moment the incident ends.

When it is the wrong call

Because the crawl looks aggressive in a report. Request counts that seem alarming in isolation are often fine relative to the number of URLs you are publishing. Compare against your own URL count before concluding you are being hammered.

As a permanent configuration. A standing 429 rule against search crawlers is a decision to be crawled slowly forever, usually made once during an incident and never revisited. Grep your edge config for user-agent-based rate rules; the ones nobody remembers adding are the ones worth questioning.

Applied to robots.txt. Rate-limiting the file that describes your crawl rules means the crawler cannot read them, and the documented response to an unfetchable robots.txt is to back off from the host. Exempt it explicitly.

Instead of fixing slow responses. Response time is itself an input to crawl rate. A site whose pages take three seconds to render is already being crawled conservatively; adding a rate limit on top solves nothing that the latency was not already solving.

Doing it with a scalpel

If you must limit, limit the surface rather than the client. A rate rule keyed to the URL patterns that are expensive is narrower and does not teach the crawler anything about your site as a whole.

nginx, limiting one generated path:

# nginx: limit requests to internal search only
limit_req_zone $binary_remote_addr zone=search:10m rate=2r/s;

location /search {
    limit_req zone=search burst=5 nodelay;
    limit_req_status 429;
}

limit_req_status 429 is worth setting explicitly — nginx returns 503 by default, which mislabels a client-specific limit as a server outage.

Alongside it, add Retry-After if you have a real number, and make sure the limited paths are ones you would be happy never to have indexed. Better still: make those paths uncrawlable rather than rate-limited, so the requests never arrive.

Verifying who you are actually limiting

Most rate-limit rules aimed at “bots” are keyed on the user-agent string, which is trivially spoofed in both directions. The consequence is that a rule intended to slow a scraper claiming to be a search crawler also slows the real one.

Verify by source, not by string. Major crawlers publish IP ranges and support reverse DNS verification; a rule that checks the claimed identity against the source before acting will treat the real crawler and the impostor differently. Without that check you are guessing, and the guess is expensive in one direction.

# confirm what a limited path returns, and to whom
curl -sI -A 'Mozilla/5.0 (compatible; ExampleBot/1.0)' \
  https://example.com/search?q=test | head -3

What to watch afterwards

Pull request counts per day for the verified crawler from your logs, spanning the period before, during and after the limit. You want three numbers: the baseline rate, the throttled rate, and the number of days until the rate returns to baseline. That third number is the actual price of the decision, and it is the only one that is specific to your site.

If discovery of new URLs matters to you — a news site, a marketplace with fresh listings — measure that too, because slower crawl means slower first-fetch, and that is the consequence that reaches your readers rather than your dashboards. Where discovery is the priority, the lever to reach for first is not the rate limit but the sitemap, covered in what an XML sitemap is actually for.