Rate-Limiting Scraper Traffic Without Blocking Real Crawlers

The instinct, once a spike of automated traffic shows up in a dashboard, is to cap requests per IP or per user agent and move on. That works right up until the cap catches a client you didn’t mean to throttle — most often a legitimate search crawler that happens to be running a deep pass the same day a scraping API or an aggressive bot is hitting the same endpoints.

What a 429 does to your crawl rate covers what happens once you’ve decided to throttle a given client. This is the step before that: deciding which client a given request belongs to, so the limit lands on the traffic you actually intended to slow down.

Why “requests per IP” isn’t the right key

A rate limit needs a key — some property of the request that groups it with the same client’s other requests. IP address is the obvious choice and the wrong one for several classes of client at once:

Major search crawlers operate from large, shared IP ranges, and a limit keyed on a single address either does nothing (the crawler simply uses a different address in the range for its next request) or, if you limit the whole range, throttles a client you had no intention of touching.

A paid scraping or SERP API typically runs from cloud infrastructure shared with unrelated traffic. The IP tells you it’s a data center request, not which service is making it or what it’s fetching on whose behalf.

A distributed scraper rotates addresses specifically to avoid an IP-keyed limit. If avoiding your rate limit is cheap, an IP key only slows down the traffic that wasn’t trying to evade it.

Verify before you key

The reliable key is not the address alone but the address confirmed against the identity it claims, using the same method covered in reading log files for crawl evidence: reverse DNS on the IP, forward DNS on the result, and a match against the hostname the operator publishes.

dig +short -x 66.249.66.1
# → crawl-66-249-66-1.googlebot.com.
dig +short crawl-66-249-66-1.googlebot.com
# → 66.249.66.1

A request that verifies as a named search crawler gets its own limit — generous, or none at all, depending on how much load you can absorb. A request that claims to be that crawler in its user agent but fails the DNS check is, by definition, not that crawler, and can be rate-limited or blocked without touching the real one.

Everything else — verified scraping APIs that publish their own IP ranges, unverified automated traffic, and everyone else — gets whatever limit you’d otherwise have applied to all automated traffic indiscriminately.

A minimal shape, at the edge

The mechanics differ by platform, but the decision tree is the same regardless of where you implement it:

if user-agent claims major crawler:
    if reverse+forward DNS confirms it → allow, no limit (or a high one)
    else → treat as unverified automated traffic
if IP is in a published scraping-API range → apply that vendor's documented limit
else → apply your default automated-traffic limit

Doing the DNS check per request is too slow for high-traffic sites; the practical version resolves against a cached, periodically refreshed list of verified ranges rather than performing a live lookup on every hit. This is the same operational trade-off covered in the log-reading post: resolve once against a range list, not twice per line.

What a coarse limit costs you if you skip this

If a limit is applied by user agent string alone, anything can claim to be Googlebot and inherit its exemption — which means the limit either does nothing (spoofed strings sail through) or you tighten the string match and start catching genuine crawler requests that vary slightly in formatting between crawler versions.

If a limit is applied by raw IP count with no verification step, a shared-range crawler gets throttled alongside whatever else shares that range, and you have no way to tell, after the fact, whether the drop in indexing activity that follows was your rate limit or something else — because you never established which client the limit actually hit.

Where the limit sits changes what it can key on

A limit implemented at the CDN or edge layer sees the raw request before your application does, which is usually where the verified-crawler allowlist is cheapest to apply — the edge already terminates the connection and can hold a cached range list without a round trip to origin. A limit implemented in application code sees whatever the framework’s request object exposes, which is often the same headers but at a higher latency cost per check.

The two layers can also disagree, which is worth checking for rather than assuming away: an edge rule that rate-limits before a request reaches your application, and an application-level limit keyed differently behind it, can each believe the other is handling verification. The result is a client that passes the edge check and then gets throttled anyway by an application limit keyed on plain IP count, or the reverse — a client the application would allow getting stopped at the edge before its identity is ever evaluated. If a rate limit isn’t behaving as configured, checking both layers separately, rather than assuming a single limit is in effect, is the first diagnostic step.

What this doesn’t fix

A rate limit changes how much of a given client’s traffic you serve; it doesn’t change what that client does with a 429 once it gets one, which is the separate question the crawl-rate post addresses. And it doesn’t tell you whether an unverified client is honoring the pace you’d prefer even without a hard limit in place — that’s a question about behavior over time, not about a single request’s identity, and it’s worth checking separately once the key is right.

The durable point

A rate limit is only as precise as the key it’s built on. Keying on a verified client identity instead of a raw address or a self-reported string is what lets you throttle the traffic you mean to throttle without also throttling the crawler you’re trying to stay indexed by.