What a Redirect Chain Looks Like From a Scraping API's Perspective

Redirect chains and how to flatten them covers chains from the perspective of the site that owns them — how they form, why length matters, how to find and collapse them. There’s a second perspective worth separating out: a scraping or SERP API that fetches a URL on a customer’s behalf has to make its own decisions about a chain it didn’t create and doesn’t control, and those decisions shape what the customer gets back.

The client has to choose a hop budget

Every HTTP client that follows redirects automatically enforces some limit on how many it will follow before giving up — a fixed number of hops, because an unbounded follow would let a redirect loop hang the request forever. That limit is a property of the client, not of the URL being fetched, which means the same chain can resolve cleanly for one API and fail as “too many redirects” for another with a lower budget.

A four-hop chain that a browser follows without incident can come back from a scraping API as an error, not because anything changed about the target site, but because the API’s client library drew its limit at three.

The client has to choose what to report

Once a chain resolves, the API has to decide what it hands back to the customer: the original URL requested, the final URL after all hops, or both. This matters concretely for anything downstream that keys off “the URL” as a single value — a rank-tracking integration recording a position, or a monitoring job checking that a URL still returns 200, gets a different answer depending on which URL the API considered canonical for the response.

Reporting only the final URL silently launders a chain — the customer sees a clean single result and never learns that three hops happened, which is one way stale integrations keep referencing an origin URL for years after a migration moved it, per preserving links through a site migration.

Reporting only the original URL with the final status hides where the response actually came from, which matters if the intent was to verify that a specific final destination is reachable, not merely that the starting point eventually resolves somewhere.

Reporting both — the original URL, each intermediate hop, and the final URL with its status — is the only version that lets the customer distinguish “resolves in one hop” from “resolves after four,” which is the distinction that matters for the reasons the flattening post covers: latency, fragility, and the chance a hop budget elsewhere in the pipeline gets exceeded.

Timeouts compound across hops

A chain is not one request with extra steps; it’s N sequential round trips, and most clients apply their timeout to the total fetch rather than generously to each hop. A slow intermediate hop — a server under load, a DNS lookup on a domain that’s since changed providers — can consume the whole timeout budget before the chain reaches its destination, producing a timeout error that has nothing to do with the final page being slow.

This is the same latency cost redirect chains and how to flatten them describes for a human visitor, applied to an automated client with a harder cutoff and no patience past it.

HTTPS-only clients add a hop you might not have counted

A migration that moves a site from HTTP to HTTPS, or that adds a www redirect on top of it, is usually described as “one redirect” from the perspective of a browser that’s used to both steps happening instantly. Some automated clients enforce HTTPS by refusing plaintext requests outright rather than following an upgrade redirect, which means the same URL that resolves in what a human would call one hop can fail differently depending on whether the API’s client attempted the plaintext request first or required HTTPS from the start.

This matters for the hop count specifically when a chain already combines a scheme upgrade with a path change — the pattern covered in consolidating www and HTTPS in a single hop. A chain built to resolve in one combined hop for exactly this reason will look fine to a scraping API with a normal hop budget; a chain that still does the scheme upgrade and the path change as two separate redirects consumes twice the budget for the same logical change, and an API with a lower tolerance is more likely to hit its limit on that URL specifically.

What to check if results look wrong

If a URL that resolves fine in a browser comes back from a scraping or monitoring API as an error, the chain — not the destination — is the first thing to check:

curl -sIL https://example.com/old-path | grep -E '^(HTTP|location)'

Count the hops. If the count is at or near whatever limit the API documents (check its own docs; this varies by vendor and isn’t a value worth guessing at), the chain is the likely cause, and flattening it per the sibling post fixes the API result along with everything else a long chain affects.

What you can and can’t conclude from an API error

You can conclude that the chain exceeded that particular client’s tolerance for hops or time. You can’t conclude the chain is broken in general — a different client with a higher hop budget, or the same client with a longer timeout, may resolve it without incident. An API’s redirect-following failure is evidence about that client’s limits, not a verdict on the URL.

The durable point

A redirect chain has one length, but every client that follows it applies its own budget for hops and time to that length. A scraping API’s report is a description of what its own client did with your chain, not an independent measurement of the chain itself — which is one more reason a chain worth having at all is a chain worth keeping short.