Does robots.txt Actually Stop a Paid Scraping API?

robots.txt is a text file at a fixed URL. Nothing about HTTP requires a client to fetch it, and nothing about HTTP enforces what it says once fetched. Whether it stops a given scraping API’s requests is a question about that vendor’s policy, not about the protocol.

Robots.txt cannot deindex a page covers the indexing side of this same fact — Disallow controls crawling, not what shows up in results. This is the adjacent question, from the other direction: does Disallow even stop the fetching, for a client that isn’t a search engine at all.

The file is a convention, not a lock

robots.txt exists because major search crawlers agreed, as a matter of documented policy, to fetch it first and skip anything it disallows. That agreement is what makes the file mean anything for Googlebot or Bingbot. It is not a mechanism the origin server enforces — the server still returns a normal 200 and the requested body to any client that asks for a disallowed path, robots-compliant or not.

A paid scraping or SERP API sits outside that agreement by default. Some vendors document that they honor robots.txt as a matter of policy; others document that they don’t, because their customers are asking them to fetch a specific page on demand and a blanket Disallow would break that product. Both are legitimate, disclosed positions — the point is that it’s a per-vendor choice, not a property of the file.

What actually determines the outcome

Read the vendor’s own documentation before assuming either way. A scraping API built for on-demand, customer-directed fetches (fetch this one URL right now) has a different reason to ignore robots.txt than a general-purpose crawler that walks a site on its own schedule — the former isn’t crawling in the sense the file was written to govern.

A Disallow entry is not evidence of compliance you can verify from your own side without checking. Seeing a drop in requests to a disallowed path after adding it tells you a well-behaved client changed course. It does not tell you every client requesting your site is well-behaved — see reading log files for crawl evidence for how to check which clients are actually hitting a given path, rather than inferring compliance from the directive alone.

Group matching still applies if a vendor does comply. If a scraping operator publishes its own user-agent token and states that it honors robots.txt, the same rule-matching and precedence mechanics apply to it as to any named crawler — a group written for Googlebot doesn’t cover it, and a wildcard * group does unless it has its own named group.

Why a vendor might document non-compliance as a feature, not an oversight

It’s tempting to assume any scraping API that ignores robots.txt is simply being careless about a norm it should respect. For a class of product built around on-demand, customer-directed fetches, that’s usually not what’s happening.

Consider the difference between a general crawler that walks a site’s link graph on its own schedule to build an index, and an API that fetches exactly one URL because a customer’s application asked it to, right now, in response to something a human did. robots.txt was written to govern the first case — an automated agent deciding for itself what to visit and how often. The second case has a human decision behind every fetch; the API is acting as a proxy for a request that would have happened anyway had the customer’s own server made it directly. A vendor documenting “we do not consult robots.txt for on-demand fetches” is describing that distinction, not disclosing a violation.

This doesn’t mean every scraping API takes that position, or that the distinction is airtight — a vendor’s documentation is the only reliable source for which category its product falls into, and some products blend both modes (scheduled bulk crawling alongside on-demand single fetches) with different policies for each.

What actually stops unwanted fetching

If the goal is to prevent a specific client from reaching a path — rather than to ask it not to, and hope — robots.txt is the wrong tool regardless of which vendor is on the other end. It communicates intent to clients that choose to read it; it does not gate access.

Mechanisms that do gate access sit lower in the stack: authentication in front of the path (covered for a related case in staging hosts and HTTP authentication), or a rate limit keyed on a verified client identity as in rate-limiting scraper traffic without blocking real crawlers, which acts on every request regardless of whether the requester consulted robots.txt first.

What you can and can’t conclude

You can conclude, from a vendor’s published policy, whether that specific vendor’s product is designed to respect Disallow. You can’t conclude anything about a vendor that hasn’t documented a position, and you can’t treat the absence of a drop in traffic after adding a rule as proof that a given client ignored it — it might simply not have made a request in that window for unrelated reasons.

The honest summary

robots.txt changes behavior only for clients that have chosen, as policy, to read and follow it. For search crawlers that’s a long-standing, documented agreement. For a paid scraping API, it’s whatever that vendor says it does — worth checking directly, and worth backing up with server logs rather than assuming either compliance or defiance from the file’s presence alone.