How to Tell Scraper Traffic From Real Crawlers in Your Logs

Traffic from an unfamiliar user agent or a rotating set of residential IPs in your access logs is not automatically Googlebot, and it’s not automatically an attack either. Most of the time it’s a paid scraping or search-data API, hitting your pages on behalf of a customer who is running an SEO tool, a price monitor, or a research script — not indexing you for search, just reading you for data. Telling that apart from real search-engine crawling changes what, if anything, you should do about it.

Why this traffic exists at all

There’s a large commercial market of vendors that fetch web pages programmatically and resell the results as a service — rank-tracking data, search-result snapshots, general-purpose page scraping. Serply’s SERP API pricing comparison maps that market by category: real-time SERP APIs, enterprise scraping infrastructure (Bright Data, Oxylabs, Decodo, NetNut, Zyte), and general scraping platforms (Apify, ScrapingBee, ZenRows, ScraperAPI). Some of those fetch Google’s result pages on a customer’s behalf; others fetch arbitrary sites, including yours, because a customer pointed a script at you. Either way, the request in your log wasn’t sent by a search engine deciding to crawl you — it was sent by infrastructure a third party is renting.

Distinguishing the traffic

Verified search-engine crawlers self-identify and are checkable. Googlebot’s IP ranges are published and reverse-DNS-verifiable; Bingbot equivalents exist too. If a hit claims to be Googlebot but doesn’t resolve back to Google’s published ranges, it isn’t Googlebot — full stop, regardless of what the user-agent string says.

Scraping infrastructure often runs on residential or datacenter proxy pools, not a fixed, documented range. That’s the entire point of enterprise scraping infrastructure — it exists to look like ordinary browser traffic, distributed across many IPs and ASNs, specifically so it isn’t easy to block as a block. A burst of requests to the same URL pattern from many unrelated-looking IPs, with no referrer and a generic or spoofed browser user agent, is the signature.

Rate and pattern matter more than the user agent. A real crawler paces itself against your robots.txt crawl-delay signals and your server’s response times, covered in what a 429 does to your crawl rate. A scraping job commissioned by someone else has no reason to know or respect those signals, so it’s more likely to hit a narrow set of URLs repeatedly in a short window.

# Example: isolate hits by user-agent and check ASN/reverse-DNS before assuming
grep "GET /pricing" access.log | awk '{print $1, $NF}' | sort | uniq -c | sort -rn

A worked example

Say your logs show 400 requests to /pricing/ in an hour, spread across 60 distinct IPs, no referrer, no session cookies, and a user agent string that looks like a normal desktop browser. Compare that to a verified Googlebot hit on the same page: it comes from a published Google IP range, respects your crawl-delay if you set one, and typically spaces requests to the same URL hours or days apart, not dozens per hour.

The 400-request pattern doesn’t match either a single motivated human or a verified search engine. It matches a scraping job: something requested that page many times, from many vantage points, probably because a customer’s script told it to check that URL on a schedule — a price monitor, a competitive-research tool, or an SEO tool doing a periodic audit crawl of a list of competitor pages that happens to include yours. The distributed IPs and browser-shaped user agent aren’t an accident; that’s what buying access to residential or rotating datacenter proxy pools looks like from the receiving end, and it’s a normal, advertised feature of the enterprise scraping category, not evidence of anything being hidden from you specifically.

What this traffic means for your logs and your crawl budget

It is not evidence of a manual action, a hostile actor, or a search-engine penalty — it’s evidence that your page is a data source somebody found useful enough to fetch programmatically, which more often than not means your content is being read by an SEO or research tool, not attacked. See reading log files for crawl evidence for what log patterns actually do and don’t tell you about how a page is treated.

It does consume server resources and crawl-adjacent capacity the same way any other automated traffic does, so if volume becomes a real load problem, the fix is the same one you’d apply to any high-volume automated client: rate limiting at the edge, not a robots.txt disallow, which only asks politely and only binds crawlers that choose to honor it — see robots.txt cannot deindex a page for the same limitation applied to a related problem.

Where enterprise scraping infrastructure differs from a plain SERP API

It’s worth separating the two categories Serply’s comparison lists, because they show up in your logs differently. A pure SERP API — the kind a rank tracker buys — fetches Google’s result pages on a customer’s behalf; you’d only see that traffic in your own logs if your page happened to be the result being fetched (someone tracking a keyword you rank for), and even then it’s Google’s crawler doing the actual fetch, not the API vendor’s. Enterprise scraping infrastructure and general scraping platforms are different: they fetch arbitrary target pages, including yours directly, at a customer’s instruction. If your logs show direct hits to specific internal pages rather than just your homepage, you’re almost certainly looking at the second category, not the first.

What to actually do

1. Verify identity before reacting — reverse-DNS or published IP ranges for claimed search-engine bots, nothing assumed from the user-agent string alone.
2. If it’s unverified and high-volume, rate-limit by IP/ASN at the edge rather than relying on robots.txt.
3. Don’t read scraper traffic as a ranking signal in either direction — it’s evidence your page is reachable and worth fetching, not evidence of anything Google itself decided.