Tagged “http”
-
An Intermittent 404 Is Worse Than a Permanent One
A URL that answers 200 on refresh and 404 under load teaches a crawler nothing it can act on. What gets recorded, and how to make errors deterministic.
-
One robots.txt Per Host, Not Per Folder
A crawler fetches exactly one robots.txt per host, at the root. A file your CMS serves under /blog/ is inert — and the root rules still apply to /blog/.
-
Does robots.txt Actually Stop a Paid Scraping API?
Disallow is a request a well-behaved client honors, not a technical barrier. Whether a scraping API respects it depends entirely on that vendor's policy.
-
How to Tell Scraper Traffic From Real Crawlers in Your Logs
Unfamiliar bot traffic in your access logs is often a third-party scraping API, not Googlebot or a hostile attack. Here's how to tell the difference.
-
Rate-Limiting Scraper Traffic Without Blocking Real Crawlers
A blanket rate limit throttles Googlebot along with everything else. How to key a limit on a verified client instead of a raw request count.
-
What a Redirect Chain Looks Like From a Scraping API's Perspective
A scraping API fetching a URL on someone else's behalf has to decide how many hops to follow and what to hand back. Both choices are easy to miss.
-
How Do You Verify a Scraper Is Actually Respecting Your Crawl-Delay?
Crawl-delay isn't enforced by any protocol. Confirming a client honors it means measuring request intervals in your own logs, not trusting the directive.
-
Why an SEO Tool Might Report a Different Status Code Than Your Browser
The same URL, checked two ways, can legitimately return two different status codes. Why that happens and which one to trust for which question.
-
Keeping a Staging Host Out of the Index
Why noindex and Disallow both fail on a staging site, why 401 settles it, and how to clean up a staging host that is already indexed.
-
Navigation That Only Exists After JavaScript Runs
A crawler follows anchor elements with href attributes. Click handlers, buttons and hash routes are not links, and the pages behind them go undiscovered.
-
Vary, Content Negotiation, and One URL Serving Two Pages
When a single URL returns different content per request, Vary is what tells caches and crawlers which version they are holding.
-
How Robots.txt Rules Are Matched
Group selection, path prefixes, wildcards, and the longest-match rule that decides whether Allow or Disallow wins for a given URL.
-
404 Versus 410 and How Long a Dead URL Lingers
Both codes mean the page is gone. One says it might come back. What each does to recrawl scheduling, and when the difference is worth the config.
-
Soft 404s: When a 200 Response Is Treated as Missing
A soft 404 is a page that returns 200 but reads as empty. What triggers the reclassification, how to find them, and how to fix each cause.
-
Canonical Tags Are a Hint, Not a Command
Why search engines ignore canonicals, which signals outvote them, and how to make your declared canonical the one that gets picked.
-
301 Versus 302 and What Each Does to Link Equity
What the four redirect status codes mean, how crawlers treat them, and why the 85%-decay figure you've read has no source.
-
Serving 503 During Planned Downtime
How to take a site offline without losing index coverage: what 503 tells a crawler, how long it stays tolerant, and the codes that go wrong instead.
-
What a 429 Does to Your Crawl Rate
Rate limiting a crawler is a legitimate move with a delayed cost. How 429 and 503 affect request rate, and how to throttle without going dark.
-
Robots.txt Cannot Deindex a Page
Disallow controls crawling, not indexing. Why blocked URLs still appear in results, and why blocking hides the noindex that would have worked.
-
X-Robots-Tag and Files With No Head Element
A meta robots tag needs HTML. PDFs, images and feeds have none. How the header version works, and how it combines with the tag on the same URL.
-
Noindex and Nofollow on the Same Page
Page-level nofollow is not the same as rel=nofollow on a link, and combining it with noindex has a consequence people don't expect.
-
Trailing Slashes Are Different URLs
/page and /page/ are two addresses that can serve two responses. How to pick one, enforce it in a single hop, and stop generating both.
-
Uppercase URLs and the Duplicates They Create
Paths are case-sensitive, hosts are not. Where mixed-case URLs come from, what they duplicate, and how to normalise without breaking real files.
-
Consolidating www and HTTPS in a Single Hop
Four host and protocol variants, one canonical address. Why separate rules chain, how to combine them, and where HSTS fits.
-
Query Parameters That Multiply Your URL Count
Tracking, sorting and session parameters each create a distinct URL. How to tell which ones need a canonical, a Disallow, or removing entirely.
-
What an XML Sitemap Is Actually For
A sitemap is a discovery hint, not an indexing instruction. What it can fix, what it cannot, and why lastmod is the only optional field worth setting.
-
Reading Log Files for Crawl Evidence
Server logs are the only record of what a crawler actually requested and what you actually served. How to verify the client and what to count.
-
Meta Refresh and JavaScript Redirects
Client-side redirects happen after a 200 response. How crawlers treat them, when they are the only option, and the delay that makes one a dead end.
-
Redirect Chains and How to Flatten Them
Chains form by accretion, not by design. How to find them, why length matters, and how to flatten one that crosses three systems.