Blog
-
When Google Canonicalises Two Unrelated Pages Together
A Google-selected canonical from a different section usually means two pages share an identifier or a pre-render state. Here's how to find which one.
-
An Intermittent 404 Is Worse Than a Permanent One
A URL that answers 200 on refresh and 404 under load teaches a crawler nothing it can act on. What gets recorded, and how to make errors deterministic.
-
One robots.txt Per Host, Not Per Folder
A crawler fetches exactly one robots.txt per host, at the root. A file your CMS serves under /blog/ is inert — and the root rules still apply to /blog/.
-
Moving Content Between a Subdomain and a Subdirectory
What actually changes when blog.example.com becomes example.com/blog: host boundaries, robots and sitemap scope, redirect mapping, and geo signals.
-
URL Depth and Click Depth Are Different Things
Slashes in a path are not a crawl cost. Clicks from an entry point are. Here's how to measure each and why flattening URLs rarely changes anything.
-
Does robots.txt Actually Stop a Paid Scraping API?
Disallow is a request a well-behaved client honors, not a technical barrier. Whether a scraping API respects it depends entirely on that vendor's policy.
-
How to Tell Scraper Traffic From Real Crawlers in Your Logs
Unfamiliar bot traffic in your access logs is often a third-party scraping API, not Googlebot or a hostile attack. Here's how to tell the difference.
-
Rate-Limiting Scraper Traffic Without Blocking Real Crawlers
A blanket rate limit throttles Googlebot along with everything else. How to key a limit on a verified client instead of a raw request count.
-
What a Redirect Chain Looks Like From a Scraping API's Perspective
A scraping API fetching a URL on someone else's behalf has to decide how many hops to follow and what to hand back. Both choices are easy to miss.
-
How Do You Verify a Scraper Is Actually Respecting Your Crawl-Delay?
Crawl-delay isn't enforced by any protocol. Confirming a client honors it means measuring request intervals in your own logs, not trusting the directive.
-
Why an SEO Tool Might Report a Different Status Code Than Your Browser
The same URL, checked two ways, can legitimately return two different status codes. Why that happens and which one to trust for which question.
-
Cross-Domain Canonicals and When They Hold
A canonical tag can point at another hostname. What has to be true for that to be honoured, and the six ways it quietly gets ignored.
-
Keeping a Staging Host Out of the Index
Why noindex and Disallow both fail on a staging site, why 401 settles it, and how to clean up a staging host that is already indexed.
-
Navigation That Only Exists After JavaScript Runs
A crawler follows anchor elements with href attributes. Click handlers, buttons and hash routes are not links, and the pages behind them go undiscovered.
-
Vary, Content Negotiation, and One URL Serving Two Pages
When a single URL returns different content per request, Vary is what tells caches and crawlers which version they are holding.
-
How Robots.txt Rules Are Matched
Group selection, path prefixes, wildcards, and the longest-match rule that decides whether Allow or Disallow wins for a given URL.
-
404 Versus 410 and How Long a Dead URL Lingers
Both codes mean the page is gone. One says it might come back. What each does to recrawl scheduling, and when the difference is worth the config.
-
Soft 404s: When a 200 Response Is Treated as Missing
A soft 404 is a page that returns 200 but reads as empty. What triggers the reclassification, how to find them, and how to fix each cause.
-
Preserving Links Through a Site Migration
The redirect map is the migration. How to build one from the URLs that actually matter, and the order of operations on launch day.
-
Canonical Tags Are a Hint, Not a Command
Why search engines ignore canonicals, which signals outvote them, and how to make your declared canonical the one that gets picked.
-
301 Versus 302 and What Each Does to Link Equity
What the four redirect status codes mean, how crawlers treat them, and why the 85%-decay figure you've read has no source.
-
Serving 503 During Planned Downtime
How to take a site offline without losing index coverage: what 503 tells a crawler, how long it stays tolerant, and the codes that go wrong instead.
-
What a 429 Does to Your Crawl Rate
Rate limiting a crawler is a legitimate move with a delayed cost. How 429 and 503 affect request rate, and how to throttle without going dark.
-
Retiring a Page That Still Has Inbound Links
A decision path for killing a URL other sites point at: when to redirect, when to merge, when to leave it up, and when 410 is honest.
-
Robots.txt Cannot Deindex a Page
Disallow controls crawling, not indexing. Why blocked URLs still appear in results, and why blocking hides the noindex that would have worked.
-
Hreflang and the Return-Tag Requirement
Hreflang only works when every page in a cluster points back at every other. Why one-directional annotations are ignored, and how the sets break.
-
X-Robots-Tag and Files With No Head Element
A meta robots tag needs HTML. PDFs, images and feeds have none. How the header version works, and how it combines with the tag on the same URL.
-
Noindex and Nofollow on the Same Page
Page-level nofollow is not the same as rel=nofollow on a link, and combining it with noindex has a consequence people don't expect.
-
Canonical, Redirect, or Noindex: Picking the Right One
Three tools for three different problems, routinely swapped for each other. A decision path based on what you want to happen to users and signals.
-
Trailing Slashes Are Different URLs
/page and /page/ are two addresses that can serve two responses. How to pick one, enforce it in a single hop, and stop generating both.
-
Uppercase URLs and the Duplicates They Create
Paths are case-sensitive, hosts are not. Where mixed-case URLs come from, what they duplicate, and how to normalise without breaking real files.
-
Consolidating www and HTTPS in a Single Hop
Four host and protocol variants, one canonical address. Why separate rules chain, how to combine them, and where HSTS fits.
-
Query Parameters That Multiply Your URL Count
Tracking, sorting and session parameters each create a distinct URL. How to tell which ones need a canonical, a Disallow, or removing entirely.
-
Faceted Navigation and the Crawl Space It Opens
Five filters with four options each is a combinatorial URL space. How to decide which combinations get paths, which get blocked, and which get no links.
-
Pagination Without rel=next and rel=prev
Google stopped using the pagination link elements. What replaced them: crawlable links, self-canonicals per page, and no canonical to page one.
-
What an XML Sitemap Is Actually For
A sitemap is a discovery hint, not an indexing instruction. What it can fix, what it cannot, and why lastmod is the only optional field worth setting.
-
Reading Log Files for Crawl Evidence
Server logs are the only record of what a crawler actually requested and what you actually served. How to verify the client and what to count.
-
Where to Put a Redirect: Edge, Server, or Application
The same 301 behaves differently depending on which layer emits it. Latency, visibility, deploy coupling, and the shadowing that hides broken rules.
-
Meta Refresh and JavaScript Redirects
Client-side redirects happen after a 200 response. How crawlers treat them, when they are the only option, and the delay that makes one a dead end.
-
Redirect Chains and How to Flatten Them
Chains form by accretion, not by design. How to find them, why length matters, and how to flatten one that crosses three systems.
-
Internal Linking as Plumbing
Click depth, orphan pages, and crawl paths — internal links are the only part of the link graph you fully control.