Link Plumbing

Navigation That Only Exists After JavaScript Runs

URL discovery has one primary mechanism: a crawler parses a document, extracts <a> elements that have an href attribute, resolves each one against the document’s base URL, and queues the results. Sitemaps and external links add to the queue, but the link graph inside your own site is built from anchor elements and nothing else.

A great deal of modern navigation is not built from anchor elements. That is where internal link architecture quietly stops existing.

The two-phase crawl, and why it is not a safety net

The usual objection is that crawlers execute JavaScript now, so a client-rendered menu is fine. As of this writing that is partly true and structurally unreliable.

The dominant crawler fetches the raw response first and parses it for links. Rendering — running the page’s JavaScript in a headless browser and re-extracting links from the resulting DOM — happens as a second, deferred pass, queued separately and not guaranteed for every URL. The practical consequence: links present only in the rendered DOM are discovered later, if at all, and links present in the served HTML are discovered on the first pass.

So rendering is a fallback, not a substitute. Anything that must be discovered reliably belongs in the initial response.

Two further constraints on the rendered pass, both worth knowing because they are absolute rather than probabilistic:

  • No interaction. The renderer loads the page. It does not click, scroll, hover, submit, type, or expand. Anything that appears only in response to a user event does not appear.
  • No session. It arrives with no cookies, no local storage, no authenticated state. Content behind any of those is not part of the page as far as discovery is concerned.

Each of these renders something a human will happily click, and none of them produces an entry in a crawler’s queue.

A click handler on a non-anchor element.

<!-- not a link: nothing to extract, nothing to queue -->
<div class="card" onclick="location.href='/products/pump-42/'">Pump 42</div>
<button onclick="router.push('/products/pump-42/')">Pump 42</button>

An anchor with no destination. <a href="#">, <a href="javascript:void(0)">, or an <a> with no href at all. The element is an anchor; the URL is absent or meaningless. An <a> without href is not even a link as far as the HTML parser is concerned.

A router link that intercepts a real href. This one is fine — <a href="/products/pump-42/"> with a click handler that calls preventDefault() and does a client-side transition is exactly the right pattern, because the href is real and the interception only affects browsers. The failure is the framework component that emits <a> with no href, or emits a <span>. Check the rendered output rather than the component’s name.

Hash-only URLs. /#/products/pump-42/ sends nothing after the # to the server. The fragment is client-side only, so every route in a hash-routed application is the same URL from the network’s point of view. There is exactly one page to index, and its served content is usually an empty shell. Hash routing and indexable URLs are mutually exclusive; use the History API with real paths.

Links behind interaction. A mega-menu that fetches its contents on hover. An accordion whose panels are empty until opened. A “load more” button that appends the next twenty items. Infinite scroll. In every case the links exist for a user and do not exist for a crawler, because the triggering event never happens.

Links behind a fetch that needs state. A category listing populated by an XHR that requires a session cookie or an Authorization header returns nothing to an anonymous renderer, so the listing renders empty and the products it would have linked are undiscovered.

Navigation blocked at the transport layer. If the JavaScript bundle that builds the menu is under a Disallowed path, the rendered pass has no menu to extract — the render is executed with the same crawl restrictions as the fetch. Blocking /assets/ or /_next/ is a routine way to make a client-rendered site unrenderable. The matching semantics that make this easy to do by accident are in how robots.txt rules are matched.

What this looks like when it goes wrong

The symptom is rarely “the crawler can’t see my site.” It is a subset of pages that never get discovered, usually the deep commercial ones, while the homepage and blog index are fine — because those are linked from somewhere flat and static.

Concretely: product detail pages reachable only through a filtered, client-rendered grid; documentation pages reachable only through a collapsed sidebar tree; the fourth page of every listing and everything after it; anything reachable only from the mega-menu.

These are orphan pages produced by the rendering layer rather than by a content decision, and they behave the same way — discoverable only via the sitemap, which is a weaker and slower path, and receiving no internal signal at all. The general mechanism is in internal linking as plumbing.

Diagnosing it with two requests

The whole diagnosis is a comparison between what the server sends and what the browser ends up with.

U=https://example.com/category/pumps/

# every href in the served HTML
curl -s "$U" \
  | grep -o 'href="[^"]*"' \
  | sed 's/href="//;s/"$//' \
  | sort -u

Run that on a page whose navigation you are suspicious of, and compare the list against what you can click on in a browser. If the browser offers thirty product links and the served HTML contains four, you have found the gap and its size.

Two more checks worth building into the routine:

  • Pick a page you believe should be discoverable and grep for its path across the served HTML of its parents. If no parent’s raw response contains the path, nothing links to it in the first pass. This is the same set-difference logic as orphan detection, run against one URL.
  • Use Search Console’s URL inspection and read the rendered HTML it reports. That is the crawler’s own view after rendering, which is the only authoritative answer about whether the second pass recovered your links.

Do not use “the page ranks, so it must be fine” as evidence. A page can be indexed via the sitemap while receiving no internal links at all; ranking is downstream of a lot of things and does not tell you which path found the URL.

Fixes, in order of leverage

1. Emit real anchors with real hrefs. Every navigable destination gets <a href="/real/path/">. Client-side routing layers on top of that, it does not replace it. This single change fixes most of the list above and costs almost nothing in a framework that supports it — which is all of them.

2. Server-render or pre-render the navigation. The menu, the breadcrumb, the listing’s first page of links. The rest of the page can hydrate later; the links need to be in the first response.

3. Give every paginated set real URLs. Infinite scroll is a presentation choice, and it is compatible with ?page=2 existing as a crawlable URL that returns that slice. Without it, everything past the first screen is unreachable — the mechanics are in pagination without rel=next and rel=prev.

4. Add hub pages that are plain HTML. A topic index, a full product index, an HTML sitemap. Boring, static, and a reliable path to everything that matters.

5. Keep the sitemap, but do not rely on it. It is a discovery aid, not a link. It conveys no internal signal and no anchor text, and it does not make a page any less orphaned — what an XML sitemap is actually for covers the distinction.

6. Unblock your own assets. If the render depends on a bundle, the bundle has to be fetchable.

The framing that keeps this straight: JavaScript can change what a page looks like without consequence, but the set of URLs your site claims to have is a statement made in HTML. If a URL is not in an href somewhere in a served response, your site has not claimed it — whatever the browser shows.

internal-linkshttp