Crawling, rendering, and indexing: diagnose the exact failure

“Not indexed” is an outcome, not a diagnosis.

A search engine can know a URL exists without fetching it, fetch it without rendering its intended content, render it without selecting it for the index, index it without serving it for the query you care about, and serve it without producing a useful visit.

The fix is to stop debugging “SEO” as one state. Follow one URL through the system and identify the first transition where observed behavior diverges from expected behavior.

Begin with one URL and one expected outcome

Pick a representative URL with business value. Record:

• the exact canonical URL you expect;
• the page type and intended search intent;
• the response status and redirect behavior;
• whether it should be crawlable and indexable;
• the primary content and links that should be present;
• the query family and conversion event it should support.

This prevents tool drift. If the investigation starts with a 100,000-row crawl, the team can spend days classifying symptoms without agreeing what one good product, service, or article URL should do.

Discovery: does the engine know the URL?

URLs are commonly discovered through crawlable links, XML sitemaps, feeds, redirects, and previously known URLs. A page can exist in a CMS and remain operationally orphaned if no discoverable route points to it.

Check the rendered navigation and content, not just the CMS relationship model. A recommendation widget that appears only after an interaction may not provide the same persistent discovery path as a normal anchor in the document.

Evidence for discovery includes internal link exports, sitemap reports, Search Console URL Inspection, and server logs showing crawler requests. None is complete alone. A sitemap entry says you submitted a URL; it does not prove that the internal graph treats it as important.

Fetch: what did the server return?

Request the final URL and preserve the response. Inspect:

• status code and redirect chain;
• robots and caching headers;
• content type and encoding;
• canonical and robots tags in the HTML;
• body content, size, and timing;
• differences by hostname, protocol, slash, locale, user agent, or authentication state.

A correct-looking browser page can conceal an incorrect response. Client-side routing may render a friendly not-found view over 200 OK. A CDN may cache an error body with a successful status. An edge rule may redirect bots differently from users.

Start with curl or an equivalent raw request before opening a rendering tool. The HTTP layer is evidence, not plumbing.

Render: what document exists after processing?

Rendering turns the response into a document. With JavaScript sites, that can involve executing scripts, requesting data, changing metadata, and inserting content.

Google’s JavaScript SEO guide describes processing as a pipeline. That does not mean every JavaScript site fails. It means the rendered state needs its own test.

Compare the raw response with the rendered DOM for:

• the same title, canonical, robots directives, and language;
• the H1 and primary body content;
• real internal anchor destinations;
• visible structured-data facts;
• images and essential resources;
• error, loading, consent, and empty states.

If the content is absent from the response, document which request supplies it, what happens when that request fails, how long it takes, and whether the route works without stored browser state.

Index: which document did the system select?

Indexing is not a receipt for a sitemap submission. Search systems select canonical documents and may exclude URLs because of directives, duplicates, response patterns, or quality judgments.

When the failure involves controls, test the intended outcome before editing a file: robots.txt controls requests while noindex controls search eligibility. If duplicate routes compete, use the canonical signal protocol. If submitted inventory is polluted, apply the sitemap eligibility contract.

Use URL Inspection to compare the user-declared canonical with the selected canonical. Then verify that every signal supports the intended owner:

• internal links point to it;
• the sitemap contains it, not alternates;
• redirects resolve directly to it;
• its canonical is self-consistent;
• locale and mobile variants declare correct relationships;
• duplicate templates do not offer stronger contradictory signals.

Google’s canonicalization documentation describes canonical signals as inputs, not absolute commands. Alignment is more reliable than relying on one tag to override a confused architecture.

Serve: can the indexed page answer the query?

An indexed page can receive zero impressions because it is not a relevant or competitive result for the intended query. That is no longer purely an indexing diagnosis.

Check Search Console queries, country, device, result type, landing URL, and date. Confirm that the page answers the intent exposed by the actual result set. Compare evidence, format, freshness, and information gain — not only word count and keyword placement.

This is the boundary between technical eligibility and the wider topical authority system. Technical SEO keeps the right candidate available. Content architecture and evidence make it worth selecting.

The technical SEO chain in practice

Suppose a Shopify collection is missing from search:

1. Discover: the collection exists in the admin but every product links back only through a client-side filter state.
2. Fetch: the clean collection URL returns 200 OK.
3. Render: the title and products are present, but the canonical changes to the unfiltered parent.
4. Index: Google selects that parent, so the child collection is excluded as an alternate canonical.
5. Serve: the parent ranks for broad category terms and the distinct subcategory has no owner page.

The solution is not “request indexing.” The team must decide whether the subcategory deserves a distinct search resource. If yes, give it unique intent, stable navigation, a self-consistent canonical, and useful content. If no, keep the consolidation and remove the false expectation.

The same chain works for WordPress and WooCommerce. The plugin names change; the states do not.

Read logs as behavior, not as a ranking score

Server or edge logs show requests that third-party crawlers cannot reproduce: user agent, path, timestamp, status, bytes, and sometimes response time. They can reveal wasted routes, repeated failures, slow templates, and crawl patterns.

Logs do not tell you why a page ranks. They tell you what requested your server and what the server returned. Join them with canonical inventories, sitemap membership, template types, and Search Console states before drawing conclusions.

For a small site, logs may confirm that critical pages are fetched and errors are rare. For a large faceted store, they can expose millions of parameter combinations consuming requests while revenue categories remain deeply linked.

Build monitoring around state changes

A quarterly crawl is a snapshot. Releases happen every week.

Monitor the transitions most capable of changing at scale:

• non-200 responses on canonical landing pages;
• canonical and robots changes by template;
• sitemap URLs that redirect or disappear;
• primary content absent from response or render;
• structured-data errors on supported templates;
• material Core Web Vitals regressions;
• index-state shifts in important directories;
• conversion events that stop recording.

Alert on a meaningful pattern, not every fluctuation. Include an example URL, the changed evidence, the affected template, and the release owner. A notification without reproduction context becomes noise.

The investigation record

For every material issue, preserve a compact record:

That record connects the diagnosis to a release and the release to a durable test. Without it, the same issue returns under a new crawler export six months later.

The full technical SEO guide shows how this URL-level chain expands into site architecture, semantics, performance, structured data, and platform controls.