Crawlability Issues: What Breaks Crawling and How to Fix It

TL;DR — Crawlability is whether search engines can find and reach your pages. A crawlability issue is anything that gets in the way — a page blocked in robots.txt, a link bots can’t follow, a server that keeps erroring, or a flood of junk URLs. If a page can’t be crawled, it can’t be indexed, and if it’s not indexed, it can’t show up in search. Fix the access problem first; everything else is downstream.

What crawlability is

Before a page can rank, a search engine has to crawl it — send out a bot (Googlebot for Google, Bingbot for Bing) to discover the URL and download the page. Crawlability is just how easily that can happen. A crawlability issue is any technical thing that stops or slows it down. Evidence for this claim Google describes Search as crawling, indexing, and serving, and not every page proceeds through every stage. Scope: Google Search processing model. Confidence: high · Verified: 2026-07-14 Google: How Search works

Think of search as a pipeline: discover → crawl → render → index → serve. That five-step breakdown is my own practical way of organizing the work — Google’s official model groups discovery and rendering inside a broader “crawling” stage rather than listing them separately. Either way, crawling is near the front, and everything after it depends on it. So a crawlability problem is the worst kind of SEO problem — it doesn’t matter how good your content is if the bot never gets to read it.

The four kinds of crawlability problems

Almost every issue fits in one of four buckets. This is a triage shortcut, not an exhaustive list — things like a missing sitemap or a stray noindex tag live just outside these four. When something isn’t getting found, work out which bucket you’re in first:

1. Access is blocked. Your robots.txt disallows the page, it’s behind a login, a firewall returns a “403 Forbidden,” or the CSS/JS the page needs is blocked.
2. Links are broken or uncrawlable. Your navigation only works with JavaScript clicks instead of real links, or a page has nothing linking to it at all (an “orphan”).
3. The server is failing. It returns errors (500s), the domain won’t resolve, it’s painfully slow, or pages bounce through long redirect chains.
4. The URL inventory is bloated. Filters, calendars, and tracking tags spawn endless near-duplicate URLs, so bots waste their time on junk instead of your real pages.

The thing most people get wrong

Blocking a page in robots.txt does not remove it from Google. It only stops Google from reading it. The page can still show up in results (without a description) if other sites link to it. If you actually want a page gone, you let it be crawled and add a noindex tag instead. Evidence for this claim Google says robots.txt controls crawler access and is not a mechanism for keeping a page out of Google. Scope: Google robots.txt behavior; noindex must remain crawlable to be observed. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction

And crawling is not ranking. Getting crawled more often won’t move you up the results — crawling is a gate you have to get through, not a score.

Want the full triage framework, how to find issues in Search Console and your server logs, and the fixes for each one? Switch to the Advanced tab.

TL;DR — Crawlability is the degree to which crawlers can discover, access, and fetch your pages; a crawlability issue is anything that breaks that. Almost all of them fall into four buckets — access blocked (robots.txt, login walls, 403s, blocked JS/CSS), links broken (JS-only nav, orphans, deep pages), server failing (5xx, DNS, slow/truncated fetches, redirect chains), and URL inventory bloated (spider traps, parameter duplicates, soft 404s). Crawlability is upstream of indexing and ranking — necessary to rank but not a ranking signal, and distinct from indexability. Diagnose with GSC (Page Indexing, Crawl Stats, URL Inspection) and server logs; fix the access problem first, then the inventory. This hub triages each issue and links to the deep dives.

What crawlability is (and what it isn’t)

Crawlability is stage one of the search pipeline — discover → crawl → render → index → serve. Evidence for this claim Google describes Search as crawling, indexing, and serving, and not every page proceeds through every stage. Scope: Google Search processing model. Confidence: high · Verified: 2026-07-14 Google: How Search works Every failure here is upstream of indexing and ranking. The term itself is an SEO umbrella, not a Google one: Google’s own model names three Search stages — crawling, indexing, and serving — and folds URL discovery and rendering into crawling rather than giving them their own top-level stages. The five-step pipeline and the four-bucket triage below are my synthesis for organizing that work into something actionable — useful for locating a problem fast, but not a citation for how Google’s systems are formally staged internally.

Three “not equals” prevent most of the confusion in this space:

• Crawlability ≠ indexability. A page can be perfectly crawlable but blocked from the index by a noindex tag — or technically crawlable but still not indexed because of quality signals. Crawlability is a necessary but not sufficient condition for indexing. (See crawled – currently not indexed for the downstream quality case.)
• Crawling ≠ ranking. Crawling is required to appear in results, but as Google puts it, “an increased crawl rate will not necessarily lead to better positions in Search results… while crawling is necessary for being in the results, it’s not a ranking signal.”
• Crawling ≠ rendering. Google renders the page and runs JavaScript in a separate step — “during the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome” — and that renderer runs in its own queue that can lag behind the initial fetch.

The full pipeline lives in the crawling hub; this article is the triage layer for when it breaks.

The four buckets of crawlability issues

Use these to locate your problem fast. Each one gets a short treatment here with a link to the deep dive — don’t try to fix what you can’t yet categorize. This is a triage lens, not an exhaustive classification: sitemaps, noindex, canonical tags, nofollow, mobile parity, and duplicate content are all real problems, but most of them belong to discovery, indexability, rendering, or crawl efficiency rather than a pure access/fetch failure — they get their own pages linked below instead of being folded into this hub as a flat audit checklist.

1. Access is blocked

Robots.txt blocking (intentional or accidental). A Disallow rule stops crawlers from fetching the URL at all. Google is explicit that this is the wrong tool for removing pages: “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site… It is not a mechanism for keeping a web page out of Google.” The classic accidents: a staging-site Disallow: / pushed to production, a CMS plugin adding an overly broad rule, or blocking the *.js/*.css resources the page needs to render. Evidence for this claim Google says robots.txt controls crawler access and is not a mechanism for keeping a page out of Google. Scope: Google robots.txt behavior; noindex must remain crawlable to be observed. Confidence: high · Verified: 2026-07-14 Google: robots.txt introduction (See robots.txt and the GSC status blocked by robots.txt.)

The noindex + disallow trap. This one bites constantly. If you both disallow a page in robots.txt and put a noindex on it, Google can never fetch the page to see the noindex — so the page can stay indexed via external links. To deindex a page, allow crawling and add noindex. Never use disallow to deindex. I wrote up the live experiment behind this in Indexed, though blocked by robots.txt: “crawling and indexing are two different things.”

Login walls and 401s. If content sits behind an interactive login or OAuth flow Googlebot can’t complete, it’s simply unreachable — can’t be crawled or indexed. That’s not the same claim as “every paywall is uncrawlable,” though: publishers can choose to expose gated or subscription content to crawlers differently than to a regular visitor, so treat “requires a login Googlebot can’t get past” and “metered/paywalled” as separate cases rather than assuming every paywall behaves the same way. (See blocked 401.)

403 Forbidden. Before reaching for an IP allowlist, work through the likely causes in order rather than assuming the first one you find is the only one:

1. robots.txt — confirm the exact path isn’t disallowed for that crawler’s user-agent; this is the single most common cause and the fastest to rule out.
2. Intermittent or cached blocks — re-test the same URL more than once, from more than one vantage point. A WAF/CDN rule that only trips occasionally, or a stale cached 403, can look like a permanent block when it isn’t.
3. User-agent blocks — a WAF/CDN rule denying a specific bot string (and remember the string alone can be spoofed, so don’t stop the investigation here).
4. IP-range blocks — the firewall, CDN, or geo rule denying Googlebot’s actual IP ranges or datacenter geography. Only whitelist verified Googlebot — confirmed via reverse + forward DNS, not just the user-agent header — once you’ve actually located which layer is blocking it. (See blocked 403 and blocked other 4xx.)

Blocked JS/CSS. Google renders in a headless Chrome. If robots.txt blocks *.js/*.css, or a CDN refuses Googlebot the resources, the rendered page looks broken — content that only appears after rendering may not be indexed, and links that only appear after JS may not be followed. Test with URL Inspection → Live Test and look at the rendered screenshot.

2. Links are broken or uncrawlable

Non-crawlable links. “Google can only crawl your link if it’s an <a> HTML element with an href attribute. Most links in other formats won’t be parsed and extracted by Google’s crawlers.” That rules out <a routerLink="...">, <span href="...">, and <a onclick="goto(...)">. If your main navigation is JS-only click handlers, whole sections of the site may never be discovered. (See the rendering section of the crawling hub and my JavaScript SEO Issues & Best Practices.)

Orphan pages. A page with no internal links pointing to it can only be found via a sitemap or external link. Links are the dominant discovery channel, so an orphan often shows up as discovered – currently not indexed in GSC. A sitemap entry aids discovery; it doesn’t replace the link signal. Fix: link the page from a relevant hub or category page. (See discovery and crawl depth.)

Deep pages. Pages buried many clicks from the homepage get crawled later and less often. Keep important pages shallow (see crawl depth).

3. The server is failing or throttling

5xx server errors. A 500/502/503/504 returns an error instead of content. Sustained 5xx makes Googlebot throttle — “they try not to crawl the site too fast to avoid overloading it… HTTP 500 errors mean ‘slow down’” — and eventually drop already-indexed pages. If robots.txt itself returns 5xx, it’s not an instant, indefinite site-wide halt: Google’s published behavior is staged — crawling stops for about 12 hours first, then Google may fall back to the last known-good robots.txt for up to 30 days, and what happens after that depends on the cached rules and whether the site is generally reachable. Note the 503 nuance: a 503 with a Retry-After header is the correct response for planned short downtime; only weeks of it cause harm. Google buckets 429 (too many requests) with the same back-off behavior. (See server error (5xx) and crawl rate.)

DNS failures. Upstream of HTTP entirely: if the domain doesn’t resolve — expired registration, a botched nameserver migration, broken apex-domain resolution — there’s no server response to even return a 5xx. Sustained DNS failure drops the whole site. Catch it in GSC Crawl Stats → Host Status, or faster with external uptime monitoring.

Slow responses and fetch truncation. Google’s politeness mechanism throttles based on server health, so slow responses mean fewer crawls. And there’s a hard byte limit: as of Google’s March 2026 Inside Googlebot update, Googlebot fetches roughly 2 MB per URL (down from the old 15 MB figure), with PDFs up to 64 MB. Go over and the fetch is truncated, not rejected — only the downloaded portion is indexed. If your critical content sits below 2 MB of bloat, it can get cut off.

Redirect chains and loops. Each redirect hop costs a crawl credit, and long chains have a negative effect on crawling; Google follows only about five hops before it may give up on the destination. Loops time out and land in redirect error in GSC. Common cause: stacked www → non-www → HTTPS hops from successive migrations. (See crawl budget, which lists redirect chains as crawl waste.)

4. The URL inventory is bloated

Spider traps. Structures that generate effectively infinite URLs — faceted filter combinations, calendar “next month” links with no end, session IDs in URLs, infinite pagination. Crawlers burn budget on near-duplicate junk while real pages get crawled less. Google has called faceted navigation and action parameters responsible for the bulk of its crawling issues, and “infinite spaces” has topped the crawl-waste list since 2008. There’s no fixed universal threshold for what counts as “too much” — the signature is a lopsided, disproportionate share of hits concentrated on one junk URL pattern relative to your own site’s normal baseline, not a specific percentage that applies everywhere. (See spider traps.)

URL parameters. Query strings (?key=value) create multiple URLs for the same content — active params (sort/filter/pagination) make near-duplicates; passive params (tracking, session IDs) make pure duplicates. Both waste budget and dilute link signals. Note: Google retired its URL Parameters tool in April 2022 (it found only ~1% of configs useful and now handles most cases automatically) — don’t tell people to use it. Bing still has an Ignore URL Parameters tool. Current fix: consistent parameter ordering, rel=canonical for passive params, robots.txt disallow only for genuine traps. (See url parameters and canonicalization.)

Soft 404s. A page that’s functionally “not found” (empty search results, gutted out-of-stock product) but returns 200 OK. Google keeps crawling it because the status says it’s there, wasting budget for no indexable content. Fix: return a real 404 or 410 (410 Gone is marginally faster for removal). A genuine 404 is a strong signal not to crawl that URL again.

Crawl budget exhaustion (large sites only)

This is the bucket-4 problem at scale, and it’s worth its own warning: most sites never hit it. Google says plainly, “if your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” John Mueller has put a number on it — 100,000 URLs is usually not enough to affect crawl budget. It realistically matters at ~1M+ pages changing weekly or 10k+ changing daily. Budget is per hostname, and all of Google’s crawlers (image, video, news, ads) draw from the same per-site pool. The warning light is a large and growing discovered – currently not indexed count. (Full treatment in crawl budget; I also wrote When Should You Worry About Crawl Budget?.)

How to find crawlability issues

Google Search Console.

• Page Indexing report surfaces the named failures: blocked by robots.txt, server error (5xx), not found (404), redirect error.
• Crawl Stats report shows response-code trends, average response time, and a host-status panel that flags DNS, robots.txt, and server-connectivity problems. (See crawl stats report.)
• URL Inspection (Live Test) shows what Googlebot actually fetched and rendered for a single URL — the rendered screenshot is the fastest way to catch blocked JS/CSS.
• “Discovered – currently not indexed” is the crawl warning light: Google knows the URL but hasn’t fetched it (usually a linking or priority problem).

Server log file analysis is the ground truth. Logs show exactly which URLs bots hit, how often, and with what status — and they’re the only way to tell real Googlebot (verified via reverse + forward DNS) from bots faking the user-agent. A spider trap shows up as a lopsided share of hits to one URL pattern. (See log file analysis.)

Site audit crawlers — Ahrefs Site Audit and Screaming Frog SEO Spider — simulate a crawl and surface redirect chains, blocked URLs, orphan pages, and crawl depth in one pass.

Crawlability vs. indexability: don’t confuse them

In short: crawlability is whether a bot can reach the page; indexability is whether Google chooses to store it once fetched. Once crawlability is fixed, indexability is the next gate.

The GSC status is a useful starting hypothesis, not a guaranteed diagnosis. Discovered – currently not indexed usually points to a crawl scheduling/priority problem (add internal links, raise site quality). Crawled – currently not indexed most often points to a content/quality problem downstream of crawlability (thin, duplicate, or intent-mismatched content) — but Google doesn’t publish an exhaustive causal mapping for that status, and overlapping factors like rendering delays, canonicalization signals, or indexing-eligibility checks can produce the same label. Treat the status as your first lead, confirm with URL Inspection and logs, and don’t throw crawl fixes at what turns out to be an indexability problem, or vice versa.

How Bing crawlability differs from Google

A few practical differences worth knowing if you optimize for both:

• IndexNow — Bing (and Yandex, Naver, Seznam, Yep) take it for instant change notification; Google does not participate. Don’t expect IndexNow to help Google crawlability.
• Crawl Control — Bing still has a manual grid in Bing Webmaster Tools to schedule crawl hours; Google removed its crawl-rate slider in January 2024.
• Ignore URL Parameters — Bing still offers it; Google retired its equivalent in April 2022.
• lastmod — Bing weights it more heavily as a recrawl signal; Google largely ignores inaccurate lastmod.

Fabrice Canel of Microsoft Bing sums up Bing’s philosophy as “Less is more for SEO… Less URLs to crawl, better for SEO” — which is the whole crawlability game in one line. (See bingbot.)

Where to go next

This hub is the map; the deep dives do the work. For access controls, see robots.txt and noindex. For the server side, see server error (5xx), crawl rate, and redirect error. For inventory bloat, see spider traps, url parameters, and crawl budget. For discovery and links, see discovery, crawl depth, and orphan pages. And for the whole pipeline in one place, start at the crawling hub.

AI summary

A condensed take on the Advanced version:

• Crawlability = how well crawlers can discover, access, and fetch your pages. A crawlability issue is anything that breaks that. It’s stage one of the pipeline (discover → crawl → render → index → serve — Patrick’s practitioner synthesis; Google’s own model groups discovery and rendering inside a broader “crawling” stage), so it’s upstream of everything.
• Three “not equals”: crawlability ≠ indexability (a crawlable page can still be noindex’d); crawling ≠ ranking (“not a ranking signal,” per Google); crawling ≠ rendering (JS runs in a separate, lagging step).
• Four buckets: (1) access blocked — robots.txt, login walls/401, 403 from WAF/CDN, blocked JS/CSS; (2) links broken — JS-only nav with no <a href>, orphan pages, deep pages; (3) server failing — 5xx (throttles then drops), DNS failures (whole site), slow/truncated fetches (~2 MB limit), redirect chains (~5 hops); (4) inventory bloated — spider traps, parameter duplicates, soft 404s (200 OK with no content).
• The big trap: noindex + disallow = Google never sees the noindex. To deindex, allow crawling + noindex; never use robots.txt to remove a page.
• Crawl budget only matters for huge/fast-changing sites (~1M+ pages weekly; 100k URLs usually doesn’t). Per hostname, shared across all Google crawlers.
• Diagnose with GSC (Page Indexing, Crawl Stats, URL Inspection) and server logs (the ground truth; only way to verify real bots). Fix access first, then inventory.
• Bing differs: IndexNow (not Google), Crawl Control, Ignore URL Parameters, heavier lastmod weighting.

Official documentation

Primary-source documentation on what breaks crawling.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve pipeline, URL discovery, and the “slow down” throttling mechanism.
• Crawling and Indexing hub — robots, sitemaps, canonicalization, and crawl controls in one place.
• Introduction to robots.txt — what robots.txt does and (crucially) doesn’t do.
• Make your links crawlable — the <a href> requirement and the link formats Google can’t follow.
• Optimize your crawl budget — capacity + demand, the crawl-waste categories, and who actually needs to care.
• Overview of Google crawlers and fetchers — user-agents, IP ranges, and how Google’s crawlers identify themselves.
• Inside Googlebot (March 2026) — current byte limits (~2 MB per URL, 64 MB PDFs) and truncation behavior.
• Page Indexing report help — what each crawl/index status code means.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawling and its “crawl efficiency north star.”
• Bing Webmaster Tools — Crawl Control — schedule Bingbot’s crawl hours (Google has no equivalent).
• IndexNow — the push protocol Bing and others use for instant change notification (not Google).

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — robots.txt is not a removal tool

• “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” — Google Search Central docs. Jump to quote
• “It is not a mechanism for keeping a web page out of Google.” Jump to quote
• “The instructions in robots.txt files cannot enforce crawler behavior; it’s up to the crawler to obey them.” Jump to quote

Google — crawlable links

• “Google can only crawl your link if it’s an <a> HTML element with an href attribute.” — Make your links crawlable. Jump to quote
• “Most links in other formats won’t be parsed and extracted by Google’s crawlers.” Jump to quote

Google — server health and rendering

• “They try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean ‘slow down’).” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote

Google — crawl budget and crawling vs. ranking (2017 crawl-budget post; relayed via Search Engine Land’s verbatim reproduction)

• “An increased crawl rate will not necessarily lead to better positions in Search results… while crawling is necessary for being in the results, it’s not a ranking signal.” Read the coverage
• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” — Large site crawl-budget guide. Jump to quote

Fabrice Canel, Microsoft Bing

• “Crawling is the process by which bingbot discovers new and updated documents or content to be added to Bing’s searchable index.” Jump to quote
• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” — relayed via Search Engine Land. Read the coverage

The crawlability triage checklist

Run these in order — access problems first, because they’re the most catastrophic and the most common:

• robots.txt doesn’t disallow anything you want indexed (check for a stray Disallow: / from staging, and that *.js/*.css aren’t blocked).
• No page uses noindex + disallow together (the noindex can never be seen — to deindex, allow crawling and add noindex).
• Important content isn’t behind a login wall / 401 or returning 403 to verified Googlebot (check WAF/CDN/geo rules).
• Navigation uses real <a href> links, not JS-only click handlers.
• No orphan pages — every page you care about is linked from at least one other crawlable page.
• Server returns fast, stable responses — minimal 5xx; planned downtime uses 503 + Retry-After.
• DNS resolves reliably (check GSC Host Status + external uptime monitoring).
• No redirect chains longer than a hop or two, and no loops.
• Critical content sits under the ~2 MB fetch limit (not below page bloat).
• No spider traps generating infinite URLs (calendars, facets, session IDs).
• URL parameters aren’t spawning crawlable duplicates (consistent ordering, canonicals for passive params).
• No soft 404s — “not found” pages return a real 404/410, not 200.
• GSC Crawl Stats reviewed for response-code spikes and average response time.
• Server logs checked for crawl waste and important URLs bots never reach.

The mental models

1. The four buckets. Every crawlability issue is one of: access blocked, links broken, server failing, or inventory bloated. Categorize before you fix — the bucket tells you which tool to reach for.

2. The pipeline — discover → crawl → render → index → serve. Locate which stage a page is failing at before changing anything. Was it discovered? Crawled? Rendered? Indexed? Served? A fix aimed at the wrong stage does nothing.

3. The three “not equals.”

• Crawling ≠ indexing (a robots-blocked page can still be indexed via links).
• Crawling ≠ ranking (crawl rate isn’t a ranking signal).
• Crawling ≠ rendering (JS runs in a separate, lagging step).

4. The deindex decision rule. Want a page gone from search? Allow crawling + noindex. Want bots to skip a URL space entirely (and don’t care about indexing)? robots.txt disallow. Never use disallow to deindex — the bot can’t see your noindex through a block.

5. Fix the source, then block. For inventory bloat, stop generating the junk URLs first (the clean fix), and only then block whatever remains in robots.txt. Blocking alone doesn’t reclaim budget unless you’re already at your server’s capacity ceiling.

6. Severity triage. Site-wide first (DNS failure, robots.txt Disallow: /, robots.txt returning 5xx) → section-wide (JS-only nav, a blocked directory) → page-level (one redirect chain, one orphan). Stop the bleeding before you tweak.

Crawlability issues — cheat sheet

The four buckets and their signature symptoms

Status codes bots care about

• 200 — fetched fine (but a “not found” page returning 200 is a soft 404).
• 301/308 — permanent redirect (keep chains short; ~5 hops max).
• 403/401 — access denied / auth required (check WAF, verify it’s not real Googlebot being blocked).
• 404/410 — gone; strong do-not-crawl signal (410 a touch faster).
• 429/500/503 — “slow down” / try later; 503 + Retry-After is correct for planned downtime; sustained = crawl drops.

Fast facts

• Fetch limit: ~2 MB per URL (PDFs 64 MB) — truncated, not rejected, beyond.
• Redirect hops Google follows: ~5 before it may abandon the destination.
• Crawl budget rarely matters under ~100k URLs; budget is per hostname.
• GSC URL Parameters tool: retired April 2022. Manual crawl-rate slider: removed Jan 2024.
• IndexNow = Bing/Yandex/others, not Google.

Tools for finding crawlability issues

• Google Search Console — Page Indexing report — surfaces the named failures: blocked by robots.txt, server error (5xx), not found (404), redirect error, and the discovered/crawled – currently not indexed states.
• GSC — Crawl Stats report — response-code trends, average response time, and a host-status panel that flags DNS, robots.txt, and connectivity problems. (See crawl stats report.)
• GSC — URL Inspection (Live Test) — see exactly what Googlebot fetched and rendered for one URL; the rendered screenshot is the fastest blocked-JS/CSS check.
• Server log file analysis — the ground truth: which URLs bots actually hit, how often, and what they got back. The only way to verify real Googlebot (reverse + forward DNS) and to spot a spider trap’s lopsided hit pattern. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysis.)
• Site audit crawlers — Ahrefs Site Audit and Screaming Frog SEO Spider simulate a crawl and surface redirect chains, blocked URLs, orphan pages, and crawl depth.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.
• Bing Webmaster Tools — crawl info, Crawl Control, and Site Scan; plus the Ignore URL Parameters tool Google retired.

Test yourself: Crawlability issues

Five quick questions on what breaks crawling and how to fix it. Pick an answer for each, then check.

Why can’t the crawler reach the page?

Incident playbook: an important section disappears from crawling

1. Confirm scope. Test representative URLs and compare first-failure time with releases, firewall changes, and server incidents.
2. Check the fetch path. Resolve DNS, TLS, redirects, status codes, authentication, and response completeness. Fix failures before deeper SEO analysis.
3. Check crawler access. Test the exact URLs against robots.txt and verify that WAF/CDN rules do not block verified bots.
4. Check discovery. Crawl from the homepage and section hubs. If URLs are orphaned or JS-click-only, restore normal anchor links and sitemap membership.
5. Check rendering and inventory. Confirm required resources load and identify traps or duplicate paths consuming requests.
6. Validate recovery. Re-fetch representative URLs and watch verified crawler logs; if access works but indexing does not recover, move to an indexability diagnosis.

URL returns 200 in a browser but crawlers get 403

Likely causes, in the order I check them: robots.txt disallow for that user-agent, then an intermittent or cached block (re-test more than once), then a user-agent-specific WAF/CDN rule, then an IP-range or geo block. Fix: confirm which layer is actually responsible before touching anything, verify the crawler’s IP via reverse + forward DNS, inspect edge logs, and narrow the blocking rule. Confirm: the verified crawler and a normal user both receive the intended response.

Page exists but crawlers never request it

Likely causes: orphaning, JS-only controls, broken pagination, or missing sitemap discovery. Fix: add crawlable <a href> links from relevant pages and include the canonical URL in the correct sitemap. Confirm: a site crawl reaches it and later server logs show a verified crawler fetch.

Crawl tools report blocked resources

Likely causes: robots.txt rules cover CSS, JavaScript, or API resources required for rendering. Fix: allow the required resources without exposing private endpoints. Confirm: rendered output contains the same primary content and links available to users.

Crawl volume spikes on junk URLs

Likely causes: faceted parameters, calendars, internal search, session identifiers, or infinite navigation. Fix: remove crawlable generation paths and consolidate or block only after understanding index and rendering effects. Confirm: equivalent log windows show the pattern declining while valuable URLs stay discoverable.

Prompt: turn crawl evidence into a triage table

Analyze the crawlability evidence below. For each URL, classify the failure as access, discovery/linking, server/redirect, rendering/resource, or URL-inventory. Cite only the supplied evidence, list missing checks, and return symptom → likely cause → safest fix → pass/fail verification. Keep crawlability separate from indexability.

[PASTE STATUS, HEADERS, ROBOTS TEST, CRAWL PATH, RENDERED HTML, AND LOG OBSERVATIONS]

Prompt: review a robots.txt change safely

Review this proposed robots.txt change against the supplied sample URLs and user agents. Build an allow/block matrix, identify the winning rule, flag required resources or valuable pages that would become blocked, and propose the smallest safe change. Do not claim the file removes URLs from the index.

[PASTE CURRENT FILE, PROPOSED FILE, USER AGENTS, AND URL SAMPLES]

Check a URL list for crawl responses

while IFS= read -r url; do curl -L -sS -o /dev/null -w '%{http_code}\t%{url_effective}\t%{time_total}\n' "$url"; done < urls.txt

PowerShell equivalent:

Get-Content .\urls.txt | ForEach-Object { try { $r = Invoke-WebRequest -Uri $_ -MaximumRedirection 10; "{0}`t{1}" -f $r.StatusCode,$r.BaseResponse.ResponseUri } catch { "ERROR`t$_" } }

Find non-crawlable link controls in rendered HTML

Run this in the browser DevTools Console. It lists elements with click behavior that are not normal links:

[...document.querySelectorAll('[onclick], [role="link"]')].filter(el => !el.matches('a[href]')).map(el => ({text: el.textContent.trim(), html: el.outerHTML.slice(0, 300)}));

The result is a review queue, not proof that every item should become a link.

Resources worth your time

My related writing

• Indexed, though blocked by robots.txt — why a blocked page can still be indexed, and the noindex-vs-disallow trap.
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party experiment on what actually happens when you block ranking pages.
• When Should You Worry About Crawl Budget? — who needs to care, and what wastes budget.
• JavaScript SEO Issues & Best Practices — the rendering side: when JS-dependent links and content break discovery.
• Meet the New Web Crawlers — who’s actually crawling you now, from Cloudflare Radar data.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (Standing disclaimer: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Google’s Crawling December series — the best concentrated set of official crawl explainers (Googlebot, HTTP caching, faceted nav, CDNs).
• Google Explains Crawl Budget for Webmasters (Search Engine Land) — verbatim reproduction of Gary Illyes’ 2017 crawl-budget post, including the “not a ranking signal” line.
• Google: 100,000 URLs Usually Won’t Impact Crawl Budget (Search Engine Roundtable) — the John Mueller quote that sets the scale.
• The five infrastructure gates behind crawl, render, and index (Search Engine Land) — Fabrice Canel’s “less is more for SEO” framing of crawl efficiency.
• Google Explains Googlebot Byte Limits and Crawling Architecture (Search Engine Journal) — coverage of the March 2026 update; 2 MB per URL, 64 MB PDFs, truncation.
• Googlebot File Size Limit (DebugBear) — what happens when a page exceeds the fetch limit (truncation, not rejection).

Crawlability

Crawlability is how well search engine crawlers can discover, access, and fetch a site's pages. A crawlability issue is any technical condition — blocked access, broken links, server failures, or bloated URL inventory — that stops pages from reaching the index.

Related: Crawling, Crawl Budget, Indexing

Crawlability

Crawlability is the degree to which search engine crawlers can discover, access, fetch, and process the pages on a site. A crawlability issue is any technical condition that prevents or degrades that access — so pages get missed entirely, fetched inefficiently, or rendered incorrectly, and never make it to the index.

Crawlability sits at stage one of the search pipeline — discover → crawl → render → index → serve — so every failure here is upstream of indexing and ranking. You can have perfect content, strong links, and clean structured data and still rank for nothing if crawlers can’t get to the page. That five-step breakdown is a practitioner synthesis for organizing the work, not Google’s own model, which names three Search stages (crawling, indexing, serving) and folds URL discovery and rendering into crawling rather than listing them separately.

The term is an SEO umbrella, not a Google term. Google talks about URL discovery, crawl budget, crawl capacity, and crawl demand; “crawlability” is how SEOs group the barriers that break any of those steps. Most issues fall into one of four buckets: access is blocked (robots.txt, login walls, 403s, blocked JS/CSS), links are broken or uncrawlable (JS-only navigation, orphan pages), the server is failing or throttling (5xx, DNS failures, slow responses, redirect chains), or the URL inventory is bloated (spider traps, parameter duplicates, soft 404s). This is a triage lens, not an exhaustive classification — sitemaps, noindex, canonicals, and mobile parity are related but distinct problems that live outside these four buckets.

Crawlability is necessary but not sufficient for indexing: a page can be perfectly crawlable and still not be indexed because of a noindex tag or quality signals. That’s why diagnosis starts by asking which stage a page is failing at before changing anything.

Related: Crawling, Crawl Budget, Indexing

Changelog