Crawlability Issues: What Breaks Crawling and How to Fix It

Crawlability issues are the technical conditions that stop search engines from discovering, accessing, or fetching your pages. A triage-first guide to the four buckets — blocked access, broken links, failing servers, bloated URL inventory — how to find them, and how to fix them.

First published: Jun 27, 2026 · Last updated: Jul 17, 2026 · Advanced
demand #7 in Crawling#15 in How Search Works#82 in Technical SEO#108 on the site
AIAuthored byAI v2

Crawlability is how well search engine crawlers can discover, access, and fetch your pages — and a crawlability issue is anything that breaks that. Almost every issue falls into one of four buckets: access is blocked (robots.txt, login walls, 403s, blocked JS/CSS), links are broken or uncrawlable (JS-only nav, orphan pages), the server is failing or throttling (5xx, DNS failures, slow/truncated fetches, redirect chains), or the URL inventory is bloated (spider traps, parameter duplicates, soft 404s). Crawlability is upstream of indexing and ranking — necessary but not a ranking factor, and distinct from indexability. This hub triages the common, high-impact problems and points you to the deep dives.

TL;DR — CrawlabilityCrawlability is how well search engine crawlers can discover, access, and fetch a site's pages. A crawlability issue is any technical condition — blocked access, broken links, server failures, or bloated URL inventory — that stops pages from reaching the index. is the degree to which crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can discover, access, and fetch your pages; a crawlability issue is anything that breaks that. Almost all of them fall into four buckets — access blocked (robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., login walls, 403s, blocked JS/CSS), links broken (JS-only nav, orphans, deep pages), server failing (5xx, DNS, slow/truncated fetches, redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency.), and URL inventory bloated (spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content., parameter duplicates, soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.). Crawlability is upstream of indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and ranking — necessary to rank but not a ranking signal, and distinct from indexability. Diagnose with GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. (Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason., Crawl Stats, URL Inspection) and server logs; fix the access problem first, then the inventory. This hub triages each issue and links to the deep dives.

What crawlability is (and what it isn’t)

Crawlability is stage one of the search pipeline — discover → crawl → render → index → serve. Evidence for this claim Google describes Search as crawling, indexing, and serving, and not every page proceeds through every stage. Scope: Google Search processing model. Confidence: high · Verified: Google: How Search works Every failure here is upstream of indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and ranking. The term itself is an SEO umbrella, not a Google one: Google’s own model names three Search stages — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexing, and serving — and folds URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched. and renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. into crawling rather than giving them their own top-level stages. The five-step pipeline and the four-bucket triage below are my synthesis for organizing that work into something actionable — useful for locating a problem fast, but not a citation for how Google’s systems are formally staged internally.

Three “not equals” prevent most of the confusion in this space:

  • Crawlability ≠ indexability. A page can be perfectly crawlable but blocked from the index by a noindex tag — or technically crawlable but still not indexed because of quality signals. Crawlability is a necessary but not sufficient condition for indexing. (See crawled – currently not indexedA Google Search Console Page Indexing status meaning Googlebot fetched the page but Google decided not to index it — usually a content- or site-quality signal, not a technical error. for the downstream quality case.)
  • Crawling ≠ ranking. Crawling is required to appear in results, but as Google puts it, “an increased crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. will not necessarily lead to better positions in Search results… while crawling is necessary for being in the results, it’s not a ranking signal.”
  • Crawling ≠ rendering. Google renders the page and runs JavaScript in a separate step — “during the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome” — and that renderer runs in its own queue that can lag behind the initial fetch.

The full pipeline lives in the crawling hub; this article is the triage layer for when it breaks.

The four buckets of crawlability issues

Use these to locate your problem fast. Each one gets a short treatment here with a link to the deep dive — don’t try to fix what you can’t yet categorize. This is a triage lens, not an exhaustive classification: sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., noindex, canonical tagsA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content., nofollowrel=\"nofollow\" is a value of the HTML link rel attribute that tells search engines you don't vouch for a linked page and don't want to pass ranking signals to it. Since 2019–2020 Google treats it as a hint, not a directive — and it does not reliably block crawling or indexing., mobile parity, and duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. are all real problems, but most of them belong to discovery, indexability, rendering, or crawl efficiency rather than a pure access/fetch failure — they get their own pages linked below instead of being folded into this hub as a flat audit checklist.

1. Access is blocked

Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. blocking (intentional or accidental). A Disallow rule stops crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. from fetching the URL at all. Google is explicit that this is the wrong tool for removing pages: “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site… It is not a mechanism for keeping a web page out of Google.” The classic accidents: a staging-site Disallow: / pushed to production, a CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms. plugin adding an overly broad rule, or blocking the *.js/*.css resources the page needs to render. Evidence for this claim Google says robots.txt controls crawler access and is not a mechanism for keeping a page out of Google. Scope: Google robots.txt behavior; noindex must remain crawlable to be observed. Confidence: high · Verified: Google: robots.txt introduction (See robots.txt and the GSC status blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing..)

The noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. + disallow trap. This one bites constantly. If you both disallow a page in robots.txt and put a noindex on it, Google can never fetch the page to see the noindex — so the page can stay indexed via external links. To deindex a page, allow crawling and add noindex. Never use disallow to deindex. I wrote up the live experiment behind this in Indexed, though blocked by robots.txt: “crawling and indexing are two different things.”

Login walls and 401s. If content sits behind an interactive login or OAuth flow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. can’t complete, it’s simply unreachable — can’t be crawled or indexed. That’s not the same claim as “every paywall is uncrawlable,” though: publishers can choose to expose gated or subscription content to crawlers differently than to a regular visitor, so treat “requires a login Googlebot can’t get past” and “metered/paywalled” as separate cases rather than assuming every paywall behaves the same way. (See blocked 401.)

403 ForbiddenAn HTTP 403 client-error status code meaning the server understood the request but refuses to fulfill it — access is denied, whether or not credentials are involved. For SEO, a 403 served to Googlebot on a page meant to be public and indexable usually points to a misconfigured rule; a persistent 403 keeps a page out of the index either way.. Before reaching for an IP allowlist, work through the likely causes in order rather than assuming the first one you find is the only one:

  1. robots.txt — confirm the exact path isn’t disallowed for that crawler’s user-agent; this is the single most common cause and the fastest to rule out.
  2. Intermittent or cached blocks — re-test the same URL more than once, from more than one vantage point. A WAF/CDN rule that only trips occasionally, or a stale cached 403, can look like a permanent block when it isn’t.
  3. User-agent blocks — a WAF/CDN rule denying a specific bot string (and remember the string alone can be spoofed, so don’t stop the investigation here).
  4. IP-range blocks — the firewall, CDN, or geoGenerative Engine Optimization — visibility inside AI answer engines. rule denying Googlebot’s actual IP ranges or datacenter geography. Only whitelist verified Googlebot — confirmed via reverse + forward DNS, not just the user-agent header — once you’ve actually located which layer is blocking it. (See blocked 403 and blocked other 4xx.)

Blocked JS/CSS. Google renders in a headless Chrome. If robots.txt blocks *.js/*.css, or a CDN refuses Googlebot the resources, the rendered page looks broken — content that only appears after rendering may not be indexed, and links that only appear after JS may not be followed. Test with URL Inspection → Live Test and look at the rendered screenshot.

TIP Check AI-crawler access separately

When the question is whether ChatGPT, Claude, or another AI system can fetch the site, compare each named crawler's effective robots policy with the AI-Crawler Access Checker Free

  1. Enter the public site or robots.txt URL you want to review.
  2. Compare search, training, and user-triggered agents instead of treating every AI bot as one crawler.
  3. Treat an allowed verdict as access-policy evidence, not proof of indexing, citation, or security.
AI crawler access is agent-specific: the same robots.txt can allow search and user-triggered agents while blocking a training crawler.

The completed access table lists three crawlers. OAI-SearchBot from OpenAI is verifiable and allowed because no rule matches. ClaudeBot from Anthropic is marked unverifiable and allowed because no rule matches. GPTBot from OpenAI is verifiable and blocked by Disallow slash on line 2.

Non-crawlable links. “Google can only crawl your link if it’s an <a> HTML element with an href attribute. Most links in other formats won’t be parsed and extracted by Google’s crawlers.” That rules out <a routerLink="...">, <span href="...">, and <a onclick="goto(...)">. If your main navigation is JS-only click handlers, whole sections of the site may never be discovered. (See the rendering section of the crawling hub and my JavaScript SEO Issues & Best Practices.)

Orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site.. A page with no internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. pointing to it can only be found via a sitemap or external link. Links are the dominant discovery channel, so an orphan often shows up as discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal. in GSC. A sitemap entry aids discovery; it doesn’t replace the link signal. Fix: link the page from a relevant hub or category page. (See discovery and crawl depthCrawl depth usually means click depth — how many clicks it takes to reach a page from the homepage by following internal links. It can also mean a crawler setting that limits how many levels deep a crawl goes before it stops..)

Deep pages. Pages buried many clicks from the homepage get crawled later and less often. Keep important pages shallow (see crawl depth).

3. The server is failing or throttling

5xx server errors. A 500/502/503/504 returns an error instead of content. Sustained 5xx makes Googlebot throttle — “they try not to crawl the site too fast to avoid overloading it… HTTP 500 errors mean ‘slow down’” — and eventually drop already-indexed pages. If robots.txt itself returns 5xx, it’s not an instant, indefinite site-wide halt: Google’s published behavior is staged — crawling stops for about 12 hours first, then Google may fall back to the last known-good robots.txt for up to 30 days, and what happens after that depends on the cached rules and whether the site is generally reachable. Note the 503 nuance: a 503 with a Retry-After header is the correct response for planned short downtime; only weeks of it cause harm. Google buckets 429 (too many requests) with the same back-off behavior. (See server error (5xx) and crawl rate.)

DNS failures. Upstream of HTTP entirely: if the domain doesn’t resolve — expired registration, a botched nameserver migration, broken apex-domain resolution — there’s no server response to even return a 5xx. Sustained DNS failure drops the whole site. Catch it in GSC Crawl Stats → Host StatusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity., or faster with external uptime monitoring.

Slow responses and fetch truncation. Google’s politeness mechanism throttles based on server health, so slow responses mean fewer crawls. And there’s a hard byte limit: as of Google’s March 2026 Inside Googlebot update, Googlebot fetches roughly 2 MB per URL (down from the old 15 MB figure), with PDFs up to 64 MB. Go over and the fetch is truncated, not rejected — only the downloaded portion is indexed. If your critical content sits below 2 MB of bloat, it can get cut off.

Redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. and loops. Each redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. hop costs a crawl credit, and long chains have a negative effect on crawling; Google follows only about five hops before it may give up on the destination. Loops time out and land in redirect error in GSC. Common cause: stacked www → non-www → HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.' hops from successive migrations. (See crawl budgetThe number of URLs an engine will crawl in a timeframe., which lists redirect chains as crawl waste.)

4. The URL inventory is bloated

Spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.. Structures that generate effectively infinite URLs — faceted filter combinations, calendar “next month” links with no end, session IDs in URLs, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does.. Crawlers burn budget on near-duplicate junk while real pages get crawled less. Google has called faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals. and action parameters responsible for the bulk of its crawling issues, and “infinite spaces” has topped the crawl-waste list since 2008. There’s no fixed universal threshold for what counts as “too much” — the signature is a lopsided, disproportionate share of hits concentrated on one junk URL pattern relative to your own site’s normal baseline, not a specific percentage that applies everywhere. (See spider traps.)

URL parametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO.. Query strings (?key=value) create multiple URLs for the same content — active params (sort/filter/pagination) make near-duplicates; passive params (tracking, session IDs) make pure duplicates. Both waste budget and dilute link signals. Note: Google retired its URL Parameters tool in April 2022 (it found only ~1% of configs useful and now handles most cases automatically) — don’t tell people to use it. Bing still has an Ignore URL Parameters tool. Current fix: consistent parameter ordering, rel=canonical for passive params, robots.txt disallow only for genuine traps. (See url parameters and canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it..)

Soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing.. A page that’s functionally “not found” (empty search results, gutted out-of-stock product) but returns 200 OK. Google keeps crawling it because the status says it’s there, wasting budget for no indexable content. Fix: return a real 404 or 410 (410 Gone is marginally faster for removal). A genuine 404 is a strong signal not to crawl that URL again.

Crawl budget exhaustion (large sites only)

This is the bucket-4 problem at scale, and it’s worth its own warning: most sites never hit it. Google says plainly, “if your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” John Mueller has put a number on it — 100,000 URLs is usually not enough to affect crawl budget. It realistically matters at ~1M+ pages changing weekly or 10k+ changing daily. Budget is per hostname, and all of Google’s crawlers (image, video, news, ads) draw from the same per-site pool. The warning light is a large and growing discovered – currently not indexed count. (Full treatment in crawl budget; I also wrote When Should You Worry About Crawl Budget?.)

How to find crawlability issues

Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results..

  • Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. surfaces the named failures: blocked by robots.txt, server error (5xx), not found (404), redirect errorA Google Search Console Page Indexing status meaning Googlebot couldn't follow a redirect — a chain too long, a loop, a URL over the max length, or a bad/empty URL in the chain. It's a broken redirect to fix, not the normal \"Page with redirect.\".
  • Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). shows response-code trends, average response time, and a host-status panel that flags DNS, robots.txt, and server-connectivity problems. (See crawl stats report.)
  • URL Inspection (Live Test) shows what Googlebot actually fetched and rendered for a single URL — the rendered screenshot is the fastest way to catch blocked JS/CSS.
  • “Discovered – currently not indexed” is the crawl warning light: Google knows the URL but hasn’t fetched it (usually a linking or priority problem).

Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. is the ground truth. Logs show exactly which URLs bots hit, how often, and with what status — and they’re the only way to tell real Googlebot (verified via reverse + forward DNS) from bots faking the user-agent. A spider trap shows up as a lopsided share of hits to one URL pattern. (See log file analysis.)

Site audit crawlers — Ahrefs Site Audit and Screaming Frog SEO Spider — simulate a crawl and surface redirect chains, blocked URLs, orphan pages, and crawl depth in one pass.

Crawlability vs. indexability: don’t confuse them

In short: crawlability is whether a bot can reach the page; indexability is whether Google chooses to store it once fetched. Once crawlability is fixed, indexability is the next gate.

The GSC status is a useful starting hypothesis, not a guaranteed diagnosis. Discovered – currently not indexed usually points to a crawl scheduling/priority problem (add internal links, raise site quality). Crawled – currently not indexed most often points to a content/quality problem downstream of crawlability (thin, duplicate, or intent-mismatched content) — but Google doesn’t publish an exhaustive causal mapping for that status, and overlapping factors like rendering delays, canonicalization signals, or indexing-eligibility checks can produce the same label. Treat the status as your first lead, confirm with URL Inspection and logs, and don’t throw crawl fixes at what turns out to be an indexability problem, or vice versa.

How Bing crawlability differs from Google

A few practical differences worth knowing if you optimize for both:

  • IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — Bing (and Yandex, Naver, Seznam, Yep) take it for instant change notification; Google does not participate. Don’t expect IndexNow to help Google crawlability.
  • Crawl Control — Bing still has a manual grid in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. to schedule crawl hours; Google removed its crawl-rate slider in January 2024.
  • Ignore URL Parameters — Bing still offers it; Google retired its equivalent in April 2022.
  • lastmod — Bing weights it more heavily as a recrawl signal; Google largely ignores inaccurate lastmod.

Fabrice Canel of Microsoft Bing sums up Bing’s philosophy as “Less is more for SEO… Less URLs to crawl, better for SEO” — which is the whole crawlability game in one line. (See bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share..)

Where to go next

This hub is the map; the deep dives do the work. For access controls, see robots.txt and noindex. For the server side, see server error (5xx), crawl rate, and redirect error. For inventory bloat, see spider traps, url parameters, and crawl budget. For discovery and links, see discovery, crawl depth, and orphan pages. And for the whole pipeline in one place, start at the crawling hub.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.