Spider Traps (Crawler Traps)

Spider traps (crawler traps) are site structures that generate effectively infinite URLs — faceted filters, calendars, session IDs, redirect loops — and quietly drain crawl budget. How to find and fix them.

First published: Jun 22, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #5 in Crawling#11 in How Search Works#71 in Technical SEO#95 on the site
AIAuthored byAI v2
1 evidence signal on this page

A spider trap (crawler trap) is any part of a site that mints an effectively infinite number of URLs — faceted filters, endless calendars, session IDs, sort/tracking params, infinite pagination, redirect loops, relative-link explosion — so crawlers burn their budget on near-duplicate junk instead of your real content. Google named the problem 'infinite spaces' back in 2008; in a February 2026 Search Off the Record recap of Google's 2025 year-end crawling report, Gary Illyes put faceted navigation and action parameters at roughly 75% of the crawling issues Google sees (50% faceted nav, 25% action parameters, plus smaller shares from tracking params and plugins). It mainly hurts large, new, and ecommerce sites; small sites rarely need to care. Detect it in your server logs first — the signature is one URL pattern eating a disproportionate, lopsided share of crawl hits relative to your own site's normal baseline, not a fixed percentage that applies everywhere — then fix it at the source — ideally stop generating the URLs, or robots.txt-disallow / noindex / 404 in the right order. Canonical and nofollow help but don't stop the crawling, and order matters: block too early and the noindex never gets seen.

TL;DR — A spiderA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. trap (crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. trap) is any structure that generates effectively infinite URLs — faceted nav, calendars, session IDs, sort/tracking params, infinite paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does., redirect loopsA redirect loop is a chain of redirects that circles back on itself instead of ever reaching a live page — URL A redirects to B and B redirects back to A (or a longer cycle). No page ever returns a 200, so browsers show ERR_TOO_MANY_REDIRECTS and crawlers can't index anything., relative-link explosion — so crawlers waste capacity on near-duplicateThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. junk. Google has long described these patterns as “infinite spaces.” The impact is usually greatest on large, new, or ecommerce sites, though smaller sites can still create traps. Find it in logs first; fix it at the source. Robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. stops crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. but not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. needs crawl access to be seen, canonical is a slow hint — so the order of your fixes matters.

Evidence for this claim Faceted navigation can generate effectively unbounded URL spaces that consume crawling resources; Google recommends controlling crawlable combinations. Scope: Current Google faceted-navigation crawl guidance. Confidence: high · Verified: Google Search Central: Managing faceted navigation Evidence for this claim Large duplicate or low-value URL inventories can waste crawl activity, while most smaller sites do not need specialized crawl-budget management. Scope: Current Google crawl-budget guidance. Confidence: high · Verified: Google Search Central: Crawl budget management

What a spider trap actually is

A spider trapA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. — synonymous with crawler trap — is a part of a site that spawns an effectively infinite, or impractically large, number of URLs, so a crawler keeps requesting low-value, near-duplicate, or empty pages and burns its crawl budgetThe number of URLs an engine will crawl in a timeframe. before it reaches your real content. Google’s term for the underlying phenomenon is “infinite spaces” (or “infinite URL spaces”).

The key word is effectively. A trap doesn’t have to be truly infinite — “de facto infinite” is plenty. A few filters that combine in any order produce an astronomical number of URLs for the same handful of products. The problem isn’t that any single URL is bad; it’s the combinatorial explosion.

And here’s why crawlers can’t just sidestep it. As Google’s faceted-navigation documentation puts it, “Because the URLs created for the faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals. seem to be novel and crawlers can’t determine whether the URLs are going to be useful without crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. first, the crawlers will typically access a very large number of faceted navigation URLs.” Gary Illyes said the same thing more plainly: once a crawler discovers a set of URLs, it can’t judge whether that URL space is good until it has crawled a large chunk of it. The trap works precisely because the bot has to walk into it to find out it’s a trap.

Why spider traps hurt SEO

The damage is opportunity cost. Google is explicit: “If crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.” The web, as Google notes, is “a nearly infinite space, exceeding Google’s ability to explore and index every available URL” — so a finite crawl pool spent on your junk is content that never gets discovered.

This is a crawl-budget problem, and crawl budget is an efficiency concern, not a ranking signal. More crawling doesn’t lift your rankings — as I’ve written in my crawl budget guide, “More crawling doesn’t mean you’ll rank better, but if your pages aren’t crawled and indexed they aren’t going to rank at all.” That’s the real stakes: a bad enough trap on a large site can leave genuinely important pages stuck in “Discovered – currently not indexed” because the budget went to junk.

Who actually needs to care? Google’s own crawl-budget guide scopes itself to large sites (1 million+ unique pages, changing at least weekly), medium-or-larger sites (10,000+ unique pages) that change daily, and any site with a large share of its URLs sitting in “Discovered – currently not indexedA Google Search Console Page Indexing status meaning Google knows the URL exists but hasn't crawled it yet — the Last Crawl date is empty. Often a crawl-capacity or crawl-demand (site-quality) signal.” — plus, by extension, ecommerce sites with faceted catalogs, which are the fastest way to hit those numbers. Google calls these rough estimates, not exact thresholds. Most small sites get crawled fully regardless and can stop reading here. The trap scenario is exactly the case where crawl budget does matter — and Google frames the whole concept around crawl capacity (how much your server can take) and crawl demand (how much Google wants to crawl), not a fixed quota handed to every site.

This isn’t a new or rare problem

Google named “infinite spaces” back in 2008, in a post titled To infinity and beyond? No!, calling out endless calendars and stacked search-result filters as the source, and naming robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. and nofollow as the fixes. (That post is old and JS-rendered, so I’m paraphrasing it rather than quoting it.)

What’s striking is that it never went away — it scaled. Google ran a Crawling December blog series on faceted navigation back in December 2024, and more recently, on a Search Off the Record episode recapping Google’s 2025 year-end crawling report, Gary Illyes broke down what’s actually driving crawl waste: faceted navigation and action parameters together account for roughly 75% of the crawling issues Google sees — 50% from faceted navigation, 25% from action parameters (URL parametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO. that trigger an action rather than changing page content). The remaining quarter is smaller stuff: about 10% from “irrelevant” parameters like session IDs and UTM tags, 5% from plugins and widgets that generate problematic URLs, and 2% from other edge cases like double-encoded URLs. Eighteen years on, faceted navigation is still, by a wide margin, the single biggest source of the problem.

Types of spider traps

The catalogue, with the one-line tell for each. (See the Cheat Sheets tab for the matching one-line fixes.)

  • Faceted navigation / filter combinations — the #1 trap. Users pick a few filters; the crawler tries every permutation. Google: “This often means a very large number of possible combinations of filters, which translates to a very large number of possible URLs.”
  • Infinite calendars — “next month / next year” links with no horizon. A bot can walk forward indefinitely (Illyes’ memorable example was a calendar widget generating a valid URL for the year 3000).
  • Session IDs in URLs — ?sid=, jsessionid, etc. mint a unique URL per visitor for identical content.
  • Sort / order parameters — ?sort=price, ?order=desc. Google says “differently sorted versions of the same page” shouldn’t be crawled.
  • Infinite scrollInfinite scroll is a loading pattern where content appears automatically as a user scrolls, instead of via numbered pages. Because Googlebot doesn't scroll or click, indexable infinite scroll needs real per-chunk URLs (paginated loading) that update via the History API — otherwise deep content may never be crawled, or worse, get merged into the wrong page. / looping paginationPagination splits a large set of content — product listings, blog archives, search results — across multiple sequentially numbered URLs. For SEO, each paginated page should be crawlable, indexable, and self-canonical; Google no longer uses rel=prev/next, but Bing still does. — endless “load more” / “next” that never terminates or duplicates linked content. Google flags “infinite scrolling pages that duplicate information on linked pages.”
  • Redirect loopsA redirect loop is a chain of redirects that circles back on itself instead of ever reaching a live page — URL A redirects to B and B redirects back to A (or a longer cycle). No page ever returns a 200, so browsers show ERR_TOO_MANY_REDIRECTS and crawlers can't index anything. / long chains — A → B → A forever, often from a mistyped rewrite rule.
  • Relative-URL / path explosion — malformed relative links that keep appending directories (/abc/def/abc/def/abc/def/…) until the server gives up.
  • Internal search result pages — search URLs treated as crawlable content; a sitewide search on a single letter can mint a million pages.
  • Dynamic / “magic” URLs — a URL that accepts arbitrary text or IDs and still returns 200.
  • Tracking / “action” parameters — UTM tags and add-to-cart/compare actions create de-facto-infinite URL variants (this is the “action parameters” half of Google’s 75%).

How to detect a spider trap

  • Server logs first — this is the fastest tell. Your logs show exactly which URLs bots hit and how often. There’s no fixed universal threshold for what counts as “too much” — the signature is a lopsided, disproportionate share of crawl hits concentrated on one junk URL pattern relative to your own site’s normal baseline, not a specific percentage that applies everywhere. (Yoast founder Joost de Valk has written that in his own experience it’s “not uncommon” for a trap to take up 20-30% or more of all crawls — a useful gut-check for how skewed a real trap can get, not a benchmark to test your site against.) (See log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..)
  • Run a crawl — Screaming Frog, Ahrefs Site Audit, Sitebulb. Watch for a crawl that won’t finish, URLs that keep getting longer (path explosion), or an exploding count of parameter combinations.
  • Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — a ballooning “Discovered – currently not indexed” count, a swelling Soft 404A soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing. report (autogenerated empty pages), Crawl Stats skewed toward junk paths, or the “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. encountered an extremely high number of URLs” message.
  • A site/index sanity check — when a tiny fraction of indexed URLs actually drive traffic, you’re probably feeding a trap.
  • Search operators — site: plus inurl: on a known parameter string.
  • Bing — Site Scan and Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility..
TIP Find the URL pattern consuming crawler requests

A crawler can reveal an expanding URL space, but access logs show whether search bots are actually spending requests there. The tool reports the paths and response classes; you still decide which pattern is a trap.

Drop a representative access-log export into my Log File Analyzer and sort the wasted-crawl paths before changing robots, canonicals, or routing. Log File Analyzer Free

  1. Filter to verified search crawlers and identify repeated junk path or parameter patterns.
  2. Trace the internal links, controls, or templates generating those URLs.
  3. Fix the source, apply the appropriate crawl/indexing control, and confirm the request share falls.
The wasted-URL table behind this summary provides the exact paths needed to trace a trap back to its source.

How to fix and prevent spider traps

The fixes, roughly best to last-resort — and the sequencing matters, because these controls do different jobs.

  1. Best fix: don’t generate the URLs. Stop minting crawlable URLs at the source. The cleanest version is JavaScript/fragment-based filtering so filters update the page without creating a new crawlable address (Google’s faceted-nav doc and SEJ both recommend converting facet parameters to URL fragments). Everything below is a patch by comparison.
  2. Pick which facets deserve indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., block the rest. Google: “Oftentimes there’s no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit.” Allow one unfiltered listing plus the individual item pages; block the combinatorial filtered space.
  3. robots.txt disallow the trap patterns for spaces you don’t need indexed. This is Google’s primary recommendation for filtered/sorted junk. Caveat: robots.txt blocks crawling, not indexing — see the sequencing note below.
  4. noindex to remove already-indexed junk. If the junk is already in the index, you can’t robots-block it first — Google has to be able to crawl the page to see the noindex. The correct order is: keep it crawlable, let noindex deindex it, then (optionally) robots.txt-disallow to stop future crawling. Block too early and the noindex never gets seen.
  5. Return 404/410 for empty filter combinations. Google: “Return an HTTP 404 status code when a filter combination doesn’t return results.” And “Eliminate soft 404 errors. Soft 404 pages will continue to be crawled, and waste your budget.”
  6. rel="canonical" to consolidate sorted/variant URLs to the clean version — useful, but slow and a hint, not a directive. Google: it “may, over time, decrease the crawl volume of non-canonical versions of those URLs.” It doesn’t stop the crawling, and Bing warns “relying on canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. is not necessarily the perfect solution to fix all your duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. problems.”
  7. rel="nofollow" on filter links — only works if you apply it to every link, internal and external. Google: “Every anchor pointing to a specific URL must have the rel=“nofollowrel=\"nofollow\" is a value of the HTML link rel attribute that tells search engines you don't vouch for a linked page and don't want to pass ranking signals to it. Since 2019–2020 Google treats it as a hint, not a directive — and it does not reliably block crawling or indexing.” attribute in order for it to be effective.”
  8. Per-trap fixes: cap calendar links past a sane horizon and noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed. distant dates; strip session IDs out of URLs (use cookies); fix redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. loops so the source redirects straight to the final destination; use absolute URLs and server rules to stop relative-link explosion; noindex/disallow internal search pages.

If you keep clean URL parameters and a stable parameter order, you also avoid minting needless variants in the first place. Note that Google retired its URL Parameters tool in 2022, so robots.txt / canonical / noindex now carry that load on Google’s side. On Bing, the equivalent is URL Normalization — you tell Bing which parameters to ignore.

One nuance: don’t break legitimate pagination

The over-correction trap is real too. When people panic about “infinite” pagination, they often block or canonicalize all paginated pages — and create a new problem. As I’ve written about pagination, “Blocking the pages from crawling will again make it more difficult to find content on the website, end up orphaning pages,” and canonicalizing every page to page one “makes it harder for search engines to find and index valuable content, and also cuts off the flow of PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems..” Distinguish a genuine infinite/looping pagination trap from legitimate finite pagination — keep the finite version crawlable.

Where this sits

Spider traps are a crawl-efficiency problem, so they live next to crawl budget (what the trap drains) and log file analysis (how you catch it). If you’ve read the crawling hub, this is the failure mode that makes crawl budget worth managing on the sites where it matters.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.