Spider Traps (Crawler Traps)

TL;DR — A spider trap (or crawler trap) is a part of your site that accidentally creates a near-endless list of URLs — think filter combinations, a calendar with a “next month” link forever, or web addresses with a session ID stuck on them. Search engine bots get stuck crawling all that junk instead of your real pages. It wastes their time, and on big sites it means your good pages get crawled less.

What a spider trap is

When a search engine visits your site, it follows links and downloads pages — that’s crawling. A spider trap is a structure that hands the crawler a basically infinite supply of links to follow, most of them pointing at near-identical or empty pages. The bot keeps fetching them, never running out, and never finding anything new.

“Spider trap” and “crawler trap” are the same thing — spider and crawler are both nicknames for the bots. Google’s own name for the problem is “infinite spaces.”

What causes them

A few classics:

• Filters — pick a color, a size, a brand, a price range, sort by cheapest… every combination is a new URL, and they multiply fast.
• Calendars — a “next month” link with no end. A bot can click forward into the year 3000.
• Session IDs — a unique code added to the URL for every visitor, so the same page exists under thousands of addresses.
• Redirect loops — page A sends the bot to B, B sends it back to A, forever.

Why it matters (and when it doesn’t)

A bot only has so much time for your site. Every minute spent on junk URLs is a minute it isn’t spending on your real content. On a small site this rarely matters — there’s plenty of time to crawl everything. On a large site, or a brand new one, or a big online store, it can mean important pages get crawled slowly or not at all.

How you’d spot one and fix it

The quickest tell is your server logs — if one weird pattern of URLs is eating a disproportionate, lopsided share of all the crawling compared to your site’s normal baseline, that’s a trap. The best fix is stop generating the URLs in the first place (for example, make filters work without creating a new web address each time). If you can’t, you block the junk patterns so bots skip them.

Want the full type-by-type catalogue, the detection workflow, and the right order to apply each fix? Switch to the Advanced tab.

TL;DR — A spider trap (crawler trap) is any structure that generates effectively infinite URLs — faceted nav, calendars, session IDs, sort/tracking params, infinite pagination, redirect loops, relative-link explosion — so crawlers waste capacity on near-duplicate junk. Google has long described these patterns as “infinite spaces.” The impact is usually greatest on large, new, or ecommerce sites, though smaller sites can still create traps. Find it in logs first; fix it at the source. Robots.txt stops crawling but not indexing, noindex needs crawl access to be seen, canonical is a slow hint — so the order of your fixes matters.

What a spider trap actually is

A spider trap — synonymous with crawler trap — is a part of a site that spawns an effectively infinite, or impractically large, number of URLs, so a crawler keeps requesting low-value, near-duplicate, or empty pages and burns its crawl budget before it reaches your real content. Google’s term for the underlying phenomenon is “infinite spaces” (or “infinite URL spaces”).

The key word is effectively. A trap doesn’t have to be truly infinite — “de facto infinite” is plenty. A few filters that combine in any order produce an astronomical number of URLs for the same handful of products. The problem isn’t that any single URL is bad; it’s the combinatorial explosion.

And here’s why crawlers can’t just sidestep it. As Google’s faceted-navigation documentation puts it, “Because the URLs created for the faceted navigation seem to be novel and crawlers can’t determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs.” Gary Illyes said the same thing more plainly: once a crawler discovers a set of URLs, it can’t judge whether that URL space is good until it has crawled a large chunk of it. The trap works precisely because the bot has to walk into it to find out it’s a trap.

Why spider traps hurt SEO

The damage is opportunity cost. Google is explicit: “If crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.” The web, as Google notes, is “a nearly infinite space, exceeding Google’s ability to explore and index every available URL” — so a finite crawl pool spent on your junk is content that never gets discovered.

This is a crawl-budget problem, and crawl budget is an efficiency concern, not a ranking signal. More crawling doesn’t lift your rankings — as I’ve written in my crawl budget guide, “More crawling doesn’t mean you’ll rank better, but if your pages aren’t crawled and indexed they aren’t going to rank at all.” That’s the real stakes: a bad enough trap on a large site can leave genuinely important pages stuck in “Discovered – currently not indexed” because the budget went to junk.

Who actually needs to care? Google’s own crawl-budget guide scopes itself to large sites (1 million+ unique pages, changing at least weekly), medium-or-larger sites (10,000+ unique pages) that change daily, and any site with a large share of its URLs sitting in “Discovered – currently not indexed” — plus, by extension, ecommerce sites with faceted catalogs, which are the fastest way to hit those numbers. Google calls these rough estimates, not exact thresholds. Most small sites get crawled fully regardless and can stop reading here. The trap scenario is exactly the case where crawl budget does matter — and Google frames the whole concept around crawl capacity (how much your server can take) and crawl demand (how much Google wants to crawl), not a fixed quota handed to every site.

This isn’t a new or rare problem

Google named “infinite spaces” back in 2008, in a post titled To infinity and beyond? No!, calling out endless calendars and stacked search-result filters as the source, and naming robots.txt and nofollow as the fixes. (That post is old and JS-rendered, so I’m paraphrasing it rather than quoting it.)

What’s striking is that it never went away — it scaled. Google ran a Crawling December blog series on faceted navigation back in December 2024, and more recently, on a Search Off the Record episode recapping Google’s 2025 year-end crawling report, Gary Illyes broke down what’s actually driving crawl waste: faceted navigation and action parameters together account for roughly 75% of the crawling issues Google sees — 50% from faceted navigation, 25% from action parameters (URL parameters that trigger an action rather than changing page content). The remaining quarter is smaller stuff: about 10% from “irrelevant” parameters like session IDs and UTM tags, 5% from plugins and widgets that generate problematic URLs, and 2% from other edge cases like double-encoded URLs. Eighteen years on, faceted navigation is still, by a wide margin, the single biggest source of the problem.

Types of spider traps

The catalogue, with the one-line tell for each. (See the Cheat Sheets tab for the matching one-line fixes.)

• Faceted navigation / filter combinations — the #1 trap. Users pick a few filters; the crawler tries every permutation. Google: “This often means a very large number of possible combinations of filters, which translates to a very large number of possible URLs.”
• Infinite calendars — “next month / next year” links with no horizon. A bot can walk forward indefinitely (Illyes’ memorable example was a calendar widget generating a valid URL for the year 3000).
• Session IDs in URLs — ?sid=, jsessionid, etc. mint a unique URL per visitor for identical content.
• Sort / order parameters — ?sort=price, ?order=desc. Google says “differently sorted versions of the same page” shouldn’t be crawled.
• Infinite scroll / looping pagination — endless “load more” / “next” that never terminates or duplicates linked content. Google flags “infinite scrolling pages that duplicate information on linked pages.”
• Redirect loops / long chains — A → B → A forever, often from a mistyped rewrite rule.
• Relative-URL / path explosion — malformed relative links that keep appending directories (/abc/def/abc/def/abc/def/…) until the server gives up.
• Internal search result pages — search URLs treated as crawlable content; a sitewide search on a single letter can mint a million pages.
• Dynamic / “magic” URLs — a URL that accepts arbitrary text or IDs and still returns 200.
• Tracking / “action” parameters — UTM tags and add-to-cart/compare actions create de-facto-infinite URL variants (this is the “action parameters” half of Google’s 75%).

How to detect a spider trap

• Server logs first — this is the fastest tell. Your logs show exactly which URLs bots hit and how often. There’s no fixed universal threshold for what counts as “too much” — the signature is a lopsided, disproportionate share of crawl hits concentrated on one junk URL pattern relative to your own site’s normal baseline, not a specific percentage that applies everywhere. (Yoast founder Joost de Valk has written that in his own experience it’s “not uncommon” for a trap to take up 20-30% or more of all crawls — a useful gut-check for how skewed a real trap can get, not a benchmark to test your site against.) (See log file analysis.)
• Run a crawl — Screaming Frog, Ahrefs Site Audit, Sitebulb. Watch for a crawl that won’t finish, URLs that keep getting longer (path explosion), or an exploding count of parameter combinations.
• Google Search Console — a ballooning “Discovered – currently not indexed” count, a swelling Soft 404 report (autogenerated empty pages), Crawl Stats skewed toward junk paths, or the “Googlebot encountered an extremely high number of URLs” message.
• A site/index sanity check — when a tiny fraction of indexed URLs actually drive traffic, you’re probably feeding a trap.
• Search operators — site: plus inurl: on a known parameter string.
• Bing — Site Scan and Crawl Control in Bing Webmaster Tools.

How to fix and prevent spider traps

The fixes, roughly best to last-resort — and the sequencing matters, because these controls do different jobs.

1. Best fix: don’t generate the URLs. Stop minting crawlable URLs at the source. The cleanest version is JavaScript/fragment-based filtering so filters update the page without creating a new crawlable address (Google’s faceted-nav doc and SEJ both recommend converting facet parameters to URL fragments). Everything below is a patch by comparison.
2. Pick which facets deserve indexing, block the rest. Google: “Oftentimes there’s no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit.” Allow one unfiltered listing plus the individual item pages; block the combinatorial filtered space.
3. robots.txt disallow the trap patterns for spaces you don’t need indexed. This is Google’s primary recommendation for filtered/sorted junk. Caveat: robots.txt blocks crawling, not indexing — see the sequencing note below.
4. noindex to remove already-indexed junk. If the junk is already in the index, you can’t robots-block it first — Google has to be able to crawl the page to see the noindex. The correct order is: keep it crawlable, let noindex deindex it, then (optionally) robots.txt-disallow to stop future crawling. Block too early and the noindex never gets seen.
5. Return 404/410 for empty filter combinations. Google: “Return an HTTP 404 status code when a filter combination doesn’t return results.” And “Eliminate soft 404 errors. Soft 404 pages will continue to be crawled, and waste your budget.”
6. rel="canonical" to consolidate sorted/variant URLs to the clean version — useful, but slow and a hint, not a directive. Google: it “may, over time, decrease the crawl volume of non-canonical versions of those URLs.” It doesn’t stop the crawling, and Bing warns “relying on canonical tag is not necessarily the perfect solution to fix all your duplicate content problems.”
7. rel="nofollow" on filter links — only works if you apply it to every link, internal and external. Google: “Every anchor pointing to a specific URL must have the rel=“nofollow” attribute in order for it to be effective.”
8. Per-trap fixes: cap calendar links past a sane horizon and noindex distant dates; strip session IDs out of URLs (use cookies); fix redirect loops so the source redirects straight to the final destination; use absolute URLs and server rules to stop relative-link explosion; noindex/disallow internal search pages.

If you keep clean URL parameters and a stable parameter order, you also avoid minting needless variants in the first place. Note that Google retired its URL Parameters tool in 2022, so robots.txt / canonical / noindex now carry that load on Google’s side. On Bing, the equivalent is URL Normalization — you tell Bing which parameters to ignore.

One nuance: don’t break legitimate pagination

The over-correction trap is real too. When people panic about “infinite” pagination, they often block or canonicalize all paginated pages — and create a new problem. As I’ve written about pagination, “Blocking the pages from crawling will again make it more difficult to find content on the website, end up orphaning pages,” and canonicalizing every page to page one “makes it harder for search engines to find and index valuable content, and also cuts off the flow of PageRank.” Distinguish a genuine infinite/looping pagination trap from legitimate finite pagination — keep the finite version crawlable.

Where this sits

Spider traps are a crawl-efficiency problem, so they live next to crawl budget (what the trap drains) and log file analysis (how you catch it). If you’ve read the crawling hub, this is the failure mode that makes crawl budget worth managing on the sites where it matters.

AI summary

A condensed take on the Advanced version:

• A spider trap = a crawler trap = a structure that generates effectively infinite URLs, so crawlers waste budget on near-duplicate junk. Google’s name for it is “infinite spaces.”
• It doesn’t have to be truly infinite — combinatorial (“de facto infinite”) is enough. Crawlers can’t tell a URL is junk without crawling it, so they walk in.
• The cost is opportunity cost: “If crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.” It’s a crawl-budget efficiency issue, not a ranking signal.
• Who cares: large / new / fast-changing / ecommerce sites. Small sites usually don’t.
• Not new, just scaled: Google named it in 2008; Gary Illyes (Feb 2026, on Google’s 2025 year-end crawling report) put faceted navigation + action parameters ≈ 75% of its crawling issues (50% facets, 25% action params, plus smaller shares from tracking params and plugins).
• Types: faceted nav, infinite calendars, session IDs, sort/tracking params, infinite scroll/pagination loops, redirect loops, relative-URL explosion, internal search, “magic” URLs.
• Detect: logs first — one pattern eating a disproportionate share of crawls relative to your own baseline, not a fixed percentage — then crawl tools, GSC (“Discovered – currently not indexed”, Soft 404, “extremely high number of URLs”), Bing Site Scan.
• Fix, in order: stop generating the URLs (JS/fragment filters) → choose which facets to index, block the rest → robots.txt for junk → noindex to remove already-indexed junk (let it deindex before you robots-block) → 404/410 for empty results → canonical/nofollow as slow hints. Robots.txt ≠ noindex ≠ canonical — sequencing matters.

Official documentation

Primary-source documentation on infinite spaces, faceted navigation, and crawl budget.

Google

• To infinity and beyond? No! — the original 2008 post that named “infinite spaces,” with the endless-calendar and stacked-filter examples and the robots.txt / nofollow fixes.
• Managing crawling of faceted navigation URLs — the best current verbatim source: why facets over-crawl, when to block them, and the indexing best practices (separator, stable filter order, 404 on empty, canonical, nofollow). (Older URL search/docs/crawling-indexing/crawling-managing-faceted-navigation now 301-redirects here.)
• Faceted navigation best (and 5 of the worst) practices — the 2014 post; historical proof faceted nav has been a named trap for over a decade.
• Crawl Budget Management — the “nearly infinite space” framing, the crawl-capacity-vs-crawl-demand model, who this guide is for (site-size thresholds), plus the instruction to not crawl infinite-scroll duplicates and differently sorted versions, and to eliminate soft 404s. (This page now also contains the content that used to live at the separate “Large site owner’s guide to managing crawl budget” URL, which redirects here.)
• Crawling December: Faceted navigation (2024) — Google’s December 2024 blog post on faceted navigation specifically; a separate, earlier item from the February 2026 crawl-issue breakdown below.
• GSC Help: Googlebot encountered an extremely high number of URLs — the in-product signal of a trap.

Bing / Microsoft

• Better than canonical; URL Normalization — Bing’s stance that “the real number of URLs on the Internet is infinity,” the calendar “Next Day” example, and URL Normalization as Bing’s preferred fix for parameter duplicates.
• Bing Webmaster Tools — Site Scan — on-demand site audit.
• Bing Webmaster Tools — Block URLs — temporarily block URL patterns (pair with a NOINDEX meta tag).
• Bing Webmaster Tools — Crawl Control — throttle Bingbot.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — faceted navigation (the #1 trap)

• “This often means a very large number of possible combinations of filters, which translates to a very large number of possible URLs.” Jump to quote
• “Because the URLs created for the faceted navigation seem to be novel and crawlers can’t determine whether the URLs are going to be useful without crawling first, the crawlers will typically access a very large number of faceted navigation URLs.” Jump to quote
• “If crawling is spent on useless URLs, the crawlers have less time to spend on new, useful URLs.” Jump to quote
• “Oftentimes there’s no good reason to allow crawling of filtered items, as it consumes server resources for no or negligible benefit.” Jump to quote
• “Return an HTTP 404 status code when a filter combination doesn’t return results.” Jump to quote
• On canonical: it “may, over time, decrease the crawl volume of non-canonical versions of those URLs.” Jump to quote
• On nofollow: “Every anchor pointing to a specific URL must have the rel=“nofollow” attribute in order for it to be effective.” Jump to quote

Google — crawl budget & infinite spaces

• “The web is a nearly infinite space, exceeding Google’s ability to explore and index every available URL.” Jump to quote
• Don’t crawl “infinite scrolling pages that duplicate information on linked pages, or differently sorted versions of the same page.” Jump to quote
• “Eliminate soft 404 errors. Soft 404 pages will continue to be crawled, and waste your budget.” Jump to quote

Gary Illyes, Google (Search Off the Record, recapping Google’s 2025 year-end crawling report — reported by Search Engine Land, Feb 2026)

• Faceted navigation and action parameters “accounted for about 75% of the problems” Google encounters with crawling — broken down in the same article as 50% faceted navigation, 25% action parameters, 10% “irrelevant” tracking/session-ID parameters, 5% plugins/widgets, and 2% other edge cases like double-encoded URLs. Jump to quote
• On why crawlers can’t avoid the trap: “Once it discovers a set of URLs, it cannot make a decision about whether that URL space is good or not unless it crawled a large chunk of that URL space.” Jump to quote

Bing / Microsoft — URL Normalization

• “the real number of URLs on the Internet is infinity.” Jump to quote
• “relying on canonical tag is not necessarily the perfect solution to fix all your duplicate content problems.” Jump to quote

Detect-and-fix checklist

Detect

• Pull server logs and look for one URL pattern eating a disproportionate, lopsided share of bot hits relative to your site’s normal baseline — no fixed universal percentage defines “too much.”
• Run a crawl (Screaming Frog / Ahrefs Site Audit / Sitebulb) and watch for a crawl that won’t finish, URLs that keep getting longer, or exploding parameter counts.
• Check GSC: ballooning “Discovered – currently not indexed”, a swelling Soft 404 report, Crawl Stats skewed to junk paths, the “extremely high number of URLs” message.
• Sanity-check indexed vs. traffic-driving URLs — a tiny fraction earning traffic signals a trap.
• Spot-check with site: + inurl: on a known parameter string.
• Check Bing via Site Scan / Crawl Control.

Fix (in order — the sequence matters)

• Stop generating the URLs where you can (JS/fragment-based filtering).
• Decide which facets deserve indexing; block the rest.
• If junk is not yet indexed: robots.txt disallow the trap patterns.
• If junk is already indexed: keep it crawlable, add noindex, let it deindex, then (optionally) robots.txt-disallow. Don’t block first.
• Return 404/410 for empty filter combinations; eliminate soft 404s.
• Use rel=canonical to consolidate sorted/variant URLs (slow hint, not a stop).
• Use rel=nofollow only if applied to every link to those URLs.
• Keep a stable parameter order and clean parameters to avoid minting variants.
• Per-trap: cap calendars, strip session IDs (use cookies), fix redirect loops, use absolute URLs, noindex/disallow internal search.

The mental models

1. “De facto infinite” beats “truly infinite.” A trap doesn’t need to be literally endless. A handful of filters that combine in any order produces an astronomical URL count for the same content. It’s the combinatorial explosion, not any single bad URL, that does the damage.

2. Why crawlers can’t dodge it — the cost of finding out. A crawler can’t tell a URL is junk without fetching it. As Illyes put it, it can’t judge whether a URL space is good “unless it crawled a large chunk of that URL space.” The trap works because the bot has to walk in to learn it’s a trap.

3. The drain is opportunity cost, not a penalty. Nothing about a trap is a ranking penalty. The cost is that budget spent on useless URLs is budget not spent on new, useful ones. Finite crawl pool, infinite junk — your real pages lose the race for attention. This is why it bites large/new/ecommerce sites and ignores small ones.

4. Three controls, three jobs — and an order. robots.txt stops crawling but not indexing. noindex removes from the index but needs the page to be crawlable to be seen. canonical is a slow hint, not a stop. So to remove already-indexed junk: allow crawl → noindex → let it deindex → then robots.txt. Block first and the noindex is never read.

5. Prevention beats patching. The best fix is to never mint the URLs — JS/fragment filters, no session IDs in URLs, bounded calendars. Canonical-after-the-fact is the weakest fix because the crawling still happens.

Trap types → one-line fix

Quick reference

The detection signature: one junk pattern eating a disproportionate, lopsided share of all crawls relative to your own site’s baseline — there’s no fixed universal percentage.

The 75% breakdown (Google’s Gary Illyes, on the 2025 year-end crawling report, reported Feb 2026): faceted nav + action params ≈ 75% of crawling issues (50% faceted nav, 25% action params, 10% tracking/session-ID params, 5% plugins/ widgets, 2% other edge cases).

What each control does

Gone: Google’s URL Parameters tool (retired 2022). Bing equivalent: URL Normalization.

Tools for finding and fixing traps

• Server log file analysis — the ground truth, and the fastest way to spot a trap (look for one pattern dominating bot hits). Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysis.)
• Ahrefs Site Audit — simulates a crawl and surfaces redirect chains, parameter bloat, near-duplicate URL spaces, and crawls that balloon — the classic trap fingerprints.
• Screaming Frog SEO Spider — a desktop crawl that exposes path explosion, redirect loops, and exploding parameter combinations; watch for a crawl that won’t terminate.
• Google Search Console — Crawl Stats (junk-path skew), the Pages report (“Discovered – currently not indexed”, Soft 404), and the “extremely high number of URLs” message.
• Bing Webmaster Tools — Site Scan, Crawl Control, and URL Normalization (tell Bing which parameters to ignore).
• URL Inspection (GSC) — confirm how a single suspect URL was crawled and indexed.

The standing KPI: junk share of crawl

A spider trap doesn’t show up as a single alert — it shows up as a share of your crawl going somewhere it shouldn’t. The honesty check first: this metric matters mainly on large, fast-changing, or faceted-catalog sites; a small site’s crawl is rarely skewed enough to be worth tracking.

Junk-pattern share of crawl requests

• Metric — What fraction of verified-Googlebot (or Bingbot) requests in your server logs hit a single suspect URL pattern — a facet parameter, a session ID, a calendar path — versus your real, indexable content.
• What it tells you — Whether a specific structure is actually draining budget or just theoretically capable of it. A pattern eating a large, disproportionate slice of total crawl hits is the trap signature described earlier in this article.
• How to pull it — Run your raw logs through the Log File Analyzer and group hits by URL pattern; cross-check the same window against the GSC Crawl Stats report broken down by URL path.
• Benchmark / realistic range — No honest universal threshold — it depends on your site’s architecture and how many URL patterns exist at all. Treat it as a baseline you set for your site and drive down over time, not an absolute number to hit.
• Cadence — Check right after you spot a suspected trap or ship a fix; otherwise monthly is enough for most sites. Re-check any time you add new filters, sorting, or a new dynamic section.

”Discovered / Crawled – currently not indexed” count

• Metric — The count of URLs sitting in GSC’s Pages report under “Discovered – currently not indexed” or “Crawled – currently not indexed.”
• What it tells you — A rising count, especially alongside a known trap pattern, is consistent with Google finding URLs it doesn’t consider worth the crawl/index budget — it’s a symptom, not proof of a trap on its own, since plenty of things besides traps cause it.
• How to pull it — GSC → Pages report → filter by the “not indexed” reasons; sample a handful of URLs and check whether they match the trap pattern you’re investigating.
• Benchmark / realistic range — Situational, same caveat as above — track the trend on your own site rather than comparing to anyone else’s count.
• Cadence — Monthly, or right after a fix ships (this count lags the crawl-log metric above by days to weeks, so don’t expect it to move immediately).

Find and fence in a trap pattern

These are starting points, not finished tools — read the output before trusting it, and adjust field positions/paths to match your own log format.

Grep server logs for a suspect URL pattern

Point this at a raw access log (Apache/Nginx combined format) to count how many requests a suspect pattern is pulling, and see whether they’re really coming from Googlebot’s declared user-agent string. A user-agent string alone can be spoofed — confirm the IP with the Googlebot Verifier before treating the hits as real crawl waste and not spoofed traffic:

# Count requests to a suspect pattern (e.g. a filter parameter) by user-agent
grep -i "googlebot" access.log | grep -oE '"[A-Z]+ [^ ]*\?[^ "]*calendar[^ "]*' | wc -l

# Same, but for a session-ID pattern
grep -i "googlebot" access.log | grep -oE '"[A-Z]+ [^ "]*[?&](sid|jsessionid)=[^ "&]*' | wc -l

# Total Googlebot requests in the same window, for the share calculation
grep -ci "googlebot" access.log

Divide the pattern count by the total to get the junk share described in the How to Measure tab.

Regex to catch classic infinite-space parameters

A starting filter list for facets, sorting, session IDs, and tracking/action params — extend it with whatever your own site actually generates:

[?&](sort|order|page|sid|jsessionid|phpsessid|utm_[a-z]+|ref|affid|add_to_cart|compare)=

Run it against a crawl export or log sample to flag candidate trap URLs before you decide whether to robots.txt-disallow, canonicalize, or noindex each one — don’t block by pattern match alone without checking a sample of matched URLs first.

Screaming Frog: catch a crawl that won’t terminate

There’s no single magic setting, but three configuration choices turn Screaming Frog into a trap detector rather than a tool that just hangs:

• Configuration → Spider → Limits — set a URL crawl limit and a max crawl depth so a genuine infinite space stops the crawl instead of running for hours.
• Configuration → URL Rewriting — strip session IDs and known tracking parameters before the crawler treats each one as a new URL.
• Watch the URL length and response codes columns live during the crawl — URLs that keep getting longer (path explosion) or a parameter count that keeps climbing is the crawl-time version of the log signature above.

Prompts for spider-trap triage

Paste in your own log excerpts or URL lists where indicated — verify the model’s output against a sample of the actual URLs before acting on it; it can misjudge which parameters actually change page content.

Flag candidate trap patterns from a URL list

Paste in: a list of URLs pulled from a crawl export, server logs, or a site: search. Expect back: the URLs grouped by likely-trap pattern (faceted filters, calendars, session IDs, sort/tracking params) with a one-line reason each group is suspect.

Here is a list of URLs from my site. Group them by pattern that looks
like a spider trap / infinite space: faceted-navigation filter
combinations, calendar/date paths with no apparent end, session IDs
in the URL, sort or tracking parameters, or redirect-loop candidates.
For each group, give a one-line reason it's suspect and roughly how
many URLs are in it.

URLs:
[paste URL list here]

Draft a robots.txt disallow for a confirmed pattern

Paste in: a confirmed trap URL pattern and your existing robots.txt. Expect back: a disallow rule scoped to that pattern, plus a check for whether it would accidentally catch legitimate pages (like real pagination).

Here is a URL pattern I've confirmed is a spider trap (with example
URLs) and my current robots.txt. Write a Disallow rule scoped to just
this pattern, and tell me if it risks blocking anything else on the
site, especially legitimate finite pagination or individual item pages
that happen to share part of the path.

Pattern + example URLs:
[paste here]

Current robots.txt:
[paste here]

Sanity-check the fix sequencing before I ship it

Paste in: a plain description of what’s currently live for a suspect URL pattern (indexed already? robots.txt blocked? noindex tag present?). Expect back: whether the order is safe, given that noindex needs crawl access to be seen and robots.txt blocks crawling but not indexing.

Here's the current state of a URL pattern I think is a spider trap:
[describe: is it already indexed? currently crawlable? does it have
a noindex tag? is robots.txt already blocking it?]

Tell me if my planned fix sequence is safe, given that: robots.txt
disallow stops crawling but not indexing, and noindex needs the page
to stay crawlable to be seen and processed. Flag if I'm about to block
crawling before a noindex has had a chance to be read.

Prove the fix actually stopped the crawling

Blocking a trap pattern doesn’t mean it stopped draining budget — these checks confirm the crawl requests actually dropped, in order from fastest to slowest to confirm.

Test 1 — The blocked pattern still returns the expected response

• Test to run — Fetch a few URLs matching the pattern directly (curl -I or the Redirect Checker for redirect-based fixes) and check robots.txt itself for the new disallow line.
• Expected result — The disallow rule is present and correctly scoped, and (if the fix is noindex/404 rather than robots.txt) the URL returns the expected status or meta tag.
• Failure interpretation — A missing or mis-scoped rule means the fix never actually shipped, even if you believe it did — check for a typo in the path or a rule that’s too narrow to catch the real pattern.
• Monitoring window — Immediate.
• Rollback trigger — The rule is accidentally catching legitimate pages (finite pagination, real item pages) — narrow the pattern before it stays live long enough to hurt those pages’ crawling too.

Test 2 — Crawl requests to the pattern actually drop in your logs

• Test to run — Re-run the same Log File Analyzer pattern check from the How to Measure tab, a week or two after the fix ships.
• Expected result — The junk-pattern share of total crawl requests falls toward zero (robots.txt-blocked patterns won’t hit zero instantly — Google keeps occasionally re-checking a disallowed path for a while).
• Failure interpretation — No drop usually means either the rule didn’t actually match the real URLs being requested, or the URLs are still being discovered through a link path you haven’t fixed (an internal link, an old sitemap entry).
• Monitoring window — One to a few weeks of log data before you trust the trend over day-to-day noise.
• Rollback trigger — Don’t loosen or remove the block just because the drop is slow — confirm first whether it’s simply not-yet-recrawled versus not-working.

Test 3 — GSC Crawl Stats and “not indexed” counts move the same direction

• Test to run — Check Crawl Stats (by-URL-path breakdown, if you can infer it) and the Pages report’s “Discovered/Crawled – currently not indexed” counts.
• Expected result — Total crawl requests to the affected path class trend down, and (for noindex fixes) the “not indexed” count for that pattern eventually falls as pages get deindexed.
• Failure interpretation — Crawl Stats lags server logs by design — don’t panic if it hasn’t moved after only a few days. A stat that keeps climbing after a month suggests the fix isn’t reaching the real URL space.
• Monitoring window — Several weeks; GSC aggregates are slower to reflect changes than raw logs.
• Rollback trigger — A sustained climb with no plateau after a month is a sign to re-audit the pattern match, not to add a second, broader fix on top before confirming the first one is even scoped correctly.

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — the core piece: what wastes budget (parameters, faceted nav, sorting, session IDs), and why removing duplicates lets Google focus on unique URLs. This is the home for the trap-as-crawl-budget problem.
• Robots.txt and SEO: Everything You Need to Know — how to block trap patterns correctly (and why robots.txt isn’t a deindexing tool).
• Indexed, though blocked by robots.txt — the sequencing trap: block too early and your noindex never gets seen.
• SEOs Are Breaking Pagination After Google Changed Rel=Prev/Next — how over-correcting on pagination creates new crawl/orphaning problems; distinguish finite pagination from a real loop.
• The Beginner’s Guide to Technical SEO — where crawl efficiency fits in the bigger picture.

(I tend to write about this under crawl budget, robots.txt, and pagination rather than the literal “spider trap” label — so this page is where I pull the term and those pieces together.)

My speaking

• A Crash Course in Technical SEO (SlideShare) — covers parameters, faceted navigation, session IDs, and trailing slashes as duplicate-content sources.
• How Search Works (SlideShare) — how Googlebot discovers URLs and how crawl scheduling works; useful context for why a finite crawl pool gets drained by infinite URL spaces. (It doesn’t cover spider traps directly — cite it for crawl-discovery/budget context.)

From others

• Crawler Traps: Causes, Solutions & Prevention (Hamlet Batista, SEJ) — the most exhaustive type list, with developer-level fixes.
• Crawler Traps: How to Identify and Avoid Them (Conductor) — clean type list plus log/operator/GSC detection.
• Spider trap(s): sniffs and solutions (Joost de Valk, Yoast) — great plain definition and the log-file-detection angle.
• Field Guide to Spider Traps (Matthew Henry, Portent) — memorable framing and per-trap detection steps.
• Google’s Crawling December series — the best concentrated set of official crawl explainers, including faceted navigation.
• How to Identify, Analyze, and Fix Infinite Spaces (Glenn Gabe, GSQI) — a real enterprise case study showing the GSC + crawl + analytics triangulation workflow and the correct deindex-before-disallow sequence.
• Spider trap (Wikipedia) — authoritative definition and origin of the term, with the classic countermeasures.
• Google: 75% of crawling issues come from two common URL mistakes (Search Engine Land) — the write-up of Gary Illyes’ on-the-record admission that faceted navigation + action parameters account for ~75% of Google’s crawling problems.

Test yourself: Spider traps

Five quick questions on spider traps (crawler traps) and how to fix them. Pick an answer for each, then check.

Spider trap

A spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.

Related: Crawling, Crawl Budget

Spider trap

A spider trap — synonymous with crawler trap — is any part of a site that spawns a near-infinite stream of URLs, causing a crawler to keep requesting low-value, near-duplicate, or empty pages and waste crawl budget instead of reaching the content that matters. Google calls the underlying phenomenon “infinite spaces” or “infinite URL spaces.”

A trap doesn’t have to be truly infinite to be a problem — “de facto infinite” is enough. A handful of faceted filters that can be combined in any order, for example, can produce trillions of distinct URLs for the same few thousand products. Because a crawler can’t tell whether a newly discovered URL is useful without fetching it first, it ends up crawling huge swaths of that junk space before it figures out there’s nothing new there.

The classic sources are faceted navigation / filter combinations (the single biggest one), infinite calendars with endless “next month” links, session IDs in URLs, sort and tracking parameters, infinite scroll or looping pagination, redirect loops, and relative-link path explosion. The cost is opportunity cost: time a crawler spends on useless URLs is time it doesn’t spend on your new, useful ones. It mostly bites large, new, or ecommerce sites — small sites rarely need to worry. The fastest way to spot one is in your server logs, where a single junk pattern eats a disproportionate, lopsided share of crawl hits relative to your own site’s normal baseline — there’s no fixed universal percentage that defines a trap.

Related: Crawling, Crawl Budget

Changelog