Crawling

TL;DR — Crawling is how search engines find and download your pages. A bot (Googlebot for Google, Bingbot for Bing) follows links and reads sitemaps to discover URLs, then fetches them. A page has to be crawled before it can show up in search — but being crawled a lot doesn’t make you rank higher.

What crawling is

Search engines don’t browse the web the way you do. They send out automated programs — called crawlers, bots, or spiders — that visit pages, download what’s on them, and follow the links to find more pages. Google’s crawler is Googlebot; Bing’s is Bingbot.

Think of it as three steps, in order:

1. Crawl — the bot discovers a URL and downloads the page.
2. Index — the engine processes that page and files it away in a giant database of everything it might show in results.
3. Serve (rank) — when someone searches, the engine pulls the best matches from the index and puts them in order.

Crawling is step one. If a page never gets crawled, it can’t be indexed, and if it’s not indexed, it can’t rank. So crawling matters — but it’s a gate, not a scoreboard.

How crawlers find your pages

Two main ways:

• Following links. When a bot crawls a page, it grabs the links on it and adds those URLs to its list to crawl next. Good internal linking is how new pages get found.
• Sitemaps. An XML sitemap is a list of your URLs you hand directly to search engines so they don’t have to discover everything through links.

There are also “push” options where you actively tell an engine a page changed — more on those below — but links and sitemaps do most of the work.

How to help search engines crawl your site

• Link to your important pages from other pages (especially your homepage and main navigation).
• Submit an XML sitemap in Google Search Console and Bing Webmaster Tools.
• Don’t accidentally block pages you want found (check your robots.txt).
• Keep your server fast and healthy — if it’s slow or throwing errors, bots back off and crawl less.

The thing most people get wrong

Crawling is not ranking. Getting crawled more often won’t move you up the results. And blocking a page in robots.txt doesn’t remove it from Google — it just stops Google from reading it. Evidence for this claim robots.txt controls crawler access and is not a reliable way to keep a URL out of Google. Scope: Google Search behavior for URLs blocked by robots.txt; blocked URLs may still be indexed when discovered elsewhere. Confidence: high · Verified: 2026-07-14 Google Search Central: Introduction to robots.txt If you actually want a page gone, you let it be crawled and add a noindex tag. (I tested the blocking side of this myself — see the Advanced version.)

Want the deeper version, with how the crawl scheduler works, byte limits, and the crawl-vs-index-vs-rank distinction? Switch to the Advanced tab.

TL;DR — Crawling is the first of search’s three stages (crawl → index → serve). Bots discover URLs by pull (links + sitemaps) and push (IndexNow, Indexing API), then fetch them on an algorithmic schedule that throttles to your server’s health. Rendering JavaScript is a separate step. Crawling is required to rank but is not itself a ranking signal, and it’s distinct from indexing — a robots-blocked page can still be indexed. Most sites don’t need to manage crawl budget; logs are how you see what actually happened.

Crawling is stage one of three

Google is blunt about the pipeline: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexing, and serving. Evidence for this claim Google describes Search as three stages: crawling, indexing, and serving results. Scope: Google Search's documented processing model; it does not guarantee that a page reaches every stage. Confidence: high · Verified: 2026-07-14 Google Search Central: In-depth guide to how Google Search works The “not all pages make it” part is the whole game in technical SEO. A page can be crawled but not indexed, or indexed but never served for a query. Keeping the stages separate in your head is the single most useful mental model here.

Who actually does the crawling

“Googlebot” sounds like one program. It isn’t. In my How Search Works deck I describe it as 1,000+ systems running a family of specialized crawlers — desktop, mobile, image, news, video, ads — with the requests mostly originating out of Mountain View. They all draw from the same crawl budget pool, which is why a runaway image or parameter crawl can starve the crawling of your actual content.

And it’s not just search engines anymore. In my analysis of Cloudflare Radar data (Meet the New Web Crawlers), search-engine bots still crawl the most — but AI bots are firmly in second place and on track to overtake them in the next couple of years. If you’re reading your logs, the cast of characters has changed.

How crawlers discover URLs

Discovery is both pull and push:

• Pull — links. Google: “Other pages are discovered when Google extracts a link from a known page to a new page.” This is why orphan pages (nothing links to them) struggle to get found.
• Pull — sitemaps. “Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl.”
• Push — change notifications. Instead of waiting to be re-crawled, you tell the engine something changed: IndexNow (Bing, Yandex, and others — Google does not use it for general pages) and Google’s Indexing API (officially only for JobPosting and BroadcastEvent pages). Sitemaps’ lastmod, RSS, and WebSub round out the push options.

Google calls the whole thing “URL discovery.”

How crawlers fetch pages

• The schedule is algorithmic. “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” You influence it; you don’t set it.
• Crawling is polite. Bots throttle themselves so they don’t knock your site over: “they try not to crawl the site too fast to avoid overloading it… HTTP 500 errors mean ‘slow down.’” This is the lever behind temporarily slowing a crawl — return 503/429 and Googlebot eases off (for a day or two, not forever).
• There’s a byte limit. As of Google’s March 2026 Inside Googlebot update, Googlebot fetches up to roughly 2 MB per URL (down from the old 15 MB figure), with PDFs allowed up to 64 MB. Go over and the fetch is truncated, not rejected — only the downloaded portion is passed on for indexing. If your critical content sits below 2 MB of bloat, that matters.
• Caching cuts re-crawl cost. Google’s crawlers use conditional requests and caching so unchanged resources aren’t re-fetched every time. Google doesn’t publish an exact cache duration for rendering resources, so don’t assume a fixed refresh window for a code change — expect some lag, not an instant reflect.

Rendering is not crawling

This trips people up constantly. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Rendering is a distinct step from fetching the HTML. The renderer (the Web Rendering Service) is stateless — storage and cookies are cleared between loads, it denies permission prompts, and it rejects service workers. If your content only appears after a click or a JS-driven navigation that isn’t a real <a href> link, expect discovery and rendering problems. (Full treatment in JavaScript SEO.)

Crawling vs indexing vs ranking

The most important distinctions on this page:

• Crawling ≠ ranking. Crawling is necessary to be in results, but it is not a ranking signal. A higher crawl rate won’t lift your positions. Crawl budget is an efficiency concern, full stop.
• Crawling ≠ indexing. A page you block in robots.txt can still be indexed if other pages link to it — Google just can’t see the content, or any noindex tag you put there. As I put it in Indexed, though blocked by robots.txt: “crawling and indexing are two different things.” To actually remove a page, allow crawling and add noindex — don’t block it.

I tested the blocking side directly. In The Story of Blocking 2 High-Ranking Pages With Robots.txt I blocked two of our ranking pages. The result: “We lost a position here or there and all of the featured snippets for the pages… I expected a lot more impact, but the world didn’t end.” My takeaway stands: “Don’t block pages you want indexed. It hurts. Not as bad as you might think it does — but it still hurts.”

How to control crawling

• robots.txt controls crawling, not indexing. “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” Evidence for this claim robots.txt controls crawler access and is not a reliable way to keep a URL out of Google. Scope: Google Search behavior for URLs blocked by robots.txt; blocked URLs may still be indexed when discovered elsewhere. Confidence: high · Verified: 2026-07-14 Google Search Central: Introduction to robots.txt Use it to keep bots out of low-value spaces — not as a deindexing tool.
• Site architecture & internal links govern how deep pages sit and whether they’re discovered at all (see crawl depth).
• Crawl-rate controls. Google retired its manual crawl-rate slider in Search Console (early 2024) — it now leans on your server’s responses and a lower automatic minimum. Bing still gives you a manual Crawl Control grid in Bing Webmaster Tools (see crawl rate).

Crawl efficiency and crawl budget

Both engines think about crawling as efficiency. Gary Illyes frames Google’s side as crawl rate limit + crawl demand (popularity and staleness). Bing’s Fabrice Canel calls it their “crawl efficiency north star … to crawl a URL only when the content has been added … updated.”

The reassuring part: most sites don’t need to worry about it. Google says it plainly — if your site doesn’t have a large number of rapidly changing pages, or your pages are crawled the same day they’re published, “you don’t need to read this guide.” It starts to matter around 1M+ pages changing weekly or 10k+ changing daily (see crawl budget and crawl frequency).

When crawling goes wrong — how to see it

Before you reach for a fix, figure out which stage is actually broken — the symptoms and the remedies are different at each one:

• Not discovered at all? Nothing links to the page and it’s missing from your sitemap. Fix internal linking and sitemap coverage — see crawl depth and crawl frequency.

• Discovered but never fetched? Server errors, timeouts, or a robots.txt block are turning bots away. Check GSC Crawl Stats and your logs — see crawl budget and crawl rate.

• Fetched but rendering wrong? Content depends on a click or JS-only navigation Googlebot won’t perform, or WRS times out. See JavaScript SEO for the render-specific failure modes.

• Fetched and rendered but still not indexed? That’s a separate decision the index makes — see the Indexing hub rather than treating it as a crawl problem.

• Log file analysis is the ground truth. Your server logs show exactly which URLs bots actually hit, how often, and what status codes they got — the best way to spot wasted crawling and find pages bots never reach (see log file analysis).

• Spider traps — infinite URL spaces from calendars, faceted navigation, session IDs, or relative-link explosions — quietly burn your crawl budget on junk URLs (see spider traps).

Where to go next: the crawling cluster

This hub is the map. Each topic below is its own deep dive:

Crawl efficiency — how much and how often

• Crawl budget — what it is (capacity + demand), what wastes it, and who actually needs to care.
• Crawl rate — how fast bots fetch, why the GSC rate slider went away, and how to speed up or slow down a crawl today.
• Crawl frequency — what makes Google re-crawl a page sooner (popularity, staleness, accurate lastmod) and what doesn’t.
• Crawl depth — click depth vs. crawl traversal depth, and why important pages belong close to the homepage.

Meet the crawlers — who’s actually fetching your pages

• Crawler — what a web crawler (bot, spider) is and the fetch → parse → follow loop they all run.
• User agent — the user-agent string and robots.txt token a bot identifies itself with, and why you can’t trust the string alone.
• Googlebot — Google’s crawler: Smartphone vs. Desktop, rendering, and how to verify it.
• Bingbot — Microsoft’s crawler, how it differs, and the surfaces beyond Bing it feeds.
• AI crawlers — the bots from AI companies (training vs. AI-search vs. user-triggered fetchers) and how to control them.

Diagnosing crawl problems

• Log file analysis — verifying real bots and reading what they crawled.
• Spider traps (crawler traps) — the patterns that generate infinite URLs, and how to fix them.

Every topic above is its own deep dive nested under this hub — they’re in the sidebar too.

Qualifying your outbound links is a related on-page control that overlaps with crawling — these tell Google how to treat a link rather than whether to crawl it: nofollow (the original catch-all, now a hint), plus rel=sponsored and rel=ugc for paid/advertising and user-generated links. All three live in the on-page meta tags cluster.

Getting found in the first place is a related but separate stage. How search engines discover your URLs — internal links, sitemaps (XML, sitemap index, image, and video), and the push protocols IndexNow and the Google Indexing API (what each is actually for, and why Google doesn’t use IndexNow) — now lives in its own Discovery hub. For the broader topic, see How Search Works.

AI summary

A condensed take on the Advanced version:

• Crawling = stage one of search (crawl → index → serve). It’s required to rank but is not a ranking signal, and it’s distinct from indexing and from rendering.
• Discovery is pull + push: links and sitemaps (pull); IndexNow and the Indexing API (push). Google calls it “URL discovery.”
• Fetching is algorithmic and polite: Google decides what/how often/how many, and throttles on 5xx/429 (“slow down”). Googlebot fetches ~2 MB per URL (PDFs up to 64 MB) as of 2026, truncating beyond that.
• Rendering is separate: JS runs in a stateless headless Chrome; Google caches resources to cut re-crawl cost but doesn’t publish an exact duration, so changes can lag.
• Crawl ≠ index: a robots-blocked page can still be indexed via links; use noindex (with crawling allowed) to remove a page. Patrick’s blocking experiment showed blocked pages largely kept ranking — “but it still hurts.”
• Crawl budget = capacity + demand (popularity + staleness). Most sites don’t need to manage it.
• Diagnose with logs; watch for spider traps wasting budget. AI bots are now a major and growing share of crawl traffic.

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve overview, URL discovery, and the crawl scheduler.
• Crawling and Indexing — the hub for robots, sitemaps, canonicalization, and crawl controls.
• Introduction to robots.txt — what robots.txt does (and doesn’t) do.
• Optimize your crawl budget — crawl capacity + demand; who needs it.
• Overview of Google crawlers and fetchers — every Google user-agent and the published IP ranges.
• Inside Googlebot (March 2026) — the current byte limits and crawling architecture.
• Googlebot and the 15 MB thing (2022) — the older limit, useful for showing the change.
• Crawling December series (2024) — Googlebot, HTTP caching, faceted nav, and CDNs.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawling and its “crawl efficiency north star.”
• Bing Webmaster Tools — Crawl Control — set Bingbot’s speed and timing.
• IndexNow / indexnow.org — the push protocol for instantly signaling changed URLs.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — how crawling works

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “Other pages are discovered when Google extracts a link from a known page to a new page… Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl. This process is called ‘URL discovery’.” Jump to quote
• “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “They try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean ‘slow down’).” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote

Google — robots.txt, crawl budget

• “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” — Google Search Central docs. Jump to quote
• “Taking crawl capacity and crawl demand together, Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl.” Jump to quote
• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Jump to quote

Gary Illyes, Google (via Search Engine Land’s verbatim reproduction of Google’s 2017 crawl-budget post)

• Crawl rate limit is “the number of simultaneous parallel connections Googlebot may use to crawl the site, as well as the time it has to wait between the fetches.” Crawl demand is driven by popularity and staleness. Read the coverage

Fabrice Canel, Microsoft Bing

• “Crawling is the process by which bingbot discovers new and updated documents or content to be added to Bing’s searchable index.” Jump to quote
• “Our crawl efficiency north star is to crawl a URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links).” Jump to quote

Crawl-health checklist

A quick pass to confirm search engines can find and fetch what matters:

• Important pages are linked from somewhere crawlable (no orphans).
• robots.txt doesn’t block anything you want indexed (and does block low-value spaces like internal search results).
• XML sitemap is submitted in Google Search Console and Bing Webmaster Tools, lists only canonical, indexable URLs, and has an accurate lastmod.
• Server returns fast, stable responses — minimal 5xx/timeouts (bots slow down when your server struggles).
• You’re not using robots.txt to try to deindex — that’s noindex’s job (with crawling allowed).
• No spider traps generating infinite URLs (calendars, facets, session IDs).
• GSC Crawl Stats reviewed for response-code spikes and average response time.
• Server logs checked for crawl waste and uncrawled important URLs.
• JS-dependent content is reachable via real <a href> links, not click-only navigation.

The mental models

1. The pipeline — crawl → render → index → serve. Each stage is a filter, and “not all pages make it through each stage.” When a page isn’t performing, locate which stage it’s failing at before you change anything: Was it crawled? Rendered? Indexed? Served for the query?

2. The three “not equals.” Keep these separate and most crawl confusion disappears:

• Crawling ≠ indexing (blocked pages can still be indexed via links)
• Crawling ≠ ranking (crawl rate isn’t a ranking signal)
• Crawling ≠ rendering (JS runs in a separate, cache-able step)

3. Discovery = pull + push. Pull: links + sitemaps. Push: IndexNow / Indexing API / lastmod. If a page isn’t being found, ask which channel should be carrying it — and whether anything links to it at all.

4. Crawl budget = capacity + demand. Capacity is what your server can take; demand is popularity + staleness. You raise effective budget by removing waste (traps, duplicates, junk parameters) far more than by trying to make Google crawl “more.”

5. The decision rule for removing a page. Want it gone from search? Allow crawling + noindex. Want bots to skip a space entirely (and don’t care about indexing)? robots.txt disallow. Never use disallow to deindex.

Controlling crawling — cheat sheet

What each control actually does

Status codes bots care about

• 200 — fetched fine.
• 301/308 — permanent redirect (consolidates).
• 404/410 — gone; drops from index over time (410 a touch faster).
• 429 / 500 / 503 — “slow down” / try later; sustained = crawl drops.

Fast facts

• Googlebot fetch limit: ~2 MB per URL (PDFs 64 MB), truncated beyond.
• Manual crawl-rate slider in GSC: removed (Jan 2024) — server signals now.
• Bing equivalent: Crawl Control grid in Bing Webmaster Tools.
• IndexNow: Bing/Yandex/others — not Google. Indexing API: Google, **JobPosting
  • BroadcastEvent only**.

Verify a bot is really Googlebot

Plenty of traffic claims to be Googlebot. Confirm it with a reverse + forward DNS check (Google publishes no shortcut — fake user-agents are common).

Or do the same check by hand:

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in googlebot.com or google.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t Googlebot. You can also match against Google’s published IP ranges (googlebot.json).

A safe robots.txt starting point

User-agent: *
Disallow: /search           # internal search result pages
Disallow: /*?*sort=         # sort-order parameter spaces
Allow: /

Sitemap: https://example.com/sitemap.xml

Remember: Disallow blocks crawling, not indexing — don’t use it to remove a page from results.

Tools for seeing and managing crawling

• Google Search Console — Crawl Stats report — Google’s own view of how it crawls you: requests over time, response codes, average response time, by file type and Googlebot type.
• Bing Webmaster Tools — crawl info, Crawl Control, and Site Scan.
• Server log file analysis — the ground truth. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysis.)
• Crawlers / site audits — Ahrefs Site Audit and Screaming Frog SEO Spider simulate a crawl, surface depth, redirect chains, blocked URLs, and trap-like patterns.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.
• URL Inspection (GSC) — check how a single URL was crawled, rendered, and indexed.

Resources worth your time

My related writing

• The Beginner’s Guide to Technical SEO — where crawling fits in the bigger picture.
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party crawl-vs-rank experiment.
• Indexed, though blocked by robots.txt — why blocked pages still get indexed.
• Robots.txt and SEO: Everything You Need to Know.
• JavaScript SEO Issues & Best Practices — the rendering side.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• r/TechSEO — the community for crawl/index debugging.
• Google’s Crawling December series (official, but the best concentrated set of crawl explainers).
• Google Explains Crawl Budget for Webmasters (Search Engine Land) — verbatim reproduction of Gary Illyes’ original crawl-budget post with the crawl rate limit + crawl demand definitions.
• Googlebot Crawl Budget Explained (Search Engine Journal) — plain-language breakdown of the crawl capacity and demand model.
• Google Explains Googlebot Byte Limits and Crawling Architecture (Search Engine Journal) — coverage of the March 2026 “Inside Googlebot” update; 2 MB per URL, 64 MB PDFs, truncation behavior.
• How Bingbot Works: Discovering, Crawling, Extracting, Indexing (Search Engine Journal) — Bing’s side of the crawl pipeline, with Fabrice Canel context.
• Googlebot File Size Limit (DebugBear) — explains what happens when a page exceeds Googlebot’s fetch limit (truncation, not rejection).

Podcasts

• Search Off the Record (Google Search Relations) — How Googlebot crawls the web. Gary Illyes and Martin Splitt on the past, present, and future of Googlebot: unified crawling infrastructure, HTTP/1.1 vs HTTP/2, conditional requests, and byte limits. Listen

Videos

• Google Search Central (YouTube) — the How Google Search Works series and Martin Splitt’s crawling/rendering explainers, including the JavaScript SEO videos. Channel

Stats worth citing

• AI bots are closing in on search bots. From my Cloudflare Radar analysis, search-engine crawlers still crawl the most, but AI bots are a clear #2 and on pace to overtake them within a couple of years. Source
• Tens of billions of normalized URLs never seen before are discovered by Bing every day — the scale of the discovery problem the engines aggressively filter (Fabrice Canel, Microsoft Bing, 2022). Coverage
• Which crawlers ~140M sites block the most — robots.txt block-rate data from my study with Xibeijia Guan; useful for understanding how the open web gates bots. Source
• Byte limit: ~2 MB per URL (PDFs 64 MB) — Google’s documented Googlebot fetch limit as of March 2026 (down from 15 MB). Source

Test yourself: Crawling

Five quick questions on how search engines discover and fetch pages. Pick an answer for each, then check.

Crawling

Crawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.

Related: Crawl Budget, Indexing

Crawling

Crawling is the first of the three stages of search — crawl → index → serve — and it’s a prerequisite for the other two, not a substitute for them. Search engines run automated programs (Googlebot, Bingbot, and a growing crowd of AI crawlers) that discover URLs and download their content so it can later be rendered, indexed, and ranked.

URLs are discovered two ways: pull (a crawler extracts links from pages it already knows, or reads your sitemap) and push (you ping an engine that something changed — IndexNow for Bing and others, or Google’s Indexing API for its narrow supported cases). What gets crawled, how often, and how many pages per visit is decided algorithmically and adjusts to your server’s health.

Three distinctions trip people up:

• Crawling ≠ indexing. A page blocked in robots.txt can still be indexed if other pages link to it — Google just can’t see what’s on it (or any noindex tag). To remove a page, allow crawling and use noindex.
• Crawling ≠ ranking. Crawling is necessary to appear in results, but a higher crawl rate doesn’t lift positions. Crawl budget is an efficiency concern, not a ranking signal.
• Crawling ≠ rendering. Running a page’s JavaScript happens in a separate rendering step, so code or content changes can lag.

Most sites never need to think about crawl budget — it matters mainly for very large or rapidly changing sites. When crawling does go wrong, the ground truth lives in your server logs.

Related: Crawl Budget, Indexing

Changelog