Web Crawlers

TL;DR — A web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is an automated program that visits web pages, downloads them, and follows the links to find more pages. “Crawler,” “spider,” and “bot” all mean the same thing. Search engines use crawlers to find your pages so they can show up in search — but a page has to be crawled before it can be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and ranked, and being crawled more often doesn’t make you rank higher.

What a crawler is

A web crawler is a piece of software that browses the web automatically. It visits a page, downloads what’s on it, grabs the links, and then goes off to visit those links too — over and over, across billions of pages.

You’ll hear three names for the exact same thing: crawler, spider, and bot. They’re interchangeable. Google’s crawler is called GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.; Bing’s is BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.. Evidence for this claim Google defines a crawler as an automated program that discovers and scans websites and describes Googlebot as its web crawler. Scope: Google crawler terminology. Confidence: high · Verified: 2026-07-14 Google: Googlebot overview

Search engines need crawlers because they can’t show you a page in results unless they’ve foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. and downloaded it first. The crawler builds the raw material; the search engine’s index is the giant database it searches through when you type a query.

A crawler isn’t the same thing as a scraper. A crawler’s job is discovery at scale — following links across a wide swath of the web (or all of it). A scraper’s job is extraction — pulling specific pieces of data out of pages it’s already been pointed at. In practice the line blurs: plenty of tools do both, crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. to find pages and then scraping the data they need from each one.

How a crawler finds your pages

Mostly two ways:

• Following links. When a crawler downloads a page, it reads the links on it and adds those new URLs to its to-do list. Good internal linkingAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. is how new pages get discovered.
• SitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.. An XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. is a list of your URLs you hand straight to search engines, which can help them discover URLs. Evidence for this claim Google discovers URLs through links from known pages and through submitted sitemaps. Scope: Google URL discovery; sitemap submission does not guarantee crawling or indexing. Confidence: high · Verified: 2026-07-14 Google: How Search works

The three steps that matter

Search works in a fixed order:

1. Crawl — a bot finds a URL and downloads the page.
2. Index — the engine processes that page and files it in its database.
3. Serve (rank) — when someone searches, the engine pulls the best matches and puts them in order.

CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is step one. If a page never gets crawled, it can’t be indexed, and if it’s not indexed, it can’t rank. So crawling is a gate you have to get through — but it’s not a scoreboard. Getting crawled more won’t push you up the results.

The thing people get wrong

Blocking a page in robots.txt doesn’t remove it from Google. robots.txt just tells the crawler “don’t read this.” The page can still end up in results if other sites link to it. If you actually want a page gone, you let it be crawled and add a noindex tag instead.

Want the mechanical version — the crawl queue, fetch and parse, why renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is a separate step, and how to tell a real crawler from a fake one? Switch to the Advanced tab.

TL;DR — A crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (spider, bot) is an automated program that discovers, fetches, parses, and re-schedules URLs in a loop. Google’s own words: it “downloads text, images, and videos from pages it foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. on the internet with automated programs called crawlers.” The mechanics are a frontier (the queue of URLs), a fetch (download the HTML), a parse (extract links + content), and a scheduler that decides what to fetch next and how fast — throttling on your server’s health. RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. JavaScript is a separate step. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is required to rank but isn’t a ranking signal, and it’s distinct from indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — a robots-blocked page can still be indexed. And in 2026 it’s not just search engines crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.: AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are a large, fast-growing share of the traffic.

What a crawler actually is

Google’s definition is about as plain as it gets: it “downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.” Google’s own glossary phrasing is even broader — “a crawler is a generic term for any program that is used to automatically discover and scan websites.” Crawler, spider, bot, spiderbot: same concept, different name. Evidence for this claim Google defines a crawler as an automated program that discovers and scans websites and describes Googlebot as its web crawler. Scope: Google crawler terminology. Confidence: high · Verified: 2026-07-14 Google: Googlebot overview

The reason crawling matters at all is the pipeline. Google is blunt about it: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and serving. Crawling is gate one. A page that’s never crawled can’t be indexed, and a page that’s not indexed can’t rank.

How a crawler works, mechanically

Strip away the branding and most link-following crawlers run some version of the same loop: discover → fetch → parse → schedule → repeat. It’s a useful mental model for reasoning about crawler behavior — not a guarantee that every crawler ever built implements these exact stages identically.

The frontier (the queue). A crawler doesn’t wander randomly. It keeps a queue of URLs it knows about but hasn’t visited yet — the crawl frontier — and a scheduler that decides which URL to pull off next. It starts from a seed set of known URLs and grows the queue as it discovers more. Google describes discovery like this: “Other pages are discovered when Google extracts a link from a known page to a new page… Still other pages are discovered when you submit a list of pages (a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.) for Google to crawl. This process is called ‘URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched..’” Evidence for this claim Google discovers URLs through links from known pages and through submitted sitemaps. Scope: Google URL discovery; sitemap submission does not guarantee crawling or indexing. Confidence: high · Verified: 2026-07-14 Google: How Search works

Fetch. The crawler sends an HTTP request; the server returns the page. At this step it’s downloading the raw HTML (resources like JavaScript and CSS are fetched separately, often later). Bing’s Fabrice Canel frames the goal of this whole process cleanly: “Crawling is the process by which bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated documents or content to be added to Bing’s searchable index.”

Parse. The fetched HTML is parsed to pull out links, text, headings, images, and metadata. Newly found URLs are normalized and dropped back into the frontier — which is what keeps the loop going.

Schedule. What gets fetched, how often, and how many pages per visit is decided algorithmically, not by you. Google: “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Popular and frequently changing pages get revisited sooner; obscure, static ones get revisited rarely. (How that schedule is set — crawl budgetThe number of URLs an engine will crawl in a timeframe., crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor., crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. — each has its own deep dive in this cluster.)

”Googlebot” isn’t one program

This is the insight most explainers miss. “Googlebot” sounds like a single program; it isn’t. In my How Search Works deck I describe it as 1,000+ systems running a family of specialized crawlers — desktop, mobile, image, news, video, ads — all drawing from the same crawl budget pool. Gary Illyes has made the same point from the inside: the name is essentially a misnomer for a central crawling platform that many Google products (Shopping, AdSense, and the rest) route their requests through, under different user-agent names. So when you read your logs, you’re not watching one bot — you’re watching a fleet that happens to share a label. (The engine-specific details live in the Googlebot and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. siblings.)

Politeness — how crawlers avoid taking down your site

A well-behaved crawler deliberately throttles itself. Google: “They try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean ‘slow down’).” That’s also the lever behind temporarily slowing a crawl — sustained 5xx/429 responses make Googlebot back off (for a day or two, not forever). Politeness isn’t optional courtesy; at the scale crawlers operate, it’s the only thing keeping them from hammering servers off the web.

Robots.txt — the standard crawl control

robots.txt is the de facto access-control convention. Google: “A robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file tells search engine crawlers which URLs the crawler can access on your site.” Two things to keep straight:

• It controls crawling, not indexing. A page you disallow can still be indexed if other pages link to it — Google just can’t see the content (or any noindex tag you put there). As I put it in Indexed, though blocked by robots.txt: “crawling and indexing are two different things.” To actually remove a page, allow crawling and add noindex — don’t block it.
• Reputable crawlers obey it; bad actors don’t. Googlebot, Bingbot, Ahrefsbot and the like respect robots.txt; many scrapers and disreputable bots ignore it entirely. It’s a convention, not an enforcement mechanism.

I tested the blocking side directly. In The Story of Blocking 2 High-Ranking Pages With Robots.txt I blocked two of our ranking pages for nearly five months. They lost all their featured snippets and their custom titles (results showed “no information is available”), and clicks dropped more than position alone would explain — but the traffic estimate barely moved. My takeaway stands: don’t block pages you want indexed. It hurts. Not as bad as you might think it does — but it still hurts.

Crawl ≠ index ≠ render ≠ rank

The distinctions that cause the most confusion:

• Crawling ≠ indexing. Crawling is fetch + download; indexing is understand + store. A page can be crawled and not indexed (Google chose not to include it), and a blocked page can be indexed without ever being read. “Indexing isn’t guaranteed; not every page that Google processes will be indexed.”
• Crawling ≠ renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM.. Fetching the HTML and running its JavaScript are two different steps. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome,” but that rendering happens in a separate queue that can lag behind the initial fetch. If your content only appears after JavaScript runs, it isn’t available in the same pass as the HTML.
• Crawling ≠ ranking. Being crawled more often is not a ranking signal. Crawl rate is an efficiency concern, full stop — which is why most sites never need to manage crawl budget at all.

Types of crawlers (high level)

Keeping this brief — each has a sibling that goes deep:

• Search engine crawlers — Googlebot, Bingbot. They build the indexes that power search results.
• AI / LLM crawlers — GPTBot, ClaudeBot, CCBot, and others, gathering web content to train or power AI models. Growing fast (see below).
• SEO audit crawlers — Ahrefs Site Audit, Screaming Frog, and similar tools that simulate a crawl to surface technical issues.
• User-triggered fetchers — tools that make a single request on demand (Google’s URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version., Rich ResultsRich results (formerly 'rich snippets') are enhanced search listings — stars, images, prices, breadcrumbs, video thumbnails, and more — that Google and Bing build from structured data. They're a display feature, not a ranking factor, and eligibility never guarantees they'll show. Test). Not autonomous crawlers.

Who’s crawling the web now

It’s no longer just search engines. In my analysis of Cloudflare Radar data (Meet the New Web Crawlers), search-engine bots still crawl the most, but AI bots are firmly in second place. The scale here is hard to overstate: Bing alone discovers tens of billions of normalized URLs it’s never seen before every single day (Fabrice Canel) — which is exactly why engines have to aggressively prioritize what they actually fetch.

How to verify a crawler is who it claims to be

Any bot can put “Googlebot” in its user-agent string, so don’t trust the string alone. The real check is a reverse + forward DNS lookup (see the Scripts tab): reverse-DNS the IP from your logs, confirm it resolves to a googlebot.com / google.com hostname, then forward-DNS that hostname and confirm it points back to the same IP. Google also publishes its IP ranges. The full per-engine user-agent details live in the user-agent sibling.

Where to go next

This is the general-concept page. For specifics:

• Googlebot — Google’s crawler: user-agents, mobile-first crawling, rendering, Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. Crawl Stats.
• Bingbot — Bing’s crawler: Crawl Control and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it..
• AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. — GPTBot, ClaudeBot, CCBot, and the newer agentic bots.
• User-agent — what the user-agent string is and how to read it.
• Crawling hub — the whole pipeline, plus crawl budget, crawl rate, crawl frequency, crawl depthCrawl depth usually means click depth — how many clicks it takes to reach a page from the homepage by following internal links. It can also mean a crawler setting that limits how many levels deep a crawl goes before it stops., spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content., and log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..

AI summary

A condensed take on the Advanced version:

• A crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. = a spider = a bot. All three names mean one thing: an automated program that “automatically discover[s] and scan[s] websites,” downloading pages so search engines can indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and rank them.
• The mechanical loop is discover → fetch → parse → schedule → repeat. A frontier (queue of known-but-unvisited URLs) feeds a scheduler; each fetch downloads HTML, parsing extracts links + content, and new URLs go back in the queue. Google calls discovery “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.” (links + sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.).
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is stage one of three (crawl → index → serve), and “not all pages make it through each stage.” It’s required to rank but is not a ranking signal.
• Crawl ≠ index ≠ render ≠ rank. IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. isn’t guaranteed; renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. JavaScript is a separate, later step; a robots-blocked page can still be indexed via links.
• “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” is a fleet, not one program — a central crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. platform many Google products route through under different user-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target..
• Reputable crawlers are polite and obey robots.txt (throttling on 5xx/429 — “slow down”); bad actors and many scrapers don’t.
• It’s not just search engines anymore. AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are a clear #2 by volume and closing on search bots; Bing alone sees tens of billions of new URLs daily.
• Verify a crawler with reverse + forward DNS, not the user-agent string.

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve overview, URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched., and the crawl scheduler.
• Overview of Google crawlers and fetchers — the definition of a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., every Google user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., and the published IP ranges.
• Googlebot — the two GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. variants (Smartphone and Desktop) and their technical behavior.
• Introduction to robots.txt — what robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. does (and doesn’t) do.
• Optimize your crawl budget — crawl capacityThe number of URLs an engine will crawl in a timeframe. + demand, and who actually needs it.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and its “crawl efficiency north star.”

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is and how it works

• “Google downloads text, images, and videos from pages it foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. on the internet with automated programs called crawlers.” — Google Search Central docs. Jump to quote
• “Google Search works in three stages, and not all pages make it through each stage.” Jump to quote
• “Other pages are discovered when Google extracts a link from a known page to a new page… Still other pages are discovered when you submit a list of pages (a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.) for Google to crawl. This process is called ‘URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched.’.” Jump to quote
• “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “They try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500A 500 Internal Server Error is a generic HTTP status code — RFC 9110 defines it as an unexpected condition that stopped the server from fulfilling the request, and nothing more. The request may have been fine — something broke server-side while generating the response. For SEO, an isolated 500 is typically retried, but persistent, site-wide 500s get Google's documented response: slower crawling and, if the errors don't clear, eventual removal from the index. errors mean ‘slow down’).” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote

Google — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.

• “A robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file tells search engine crawlers which URLs the crawler can access on your site.” — Google Search Central docs. Jump to quote

Fabrice Canel, Microsoft Bing

• “CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is the process by which bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated documents or content to be added to Bing’s searchable index.” Jump to quote
• “Our crawl efficiency north star is to crawl an URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links).” Jump to quote

Make sure crawlers can do their job — checklist

A quick pass to confirm bots can find, fetch, and read what matters:

• Important pages are linked from somewhere crawlable (no orphans) — links are how crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. discover URLs.
• An XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. is submitted so discovery doesn’t rely on links alone.
• robots.txt doesn’t block anything you want indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (and does block low-value spaces you never want crawled).
• You’re not using robots.txt to try to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. — that’s noindex’s job (with crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. allowed).
• Server returns fast, stable responses — bots throttle down on 5xx/429 (“slow down”).
• JS-dependent content is reachable via real <a href> links, not click-only navigation (renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is a separate, later step).
• Bots in your logs are verified as genuine (reverse + forward DNS), not just trusted by their user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string.

The mental models

1. The loop — discover → fetch → parse → schedule → repeat. Most link-following crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. run some version of this loop. A frontier holds URLs it knows about; a scheduler picks the next one; a fetch downloads it; a parse extracts links that feed back into the frontier. If a page isn’t getting crawled, ask which step is failing: was it ever discovered (in the frontier at all)? Is it being de-prioritized by the scheduler?

2. The pipeline — crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve. Each stage is a filter, and “not all pages make it through each stage.” When a page underperforms, find which stage it’s failing at before changing anything.

3. The “not equals” set. Keep these four separate and most crawler confusion disappears:

• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (a blocked page can still be indexed via links)
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. (JavaScript runs in a separate, later step)
• Crawling ≠ ranking (crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is not a ranking signal)
• “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” ≠ one program (it’s a fleet routed through one platform)

4. The decision rule for removing a page. Want it gone from search? Allow crawling + noindex. Want bots to skip a space entirely (and don’t care about indexing)? robots.txt disallow. Never use disallow to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..

Crawler — cheat sheet

Three names, one thing: crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. = spider = bot (spiderbot). Interchangeable.

The loop: discover (frontier) → fetch (download HTML) → parse (extract links + content) → schedule (decide what’s next) → repeat.

The pipeline: crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is gate one; “not all pages make it through each stage.”

The “not equals”:

Politeness: crawlers throttle themselves; sustained 5xx/429 = “slow down.”

Control: robots.txt controls crawling, not indexing — don’t use it to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.. Reputable crawlers obey it; many scrapers don’t.

Verify a crawler: reverse + forward DNS, never the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string alone.

The landscape (2026): search-engine bots crawl the most, AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are a fast- rising #2, and Bing alone discovers tens of billions of new URLs per day.

Verify a bot is really the crawler it claims to be

A user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string is trivial to fake, so the only reliable check is DNS. For GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., confirm a reverse + forward DNS lookup resolves to a Google domain and back to the same IP.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in googlebot.com or google.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t really GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. You can also match the IP against Google’s published ranges (googlebot.json). The same reverse-then-forward pattern works for verifying other reputable crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. against their own published hostnames.

Tools for inspecting crawler access and activity

• Robots.txt Tester — test whether a named crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is allowed to request a URL before treating a crawl gap as a scheduling problem.
• Log File Analyzer — see which crawlers actually hit the server, which URLs they request, and the status codes they receive.
• HTTP Header Checker — inspect the response a crawler gets, including redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. and crawler directives sent in headers.
• Render Gap Analyzer — compare raw and rendered HTML when a crawler can fetch a page but important content depends on client-side renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM..

Test yourself: web crawlers

Resources worth your time

My related writing

• What Is Googlebot & How Does It Work? — the deep dive on Google’s specific crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fleet.
• Indexed, though blocked by robots.txt — why blocked pages still get indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (crawl ≠ index).
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party crawl-vs-rank experiment.
• When Should You Worry About Crawl Budget? — when crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. efficiency actually matters.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — the changing cast of crawlers.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Search Off the Record — “What is a web crawler, really?” — Gary Illyes and Lizzi Sassman on what a crawler is and the “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is a misnomer” point.
• Search Off the Record — “Crawling smarter, not harder” (Ep. 79) — Gary Illyes and John Mueller on crawl budgetThe number of URLs an engine will crawl in a timeframe., the scheduler, and why 90%+ of sites don’t need to worry about it.
• Google Search Central Blog — Crawling December resources (2024) — Google’s own curated deep-dive series on crawling fundamentals.
• Search Engine Journal — Google’s Crawling Priorities: Insights From Gary Illyes — covers Illyes’s stated mission to decrease crawling without sacrificing quality.
• Search Engine Land — Google explains what “crawl budget” means for webmasters — the original 2017 coverage when Google first defined crawl capacity and crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side..
• Wikipedia — Web crawler — good for the formal CS terminology (frontier, politeness policy) and history.
• r/TechSEO — the community for crawl/index debugging.

Stats worth citing

• AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are closing in on search bots. From my Cloudflare Radar analysis, search-engine crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. still crawl the most, but AI bots are a clear #2 by volume. Source
• Tens of billions of normalized URLs never seen before are discovered by Bing every day — the scale of the discovery problem the engines aggressively filter (Fabrice Canel, Microsoft Bing, 2022). Coverage

Crawler

A crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.

Related: Googlebot, Bingbot, User Agent, Crawling, Robots.txt

Crawler

A web crawler — the words spider and bot mean the same thing — is an automated software program that systematically browses the web. It fetches a page, parses it to pull out links and content, adds the newly foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. URLs to a queue, and repeats. Search engines run crawlers to discover and download pages so they can later be rendered, indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranked.

The crawl is stage one of three: crawl → index → serve. It’s a prerequisite for the other two, not a substitute for them. A page has to be crawlable to be indexed, but crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. on its own isn’t a ranking factor — getting crawled more often won’t lift your positions.

“GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” is the best-known crawler, but it’s really a label on a fleet of specialized crawlers (desktop, mobile, image, news, video, ads) routed through one central system, not a single program. Bing’s is BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., and AI companies now run a fast-growing crowd of their own crawlers. Reputable crawlers identify themselves with a user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string, respect robots.txt, and throttle themselves so they don’t overload your server.

Related: Googlebot, Bingbot, User Agent, Crawling, Robots.txt

Changelog