Web Crawlers

TL;DR — A web crawler is an automated program that visits web pages, downloads them, and follows the links to find more pages. “Crawler,” “spider,” and “bot” all mean the same thing. Search engines use crawlers to find your pages so they can show up in search — but a page has to be crawled before it can be indexed and ranked, and being crawled more often doesn’t make you rank higher.

What a crawler is

A web crawler is a piece of software that browses the web automatically. It visits a page, downloads what’s on it, grabs the links, and then goes off to visit those links too — over and over, across billions of pages.

You’ll hear three names for the exact same thing: crawler, spider, and bot. They’re interchangeable. Google’s crawler is called Googlebot; Bing’s is Bingbot. Evidence for this claim Google defines a crawler as an automated program that discovers and scans websites and describes Googlebot as its web crawler. Scope: Google crawler terminology. Confidence: high · Verified: 2026-07-14 Google: Googlebot overview

Search engines need crawlers because they can’t show you a page in results unless they’ve found and downloaded it first. The crawler builds the raw material; the search engine’s index is the giant database it searches through when you type a query.

A crawler isn’t the same thing as a scraper. A crawler’s job is discovery at scale — following links across a wide swath of the web (or all of it). A scraper’s job is extraction — pulling specific pieces of data out of pages it’s already been pointed at. In practice the line blurs: plenty of tools do both, crawling to find pages and then scraping the data they need from each one.

How a crawler finds your pages

Mostly two ways:

• Following links. When a crawler downloads a page, it reads the links on it and adds those new URLs to its to-do list. Good internal linking is how new pages get discovered.
• Sitemaps. An XML sitemap is a list of your URLs you hand straight to search engines, which can help them discover URLs. Evidence for this claim Google discovers URLs through links from known pages and through submitted sitemaps. Scope: Google URL discovery; sitemap submission does not guarantee crawling or indexing. Confidence: high · Verified: 2026-07-14 Google: How Search works

The three steps that matter

Search works in a fixed order:

1. Crawl — a bot finds a URL and downloads the page.
2. Index — the engine processes that page and files it in its database.
3. Serve (rank) — when someone searches, the engine pulls the best matches and puts them in order.

Crawling is step one. If a page never gets crawled, it can’t be indexed, and if it’s not indexed, it can’t rank. So crawling is a gate you have to get through — but it’s not a scoreboard. Getting crawled more won’t push you up the results.

The thing people get wrong

Blocking a page in robots.txt doesn’t remove it from Google. robots.txt just tells the crawler “don’t read this.” The page can still end up in results if other sites link to it. If you actually want a page gone, you let it be crawled and add a noindex tag instead.

Want the mechanical version — the crawl queue, fetch and parse, why rendering is a separate step, and how to tell a real crawler from a fake one? Switch to the Advanced tab.

TL;DR — A crawler (spider, bot) is an automated program that discovers, fetches, parses, and re-schedules URLs in a loop. Google’s own words: it “downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.” The mechanics are a frontier (the queue of URLs), a fetch (download the HTML), a parse (extract links + content), and a scheduler that decides what to fetch next and how fast — throttling on your server’s health. Rendering JavaScript is a separate step. Crawling is required to rank but isn’t a ranking signal, and it’s distinct from indexing — a robots-blocked page can still be indexed. And in 2026 it’s not just search engines crawling: AI bots are a large, fast-growing share of the traffic.

What a crawler actually is

Google’s definition is about as plain as it gets: it “downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.” Google’s own glossary phrasing is even broader — “a crawler is a generic term for any program that is used to automatically discover and scan websites.” Crawler, spider, bot, spiderbot: same concept, different name. Evidence for this claim Google defines a crawler as an automated program that discovers and scans websites and describes Googlebot as its web crawler. Scope: Google crawler terminology. Confidence: high · Verified: 2026-07-14 Google: Googlebot overview

The reason crawling matters at all is the pipeline. Google is blunt about it: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexing, and serving. Crawling is gate one. A page that’s never crawled can’t be indexed, and a page that’s not indexed can’t rank.

How a crawler works, mechanically

Strip away the branding and most link-following crawlers run some version of the same loop: discover → fetch → parse → schedule → repeat. It’s a useful mental model for reasoning about crawler behavior — not a guarantee that every crawler ever built implements these exact stages identically.

The frontier (the queue). A crawler doesn’t wander randomly. It keeps a queue of URLs it knows about but hasn’t visited yet — the crawl frontier — and a scheduler that decides which URL to pull off next. It starts from a seed set of known URLs and grows the queue as it discovers more. Google describes discovery like this: “Other pages are discovered when Google extracts a link from a known page to a new page… Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl. This process is called ‘URL discovery.’” Evidence for this claim Google discovers URLs through links from known pages and through submitted sitemaps. Scope: Google URL discovery; sitemap submission does not guarantee crawling or indexing. Confidence: high · Verified: 2026-07-14 Google: How Search works

Fetch. The crawler sends an HTTP request; the server returns the page. At this step it’s downloading the raw HTML (resources like JavaScript and CSS are fetched separately, often later). Bing’s Fabrice Canel frames the goal of this whole process cleanly: “Crawling is the process by which bingbot discovers new and updated documents or content to be added to Bing’s searchable index.”

Parse. The fetched HTML is parsed to pull out links, text, headings, images, and metadata. Newly found URLs are normalized and dropped back into the frontier — which is what keeps the loop going.

Schedule. What gets fetched, how often, and how many pages per visit is decided algorithmically, not by you. Google: “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Popular and frequently changing pages get revisited sooner; obscure, static ones get revisited rarely. (How that schedule is set — crawl budget, crawl rate, crawl frequency — each has its own deep dive in this cluster.)

”Googlebot” isn’t one program

This is the insight most explainers miss. “Googlebot” sounds like a single program; it isn’t. In my How Search Works deck I describe it as 1,000+ systems running a family of specialized crawlers — desktop, mobile, image, news, video, ads — all drawing from the same crawl budget pool. Gary Illyes has made the same point from the inside: the name is essentially a misnomer for a central crawling platform that many Google products (Shopping, AdSense, and the rest) route their requests through, under different user-agent names. So when you read your logs, you’re not watching one bot — you’re watching a fleet that happens to share a label. (The engine-specific details live in the Googlebot and Bingbot siblings.)

Politeness — how crawlers avoid taking down your site

A well-behaved crawler deliberately throttles itself. Google: “They try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean ‘slow down’).” That’s also the lever behind temporarily slowing a crawl — sustained 5xx/429 responses make Googlebot back off (for a day or two, not forever). Politeness isn’t optional courtesy; at the scale crawlers operate, it’s the only thing keeping them from hammering servers off the web.

Robots.txt — the standard crawl control

robots.txt is the de facto access-control convention. Google: “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” Two things to keep straight:

• It controls crawling, not indexing. A page you disallow can still be indexed if other pages link to it — Google just can’t see the content (or any noindex tag you put there). As I put it in Indexed, though blocked by robots.txt: “crawling and indexing are two different things.” To actually remove a page, allow crawling and add noindex — don’t block it.
• Reputable crawlers obey it; bad actors don’t. Googlebot, Bingbot, Ahrefsbot and the like respect robots.txt; many scrapers and disreputable bots ignore it entirely. It’s a convention, not an enforcement mechanism.

I tested the blocking side directly. In The Story of Blocking 2 High-Ranking Pages With Robots.txt I blocked two of our ranking pages for nearly five months. They lost all their featured snippets and their custom titles (results showed “no information is available”), and clicks dropped more than position alone would explain — but the traffic estimate barely moved. My takeaway stands: don’t block pages you want indexed. It hurts. Not as bad as you might think it does — but it still hurts.

Crawl ≠ index ≠ render ≠ rank

The distinctions that cause the most confusion:

• Crawling ≠ indexing. Crawling is fetch + download; indexing is understand + store. A page can be crawled and not indexed (Google chose not to include it), and a blocked page can be indexed without ever being read. “Indexing isn’t guaranteed; not every page that Google processes will be indexed.”
• Crawling ≠ rendering. Fetching the HTML and running its JavaScript are two different steps. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome,” but that rendering happens in a separate queue that can lag behind the initial fetch. If your content only appears after JavaScript runs, it isn’t available in the same pass as the HTML.
• Crawling ≠ ranking. Being crawled more often is not a ranking signal. Crawl rate is an efficiency concern, full stop — which is why most sites never need to manage crawl budget at all.

Types of crawlers (high level)

Keeping this brief — each has a sibling that goes deep:

• Search engine crawlers — Googlebot, Bingbot. They build the indexes that power search results.
• AI / LLM crawlers — GPTBot, ClaudeBot, CCBot, and others, gathering web content to train or power AI models. Growing fast (see below).
• SEO audit crawlers — Ahrefs Site Audit, Screaming Frog, and similar tools that simulate a crawl to surface technical issues.
• User-triggered fetchers — tools that make a single request on demand (Google’s URL Inspection tool, Rich Results Test). Not autonomous crawlers.

Who’s crawling the web now

It’s no longer just search engines. In my analysis of Cloudflare Radar data (Meet the New Web Crawlers), search-engine bots still crawl the most, but AI bots are firmly in second place. The scale here is hard to overstate: Bing alone discovers tens of billions of normalized URLs it’s never seen before every single day (Fabrice Canel) — which is exactly why engines have to aggressively prioritize what they actually fetch.

How to verify a crawler is who it claims to be

Any bot can put “Googlebot” in its user-agent string, so don’t trust the string alone. The real check is a reverse + forward DNS lookup (see the Scripts tab): reverse-DNS the IP from your logs, confirm it resolves to a googlebot.com / google.com hostname, then forward-DNS that hostname and confirm it points back to the same IP. Google also publishes its IP ranges. The full per-engine user-agent details live in the user-agent sibling.

Where to go next

This is the general-concept page. For specifics:

• Googlebot — Google’s crawler: user-agents, mobile-first crawling, rendering, Search Console Crawl Stats.
• Bingbot — Bing’s crawler: Crawl Control and IndexNow.
• AI crawlers — GPTBot, ClaudeBot, CCBot, and the newer agentic bots.
• User-agent — what the user-agent string is and how to read it.
• Crawling hub — the whole pipeline, plus crawl budget, crawl rate, crawl frequency, crawl depth, spider traps, and log file analysis.

AI summary

A condensed take on the Advanced version:

• A crawler = a spider = a bot. All three names mean one thing: an automated program that “automatically discover[s] and scan[s] websites,” downloading pages so search engines can index and rank them.
• The mechanical loop is discover → fetch → parse → schedule → repeat. A frontier (queue of known-but-unvisited URLs) feeds a scheduler; each fetch downloads HTML, parsing extracts links + content, and new URLs go back in the queue. Google calls discovery “URL discovery” (links + sitemaps).
• Crawling is stage one of three (crawl → index → serve), and “not all pages make it through each stage.” It’s required to rank but is not a ranking signal.
• Crawl ≠ index ≠ render ≠ rank. Indexing isn’t guaranteed; rendering JavaScript is a separate, later step; a robots-blocked page can still be indexed via links.
• “Googlebot” is a fleet, not one program — a central crawling platform many Google products route through under different user-agents.
• Reputable crawlers are polite and obey robots.txt (throttling on 5xx/429 — “slow down”); bad actors and many scrapers don’t.
• It’s not just search engines anymore. AI bots are a clear #2 by volume and closing on search bots; Bing alone sees tens of billions of new URLs daily.
• Verify a crawler with reverse + forward DNS, not the user-agent string.

Official documentation

Primary-source documentation from the search engines.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve overview, URL discovery, and the crawl scheduler.
• Overview of Google crawlers and fetchers — the definition of a crawler, every Google user-agent, and the published IP ranges.
• Googlebot — the two Googlebot variants (Smartphone and Desktop) and their technical behavior.
• Introduction to robots.txt — what robots.txt does (and doesn’t) do.
• Optimize your crawl budget — crawl capacity + demand, and who actually needs it.

Bing / Microsoft

• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawling and its “crawl efficiency north star.”

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what a crawler is and how it works

• “Google downloads text, images, and videos from pages it found on the internet with automated programs called crawlers.” — Google Search Central docs. Jump to quote
• “Google Search works in three stages, and not all pages make it through each stage.” Jump to quote
• “Other pages are discovered when Google extracts a link from a known page to a new page… Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl. This process is called ‘URL discovery’.” Jump to quote
• “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “They try not to crawl the site too fast to avoid overloading it. This mechanism is based on the responses of the site (for example, HTTP 500 errors mean ‘slow down’).” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote

Google — robots.txt

• “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” — Google Search Central docs. Jump to quote

Fabrice Canel, Microsoft Bing

• “Crawling is the process by which bingbot discovers new and updated documents or content to be added to Bing’s searchable index.” Jump to quote
• “Our crawl efficiency north star is to crawl an URL only when the content has been added (URL not crawled before), updated (fresh on-page context or useful outbound links).” Jump to quote

Make sure crawlers can do their job — checklist

A quick pass to confirm bots can find, fetch, and read what matters:

• Important pages are linked from somewhere crawlable (no orphans) — links are how crawlers discover URLs.
• An XML sitemap is submitted so discovery doesn’t rely on links alone.
• robots.txt doesn’t block anything you want indexed (and does block low-value spaces you never want crawled).
• You’re not using robots.txt to try to deindex — that’s noindex’s job (with crawling allowed).
• Server returns fast, stable responses — bots throttle down on 5xx/429 (“slow down”).
• JS-dependent content is reachable via real <a href> links, not click-only navigation (rendering is a separate, later step).
• Bots in your logs are verified as genuine (reverse + forward DNS), not just trusted by their user-agent string.

The mental models

1. The loop — discover → fetch → parse → schedule → repeat. Most link-following crawlers run some version of this loop. A frontier holds URLs it knows about; a scheduler picks the next one; a fetch downloads it; a parse extracts links that feed back into the frontier. If a page isn’t getting crawled, ask which step is failing: was it ever discovered (in the frontier at all)? Is it being de-prioritized by the scheduler?

2. The pipeline — crawl → index → serve. Each stage is a filter, and “not all pages make it through each stage.” When a page underperforms, find which stage it’s failing at before changing anything.

3. The “not equals” set. Keep these four separate and most crawler confusion disappears:

• Crawling ≠ indexing (a blocked page can still be indexed via links)
• Crawling ≠ rendering (JavaScript runs in a separate, later step)
• Crawling ≠ ranking (crawl rate is not a ranking signal)
• “Googlebot” ≠ one program (it’s a fleet routed through one platform)

4. The decision rule for removing a page. Want it gone from search? Allow crawling + noindex. Want bots to skip a space entirely (and don’t care about indexing)? robots.txt disallow. Never use disallow to deindex.

Crawler — cheat sheet

Three names, one thing: crawler = spider = bot (spiderbot). Interchangeable.

The loop: discover (frontier) → fetch (download HTML) → parse (extract links + content) → schedule (decide what’s next) → repeat.

The pipeline: crawl → index → serve. Crawling is gate one; “not all pages make it through each stage.”

The “not equals”:

Politeness: crawlers throttle themselves; sustained 5xx/429 = “slow down.”

Control: robots.txt controls crawling, not indexing — don’t use it to deindex. Reputable crawlers obey it; many scrapers don’t.

Verify a crawler: reverse + forward DNS, never the user-agent string alone.

The landscape (2026): search-engine bots crawl the most, AI bots are a fast- rising #2, and Bing alone discovers tens of billions of new URLs per day.

Verify a bot is really the crawler it claims to be

A user-agent string is trivial to fake, so the only reliable check is DNS. For Googlebot, confirm a reverse + forward DNS lookup resolves to a Google domain and back to the same IP.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in googlebot.com or google.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t really Googlebot. You can also match the IP against Google’s published ranges (googlebot.json). The same reverse-then-forward pattern works for verifying other reputable crawlers against their own published hostnames.

Tools for inspecting crawler access and activity

• Robots.txt Tester — test whether a named crawler is allowed to request a URL before treating a crawl gap as a scheduling problem.
• Log File Analyzer — see which crawlers actually hit the server, which URLs they request, and the status codes they receive.
• HTTP Header Checker — inspect the response a crawler gets, including redirects and crawler directives sent in headers.
• Render Gap Analyzer — compare raw and rendered HTML when a crawler can fetch a page but important content depends on client-side rendering.

Test yourself: web crawlers

Resources worth your time

My related writing

• What Is Googlebot & How Does It Work? — the deep dive on Google’s specific crawler fleet.
• Indexed, though blocked by robots.txt — why blocked pages still get indexed (crawl ≠ index).
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party crawl-vs-rank experiment.
• When Should You Worry About Crawl Budget? — when crawling efficiency actually matters.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — the changing cast of crawlers.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Search Off the Record — “What is a web crawler, really?” — Gary Illyes and Lizzi Sassman on what a crawler is and the “Googlebot is a misnomer” point.
• Search Off the Record — “Crawling smarter, not harder” (Ep. 79) — Gary Illyes and John Mueller on crawl budget, the scheduler, and why 90%+ of sites don’t need to worry about it.
• Google Search Central Blog — Crawling December resources (2024) — Google’s own curated deep-dive series on crawling fundamentals.
• Search Engine Journal — Google’s Crawling Priorities: Insights From Gary Illyes — covers Illyes’s stated mission to decrease crawling without sacrificing quality.
• Search Engine Land — Google explains what “crawl budget” means for webmasters — the original 2017 coverage when Google first defined crawl capacity and crawl demand.
• Wikipedia — Web crawler — good for the formal CS terminology (frontier, politeness policy) and history.
• r/TechSEO — the community for crawl/index debugging.

Stats worth citing

• AI bots are closing in on search bots. From my Cloudflare Radar analysis, search-engine crawlers still crawl the most, but AI bots are a clear #2 by volume. Source
• Tens of billions of normalized URLs never seen before are discovered by Bing every day — the scale of the discovery problem the engines aggressively filter (Fabrice Canel, Microsoft Bing, 2022). Coverage

Crawler

A crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.

Related: Googlebot, Bingbot, User Agent, Crawling, Robots.txt

Crawler

A web crawler — the words spider and bot mean the same thing — is an automated software program that systematically browses the web. It fetches a page, parses it to pull out links and content, adds the newly found URLs to a queue, and repeats. Search engines run crawlers to discover and download pages so they can later be rendered, indexed, and ranked.

The crawl is stage one of three: crawl → index → serve. It’s a prerequisite for the other two, not a substitute for them. A page has to be crawlable to be indexed, but crawling on its own isn’t a ranking factor — getting crawled more often won’t lift your positions.

“Googlebot” is the best-known crawler, but it’s really a label on a fleet of specialized crawlers (desktop, mobile, image, news, video, ads) routed through one central system, not a single program. Bing’s is Bingbot, and AI companies now run a fast-growing crowd of their own crawlers. Reputable crawlers identify themselves with a user-agent string, respect robots.txt, and throttle themselves so they don’t overload your server.

Related: Googlebot, Bingbot, User Agent, Crawling, Robots.txt

Changelog