Crawling

How search engines discover and download the web — Googlebot and Bingbot, URL discovery, the crawl scheduler, rendering, and how crawling differs from indexing and ranking. The hub for everything crawl-related.

First published: Jun 22, 2026 · Last updated: Jul 17, 2026 · Advanced
demand #2 in How Search Works#21 in Technical SEO#30 on the site
AIAuthored byAI v2
1 evidence signal on this page

Crawling is stage one of search (crawl → index → serve): bots like Googlebot and Bingbot discover URLs and download pages so they can be rendered, indexed, and ranked. It's necessary to appear in search but it isn't a ranking factor — and it's distinct from both indexing and rendering. Most sites never need to manage crawl budget; when crawling does break, server logs are the ground truth. This hub explains the whole pipeline and points you to the deep dives.

TL;DR — CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is the first of search’s three stages (crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve). BotsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. discover URLs by pull (links + sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.) and push (IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it., Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content.), then fetch them on an algorithmic schedule that throttles to your server’s health. RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. JavaScript is a separate step. Crawling is required to rank but is not itself a ranking signal, and it’s distinct from indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — a robots-blocked page can still be indexed. Most sites don’t need to manage crawl budget; logs are how you see what actually happened.

Crawling is stage one of three

Crawling is stage one of three. A page must clear each stage before it can rank. Source: /technical-seo/how-search-works/crawling/

Three stages run left to right. Crawl: a bot discovers a URL and downloads the page. Index: the engine processes the page and stores eligible information. Serve or rank: the best indexed matches are ordered for a query. The Crawl stage is highlighted, and a note says not every page advances through every stage.

© Patrick Stox LLC · CC BY 4.0 ·

Google is blunt about the pipeline: “Google Search works in three stages, and not all pages make it through each stage” — crawling, indexing, and serving. Evidence for this claim Google describes Search as three stages: crawling, indexing, and serving results. Scope: Google Search's documented processing model; it does not guarantee that a page reaches every stage. Confidence: high · Verified: Google Search Central: In-depth guide to how Google Search works The “not all pages make it” part is the whole game in technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own.. A page can be crawled but not indexed, or indexed but never served for a query. Keeping the stages separate in your head is the single most useful mental model here.

Who actually does the crawling

“GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” sounds like one program. It isn’t. In my How Search Works deck I describe it as 1,000+ systems running a family of specialized crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — desktop, mobile, image, news, video, ads — with the requests mostly originating out of Mountain View. They all draw from the same crawl budgetThe number of URLs an engine will crawl in a timeframe. pool, which is why a runaway image or parameter crawl can starve the crawling of your actual content.

And it’s not just search engines anymore. In my analysis of Cloudflare Radar data (Meet the New Web Crawlers), search-engine bots still crawl the most — but AI bots are firmly in second place and on track to overtake them in the next couple of years. If you’re reading your logs, the cast of characters has changed.

TIP Confirm a bot without doing DNS lookups by hand

Paste an IP or log line and get a tiered verdict with my free Googlebot Verifier Free

  1. Pull the IP address from your server logs (not the user-agent string — that part can be faked).
  2. Paste the IP in, pick "Googlebot" from the claims-to-be dropdown, and press Verify.
  3. Read the tiered verdict: reverse-DNS confirmed is the strongest verification; in published ranges is a strong list match, not FCrDNS confirmation; spoofed failed the available checks; unverifiable means the operator publishes no usable verification method; and not a known crawler is a neutral no-match.
  4. Got a whole log file instead of one IP? Use the bulk box below the form to match up to 500 IPs or log lines at once.

How crawlers discover URLs

Discovery is both pull and push — Google calls the whole thing “URL discovery.” Source: /technical-seo/how-search-works/crawling/

Two discovery routes feed one crawl queue. Pull discovery includes following links and sitemaps. Push discovery includes IndexNow for Bing, Yandex, and other participating engines but not Google for general pages; the Google Indexing API for JobPosting and BroadcastEvent pages; and change notifications through sitemap lastmod, RSS, and WebSub.

© Patrick Stox LLC · CC BY 4.0 ·

Discovery is both pull and push:

  • Pull — links. Google: “Other pages are discovered when Google extracts a link from a known page to a new page.” This is why orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. (nothing links to them) struggle to get found.
  • Pull — sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.. “Still other pages are discovered when you submit a list of pages (a sitemap) for Google to crawl.”
  • Push — change notifications. Instead of waiting to be re-crawled, you tell the engine something changed: IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. (Bing, Yandex, and others — Google does not use it for general pages) and Google’s Indexing API (officially only for JobPosting and BroadcastEvent pages). Sitemaps’ lastmod, RSS, and WebSubWebSub (formerly PubSubHubbub) is a push notification protocol for RSS/Atom feeds that lets a site instantly broadcast new or updated content to subscribing search engines instead of waiting to be re-crawled. round out the push options.

Google calls the whole thing “URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched..”

How crawlers fetch pages

  • The schedule is algorithmic. “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” You influence it; you don’t set it.
  • Crawling is polite. Bots throttle themselves so they don’t knock your site over: “they try not to crawl the site too fast to avoid overloading it… HTTP 500 errors mean ‘slow down.’” This is the lever behind temporarily slowing a crawl — return 503/429 and Googlebot eases off (for a day or two, not forever).
  • There’s a byte limit. As of Google’s March 2026 Inside Googlebot update, Googlebot fetches up to roughly 2 MB per URL (down from the old 15 MB figure), with PDFs allowed up to 64 MB. Go over and the fetch is truncated, not rejected — only the downloaded portion is passed on for indexing. If your critical content sits below 2 MB of bloat, that matters.
  • Caching cuts re-crawl cost. Google’s crawlers use conditional requestsA conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page. and caching so unchanged resources aren’t re-fetched every time. Google doesn’t publish an exact cache duration for renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. resources, so don’t assume a fixed refresh window for a code change — expect some lag, not an instant reflect.

Rendering is not crawling

This trips people up constantly. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Rendering is a distinct step from fetching the HTML. The renderer (the Web Rendering Service) is stateless — storage and cookies are cleared between loads, it denies permission prompts, and it rejects service workers. If your content only appears after a click or a JS-driven navigation that isn’t a real <a href> link, expect discovery and rendering problems. (Full treatment in JavaScript SEO.)

Crawling vs indexing vs ranking

The most important distinctions on this page:

  • Crawling ≠ ranking. Crawling is necessary to be in results, but it is not a ranking signal. A higher crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. won’t lift your positions. Crawl budget is an efficiency concern, full stop.
  • Crawling ≠ indexing. A page you block in robots.txt can still be indexed if other pages link to it — Google just can’t see the content, or any noindex tag you put there. As I put it in Indexed, though blocked by robots.txt: “crawling and indexing are two different things.” To actually remove a page, allow crawling and add noindex — don’t block it.

I tested the blocking side directly. In The Story of Blocking 2 High-Ranking Pages With Robots.txt I blocked two of our ranking pages. The result: “We lost a position here or there and all of the featured snippets for the pages… I expected a lot more impact, but the world didn’t end.” My takeaway stands: “Don’t block pages you want indexed. It hurts. Not as bad as you might think it does — but it still hurts.”

How to control crawling

  • robots.txt controls crawling, not indexing. “A robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file tells search engine crawlers which URLs the crawler can access on your site.” Evidence for this claim robots.txt controls crawler access and is not a reliable way to keep a URL out of Google. Scope: Google Search behavior for URLs blocked by robots.txt; blocked URLs may still be indexed when discovered elsewhere. Confidence: high · Verified: Google Search Central: Introduction to robots.txt Use it to keep bots out of low-value spaces — not as a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool.
  • Site architectureSite architecture is how a website's pages are organized, categorized, and interlinked. It controls how crawlers discover pages, how link equity flows, and how clearly search engines understand each page's topical context. Silo structure, hub and spoke, and topic clusters are the three common models. & internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. govern how deep pages sit and whether they’re discovered at all (see crawl depthCrawl depth usually means click depth — how many clicks it takes to reach a page from the homepage by following internal links. It can also mean a crawler setting that limits how many levels deep a crawl goes before it stops.).
  • Crawl-rate controls. Google retired its manual crawl-rate slider in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. (early 2024) — it now leans on your server’s responses and a lower automatic minimum. Bing still gives you a manual Crawl Control grid in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. (see crawl rate).

Crawl efficiency and crawl budget

Both engines think about crawling as efficiency. Gary Illyes frames Google’s side as crawl rate limit + crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. (popularity and staleness). Bing’s Fabrice Canel calls it their “crawl efficiency north star … to crawl a URL only when the content has been added … updated.”

The reassuring part: most sites don’t need to worry about it. Google says it plainly — if your site doesn’t have a large number of rapidly changing pages, or your pages are crawled the same day they’re published, “you don’t need to read this guide.” It starts to matter around 1M+ pages changing weekly or 10k+ changing daily (see crawl budget and crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.).

When crawling goes wrong — how to see it

Before you reach for a fix, figure out which stage is actually broken — the symptoms and the remedies are different at each one:

  • Not discovered at all? Nothing links to the page and it’s missing from your sitemap. Fix internal linkingLinks between pages on the same site. and sitemap coverage — see crawl depth and crawl frequency.

  • Discovered but never fetched? Server errors, timeouts, or a robots.txt block are turning bots away. Check GSC Crawl Stats and your logs — see crawl budget and crawl rate.

  • Fetched but rendering wrong? Content depends on a click or JS-only navigation Googlebot won’t perform, or WRS times out. See JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript. for the render-specific failure modes.

  • Fetched and rendered but still not indexed? That’s a separate decision the index makes — see the Indexing hub rather than treating it as a crawl problem.

  • Log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. is the ground truth. Your server logs show exactly which URLs bots actually hit, how often, and what status codes they got — the best way to spot wasted crawling and find pages bots never reach (see log file analysis).

  • Spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content. — infinite URL spaces from calendars, faceted navigationFaceted navigation (faceted search, product filtering) lets visitors refine a list of products or content by attribute — price, color, size, brand, rating. The SEO problem: each filter combination can spawn a distinct crawlable URL, turning a small catalog into millions of near-duplicate pages that waste crawl budget and dilute ranking signals., session IDs, or relative-link explosions — quietly burn your crawl budget on junk URLs (see spider traps).

TIP Find where crawler time is going

Turn raw access logs into a prioritized list of wasted crawler requests with my free Log File Analyzer Free

  1. Upload a representative access-log sample and isolate verified search-engine crawler requests.
  2. Compare successful page fetches with redirects, errors, and other waste-counting responses.
  3. Fix the highest-volume broken paths and redirect chains, then compare a later log window.
The useful output is not a generic crawl score; it is the share and exact paths consuming requests without returning a live page.

A completed Log File Analyzer finding says Googlebot wastes 49.6% of its crawl on non-200 responses. Of 129 Googlebot requests, 64 were waste-counting responses: 40 on 404 or 410 URLs, 12 on redirects, and 4 on 5xx errors. It recommends fixing broken links, collapsing redirect chains, returning 410 for permanently gone URLs, and using the wasted-URL table to find the exact paths.

Where to go next: the crawling cluster

This hub is the map. Each topic below is its own deep dive:

Crawl efficiency — how much and how often

  • Crawl budget — what it is (capacity + demand), what wastes it, and who actually needs to care.
  • Crawl rate — how fast bots fetch, why the GSC rate slider went away, and how to speed up or slow down a crawl today.
  • Crawl frequency — what makes Google re-crawl a page sooner (popularity, staleness, accurate lastmod) and what doesn’t.
  • Crawl depth — click depth vs. crawl traversal depth, and why important pages belong close to the homepage.

Meet the crawlers — who’s actually fetching your pages

  • Crawler — what a web crawler (bot, spider) is and the fetch → parse → follow loop they all run.
  • User agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. — the user-agent string and robots.txt token a bot identifies itself with, and why you can’t trust the string alone.
  • Googlebot — Google’s crawler: Smartphone vs. Desktop, rendering, and how to verify it.
  • BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. — Microsoft’s crawler, how it differs, and the surfaces beyond Bing it feeds.
  • AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. — the bots from AI companies (training vs. AI-search vs. user-triggered fetchers) and how to control them.

Diagnosing crawl problems

  • Log file analysis — verifying real bots and reading what they crawled.
  • Spider traps (crawler traps) — the patterns that generate infinite URLs, and how to fix them.

Every topic above is its own deep dive nested under this hub — they’re in the sidebar too.

Qualifying your outbound links is a related on-page control that overlaps with crawling — these tell Google how to treat a link rather than whether to crawl it: nofollowrel=\"nofollow\" is a value of the HTML link rel attribute that tells search engines you don't vouch for a linked page and don't want to pass ranking signals to it. Since 2019–2020 Google treats it as a hint, not a directive — and it does not reliably block crawling or indexing. (the original catch-all, now a hint), plus rel=sponsored and rel=ugcrel=\"sponsored\" and rel=\"ugc\" are link attribute values you put on <a> tags to tell search engines a link's nature: sponsored for paid/affiliate/ad links, ugc for user-generated links like comments and forum posts. Both are hints, not strict directives. for paid/advertising and user-generated links. All three live in the on-page meta tagsMeta tags are HTML elements in a page's head that pass metadata about the page to search engines and browsers. For SEO only a few matter — the title element, the meta description, and the robots meta tag — while meta keywords and most others are ignored. cluster.

Getting found in the first place is a related but separate stage. How search engines discover your URLs — internal links, sitemaps (XML, sitemap indexA sitemap index is a sitemap of sitemaps — a single file that lists your other sitemap files instead of listing URLs directly. It's how large sites stay under the 50,000-URL / 50MB-per-sitemap limit while submitting just one file., image, and video), and the push protocols IndexNow and the Google Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. (what each is actually for, and why Google doesn’t use IndexNow) — now lives in its own DiscoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched. hub. For the broader topic, see How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank..

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.