Googlebot

TL;DR — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is Google’s web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the program that visits your pages, downloads them, and hands them off so Google can indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and rank them. There are two of them (a smartphone one and a desktop one), and the smartphone one does most of the work. Being crawled is a requirement to show up in search — but getting crawled more doesn’t make you rank higher.

What Googlebot is

Google doesn’t browse the web the way you do. It sends out an automated program — a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., or bot — that visits pages, downloads what’s on them, and follows links to find more pages. That program is Googlebot. (Bing’s equivalent is BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share..)

I describe it simply in my Ahrefs guide to Googlebot: “Googlebot is the web crawler used by Google to gather the information needed and build a searchable index of the web.” Everything Google shows you in search results started with Googlebot fetching the page.

There are actually two Googlebots

Googlebot comes in two flavors:

• Googlebot Smartphone — pretends to be someone on a phone. This is the main one. Google mostly looks at the mobile version of your site (that’s called mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost.), so most crawls come from the smartphone bot.
• Googlebot Desktop — pretends to be someone on a desktop computer. It does a smaller share of the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor..

Here’s the catch: in your robots.txt file (the file that tells bots where they can go), both share the same name — Googlebot. So you can’t tell robots.txt “let the desktop one in but not the mobile one.” It’s all-or-nothing.

Does Googlebot run JavaScript?

Yes. Googlebot uses a recent version of Chrome under the hood, so it can read modern, JavaScript-heavy sites. As I put it in my Googlebot guide: “Googlebot is evergreen, meaning it sees websites as users would in the latest Chrome browser.” But running that JavaScript happens a little later, in a separate step — not the instant your page is first fetched.

The two things people get wrong

1. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is not ranking. Getting crawled more often won’t move you up the results. Crawling is just how Google finds and downloads your page — it’s a gate you have to get through, not a scoreboard.
2. Blocking Googlebot in robots.txt does not delete you from Google. It only stops Google from reading the page. If other sites link to it, the URL can still show up in results (just without a useful description). To actually keep a page out, you let Google crawl it and add a noindex tag.

Want the technical version — the exact user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings, how to verify a real Googlebot, byte limits, and the renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. queue? Switch to the Advanced tab.

TL;DR — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is Google Search’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., split into Smartphone (primary, mobile-first) and Desktop, which share one Googlebot robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. — you can’t target them separately. It runs an evergreen Chromium and renders JavaScript in a separate, later queue (crawl ≠ render). CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is required to rank but is not a ranking signal, and a robots-blocked URL can still be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. URL-only. Verify it by reverse + forward DNS to a Google domain or against Google’s published IP ranges — the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is trivially spoofed. And “Googlebot” is really just the Search-facing slice of a much bigger crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. platform.

What Googlebot actually is

Google is precise about the name: “Googlebot is the generic name for two types of web crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. used by Google Search.” Those two types are Googlebot Smartphone (“a mobile crawler that simulates a user on a mobile device”) and Googlebot Desktop (“a desktop crawler that simulates a user on desktop”).

It is not one little program running on one machine. “Googlebot runs on thousands of machines,” as I describe it in my Googlebot guide, “they determine how fast and what to crawl on websites,” distributed across datacenters worldwide but egressing primarily from US IP addresses. Discovery happens mostly through links — Google finds new URLs “primarily from links embedded in previously crawled pages” — plus sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.. (For the full discovery and scheduling picture, that’s the crawling hub’s job.)

Smartphone vs Desktop — and why it’s “smartphone-first”

Under mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost., the smartphone crawler is the primary one. Google: “For most sites Google Search primarily indexes the mobile version of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawler, and a minority using the desktop crawler.” Mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost. has been complete for all sites since October 2023, so the practical rule is: if content isn’t visible to the smartphone agent, it isn’t indexed. Match your content, structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., metadata, and robots tags across mobile and desktop.

The robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. gotcha: “Both crawler types obey the same product token (user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. token) in robots.txt, and so you cannot selectively target either Googlebot Smartphone or Googlebot Desktop using robots.txt.” The only way to differentiate is to read the HTTP user-agent request header in your own server-side logic. (For the mechanics of that header format, see mobile-first indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and user-agent.)

The user-agent strings

The robots.txt product token for both is just Googlebot. The full UA strings differ:

Googlebot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Googlebot Smartphone:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

W.X.Y.Z is a placeholder for the current Chrome version, which moves as Googlebot’s evergreen Chromium is updated. Don’t trust the string on its own, though — it’s trivially spoofed (see verification below).

Evergreen Chromium and the rendering queue

This is the distinction that trips people up most: crawling and renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. are separate steps. Googlebot runs “an evergreen version of Chromium,” and it became evergreen back in May 2019 (jumping from the old Chrome 41 to current stable), which is why it now handles ES6+, IntersectionObserver, Web Components, and modern CSS. But it doesn’t execute your JavaScript the moment it fetches the HTML.

Google: “Googlebot queues all pages with a 200 HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. for rendering, unless a robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. or header tells Google not to index the page. The page may stay on this queue for a few seconds, but it can take longer than that. Once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript.” The rendering service (WRS) behaves like a modern browser but with quirks worth knowing: it’s effectively stateless — local/session storage and cookies are cleared across page loads — it doesn’t fetch images or videos (to save bandwidth), caches aggressively (and may ignore your caching headers), and doesn’t support WebSockets or WebRTC. If your content only appears after a click or a JS-driven navigation that isn’t a real <a href> link, expect rendering trouble. (Depth lives in the rendering sibling.)

Byte limits

Googlebot doesn’t download an unlimited amount per URL. As of Google’s March 2026 Inside Googlebot update, it fetches roughly the first 2 MB of any individual URL (including the HTTP header) and up to 64 MB for a PDF. The figure is a moving target — Google’s own framing is that “this limit is not set in stone and may change over time as the web evolves and HTML pages grow in size,” and earlier docs listed 15 MB (which turned out to be the broader infrastructure default, not Search’s number). The practical point holds regardless: anything past the cutoff simply isn’t fetched — “to Googlebot, they simply don’t exist.” Keep critical content and markup above the bloat.

How Googlebot fetches — politely

• Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is algorithmic and self-throttling. “For most sites, Googlebot shouldn’t access your site more than once every few seconds on average.” It speeds up or backs off based on your server’s health.
• Status codes are the lever. Returning 429, 500, or 503 tells Googlebot to slow down — but that affects the entire hostname, not just the erroring URLs, and only works for a day or two before sustained errors start dropping pages from the index. John Mueller: “I’d only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts,” and “404s are generally fine & once discovered, Googlebot will retry them anyway.”
• crawl-delay is ignored. Google does not process the non-standard crawl-delay robots.txt directive at all. (Bing does honor it — one of the real Googlebot/BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. divergences.)
• Crawling tracks crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side., not a flat quota — capacity (what your server can take) plus demand (popularity and staleness). For most sites this is a non-issue; it only bites at real scale. The full treatment is in crawl budgetThe number of URLs an engine will crawl in a timeframe..

Verifying it’s really Googlebot

The user-agent header is “often spoofed by other crawlers” — so it alone proves nothing. Google’s crawlers identify themselves three ways: the user-agent header, the source IP, and the reverse-DNS hostname of that IP. Two real verification methods:

1. Manual (one-off). Reverse-DNS the source IP; confirm it resolves to a hostname ending in googlebot.com, google.com, or googleusercontent.com (the mask looks like crawl-***-***-***-***.googlebot.com); then forward-DNS that hostname and confirm it returns the original IP.
2. Automatic (at scale). Match the IP against Google’s published CIDR ranges. Google has split these from the old single googlebot.json into several JSON files by crawler category — the Googlebot one is https://www.gstatic.com/ipranges/common-crawlers.json (the legacy googlebot.json URL still redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. to the same data).

Both are in the Scripts tab, for macOS/Linux and Windows. Why bother? Plenty of traffic lies about being Googlebot, so logs that “show Googlebot” can be largely impostors — verify before you trust.

TIP

Verify a claimed Googlebot request against both published ranges and forward-confirmed reverse DNS with my free Googlebot Verifier Free

1. Enter the source IP from your server log and the crawler identity it claims.
2. Check both the published-range result and reverse-then-forward DNS confirmation; the user-agent string alone is not evidence.
3. Exclude or separately label failed claims before using crawler traffic in crawl-rate or log-analysis decisions.

Tap to enlarge

A focused Googlebot Verifier finding says the IP claims to be Googlebot but is outside Google's published ranges and fails forward-confirmed reverse DNS.

Googlebot is one bot in a fleet

“Googlebot” is genuinely a bit of a misnomer. Gary Illyes, March 2026: “I mean, calling it Googlebot, that’s a misnomer,” and “Googlebot is not our crawler infrastructure.” The infrastructure underneath, in his words, is “software as a service, if you like. SaaS” — a shared platform many Google products draw from. What you see in your logs is the Search slice of it: “When you see Googlebot in your server logs, you are just looking at Google Search.” He also notes there are “dozens, if not hundreds of different crawlers,” most too small to bother documenting.

The named ones you’ll actually meet alongside Googlebot include Googlebot-Image, Googlebot-Video, and Googlebot-News (which share Googlebot’s strings/tokens), Storebot-Google, and Google-InspectionTool (powers the URL Inspection and Rich Results tools). Two behave unusually: AdsBot ignores the global * robots.txt rule (a Disallow: / under User-agent: * still won’t stop it), and Google-Safety ignores robots.txt entirely. The AI-related crawlers — Google-Extended (controls Gemini training; not a ranking signal) and GoogleOther (R&D crawls, offloaded from Googlebot) — exist too, but the AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. sibling covers those in depth, so I’ll point there rather than duplicate.

How to control Googlebot

Three controls, three different effects:

• robots.txt stops crawling, not indexing. Use it to keep bots out of low-value URL spaces — never as a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool.
• noindex stops indexing — but Googlebot must be allowed to crawl the page to see the tag in the first place.
• Password protection blocks access entirely.

Which brings us to the single most misunderstood Googlebot fact: “There’s a difference between crawling and indexing; blocking Googlebot from crawling a page doesn’t prevent the URL of the page from appearing in search results.” A robots-blocked URL can still get indexed URL-only if something links to it. To actually remove a page, allow crawling and add noindex. I’ve written this up in detail in Indexed, though blocked by robots.txt.

For the wider pipeline Googlebot lives inside — URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched., the crawl scheduler, rendering, and the crawl-vs-index-vs-rank distinctions — see the crawling hub and How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank..

AI summary

A condensed take on the Advanced version:

• GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. = Google Search’s web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., in two variants: Smartphone (primary, under mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost.) and Desktop. They share one Googlebot robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., so you can’t target them separately — only via the HTTP user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. header.
• Mobile-first since Oct 2023: if content isn’t visible to the smartphone agent, it isn’t indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• Evergreen Chromium, separate renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. queue: Googlebot runs current Chromium but executes JavaScript later, in a stateless render step (storage/cookies cleared; no images/video fetched; aggressive cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency.). Crawl ≠ render.
• Byte limit ~2 MB per URL (PDFs 64 MB) as of 2026 — content beyond the cutoff isn’t fetched. The exact figure is “not set in stone.”
• Polite, algorithmic fetching: 429/5xx say “slow down” (hostname-wide, short-term); crawl-delay is ignored (Bing honors it).
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ ranking, and robots-block ≠ deindex — a blocked URL can still be indexed URL-only. Remove pages with noindex (crawling allowed), not robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere..
• Verify by reverse + forward DNS to *.googlebot.com/*.google.com/ *.googleusercontent.com, or against Google’s published IP ranges (common-crawlers.json) — the user-agent is trivially spoofed.
• “Googlebot” is a fleet, not one program — the Search-facing slice of a larger crawling platform, alongside Googlebot-Image/Video/News, Google-InspectionTool, AdsBot (ignores *), Google-Safety (ignores robots.txt), and the AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (Google-Extended, GoogleOther — see the AI crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. sibling).

Official documentation

Primary-source documentation from the search engines.

Google

• Googlebot — the canonical doc: the two crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. types, UA strings, byte limits, and crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. controls. Start here.
• Overview of Google crawlers and fetchers — every Google user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target., the three crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. categories, and the published IP ranges.
• Google’s common crawlers — the full UA-string and robots-tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. reference for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and its siblings.
• Verify Google crawlers and fetchers — reverse/forward DNS and the JSON IP-range files.
• Mobile-first indexing best practices — why Smartphone is primary and what to keep consistent.
• JavaScript SEO basics — the evergreen Chromium renderer and the renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. queue.
• How HTTP status codes affect Google’s crawlers — how 2xx/3xx/4xx/429/5xx change crawl behavior.
• Inside Googlebot (March 2026) — the current byte limits and “Googlebot is just Google Search” framing.

Bing / Microsoft (for contrast)

• Bingbot crawl control — Bing’s manual crawl scheduling and its crawl-delay support, which Googlebot doesn’t have.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is

• “Googlebot is the generic name for two types of web crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. used by Google Search.” — Google Search Central docs. Jump to quote
• “For most sites Google Search primarily indexes the mobile version of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., and a minority using the desktop crawler.” Jump to quote
• “Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. and ranking.” Jump to quote

Google — renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. and byte limits

• “While Google Search runs JavaScript with an evergreen version of Chromium, there are a few things that you can optimize.” Jump to quote
• “Googlebot queues all pages with a 200 HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. for renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., unless a robots meta tagThe robots meta tag is an HTML element in a page's head — <meta name=\"robots\" content=\"noindex\"> — that tells search engines how to index and serve that page. It's crawl-then-obey: a page blocked in robots.txt is never fetched, so the tag is never seen. or header tells Google not to index the page.” Jump to quote
• “Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file.” Jump to quote

Google — crawl vs index

• “There’s a difference between crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.; blocking Googlebot from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. a page doesn’t prevent the URL of the page from appearing in search results.” — Google Search Central docs. Jump to quote

Gary Illyes, Google (reported via Search Engine Journal and Google’s March 2026 blog post)

• “I mean, calling it Googlebot, that’s a misnomer.” … “Googlebot is not our crawler infrastructure.” … “it’s software as a service, if you like. SaaS.” Read the coverage
• “When you see Googlebot in your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., you are just looking at Google Search.” Read the coverage

John Mueller, Google (via Search Engine Journal’s Reddit coverage)

• “I’d only expect the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. to react that quickly if they were returning 429 / 500 / 503 / timeouts.” … “404s are generally fine & once discovered, Googlebot will retry them anyway.” Read the coverage

Googlebot readiness checklist

A quick pass to confirm GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. can find, fetch, render, and correctly handle your pages:

• Important content is reachable via real <a href> links (not click-only or JS-driven navigation the renderer can’t follow).
• Mobile and desktop versions match — same content, structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., metadata, and robots tags (mobile-first means the smartphone agent’s view is what counts).
• Critical content and markup sit above the ~2 MB fetch cutoff; PDFs under 64 MB.
• robots.txt doesn’t block anything you want indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — and you’re not using it to try to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. (that’s noindex’s job, with crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. allowed).
• Server returns fast, stable responses — minimal 5xx/429/timeouts, since those throttle crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. hostname-wide.
• You’re not relying on crawl-delay (Google ignores it).
• Server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. are checked with verified Googlebot (reverse + forward DNS or IP ranges) — not raw user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. matches, which impostors spoof.
• You understand a robots-blocked URL can still be indexed URL-only if linked to.

Googlebot — cheat sheet

The two crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. types

robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. product token (both): Googlebot

Other Google crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. you’ll see in logs

Fast facts

• Fetch limit: ~2 MB per URL (PDFs 64 MB), as of 2026 — beyond it isn’t fetched. (“Not set in stone.”)
• Renders with evergreen Chromium, in a separate, later queue (crawl ≠ render).
• crawl-delay: ignored by Google (Bing honors it).
• 429/5xx: temporary “slow down” — affects the whole hostname.
• Verify with reverse + forward DNS or published IP ranges — never the UA string alone.
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is not a ranking factor; robots-block is not deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist..

Verify a bot is really Googlebot

The user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. header is trivially spoofed, so confirm with a reverse-DNS lookup (must end in a Google domain) followed by a forward-DNS lookup (must resolve back to the same IP).

macOS / Linux

# 1) Reverse-DNS the IP from your logs — it should end in
#    googlebot.com, google.com, or googleusercontent.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward-DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..

Verify at scale against Google’s IP ranges

For large-scale checks, match source IPs against Google’s published CIDR ranges instead of doing per-request DNS. Google split the old single googlebot.json into several files by category — Googlebot lives in common-crawlers.json:

# Fetch the current Googlebot (common crawlers) ranges
curl -s https://www.gstatic.com/ipranges/common-crawlers.json
# The legacy URL still works and redirects to the same data:
# https://developers.google.com/static/search/apis/ipranges/googlebot.json

Load the CIDR prefixes from that JSON and test each logged IP for membership before trusting any “Googlebot” hit.

Resources worth your time

My related writing

• What Is Googlebot & How Does It Work? — my full Ahrefs guide on GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., with the crawl/control/verify detail.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — how Googlebot stacks up against the AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. by share and speed.
• JavaScript SEO Issues & Best Practices — the renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. side of what Googlebot does.
• Indexed, though blocked by robots.txt — why a robots-blocked URL can still be indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• The Beginner’s Guide to Technical SEO — where Googlebot fits in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• r/TechSEO — the community for crawl/index debugging.
• Googlebot: What it is, how it works & how to optimize (Search Engine Land) — thorough practical guide with SEL’s editorial depth; good for a second opinion on the basics.
• Google explains how crawling works in 2026 (Barry Schwartz, Search Engine Land) — solid write-up of Gary Illyes’s March 2026 Inside Googlebot post, including the 2 MB byte-limit clarification.
• Google Says They Deploy Hundreds of Undocumented Crawlers (Search Engine Journal) — covers Gary Illyes’s “calling it Googlebot is a misnomer” remarks in depth.
• Is Google’s Two Waves of Indexing Over? (Onely) — Martin Splitt’s on-record conversation about the crawl-then-render queue and how the gap is shrinking; still the clearest industry treatment of WRS timing.
• Google Reveals JavaScript Rendering Secrets at Chrome Dev Summit 2018 (Lumar) — detailed notes on WRS stateless behavior (cookies/localStorage cleared, no image fetches, WebSockets unsupported) that are still relevant today.
• From Googlebot to GPTBot: Who’s crawling your site? (Cloudflare blog) — Cloudflare Radar data on crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. share and speed showing Googlebot leading “good bot” traffic.
• Was That Really a Google Bot Crawling My Site? (Imperva Research) — data-backed look at Googlebot impersonation; context for why DNS verification matters.

Stats worth citing

• GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is the fastest crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. on the web. From my read of Cloudflare Radar data: “Googlebot is the fastest crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. on the web according to Cloudflare Radar, with Ahrefsbot being the 2nd fastest.” Source
• AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are closing in on search bots. From my Cloudflare Radar analysis, search-engine crawlers (Googlebot among them) still crawl the most, but AI bots are a clear #2 and on pace to overtake them within a couple of years. Source
• Byte limit: ~2 MB per URL (PDFs 64 MB) — Google’s documented Googlebot fetch limit as of 2026, with the explicit caveat that the figure isn’t fixed. Source

Googlebot mistakes to avoid

Using robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. to try to deindexDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. a page. robots.txt stops crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. A robots-blocked URL can still show up in results (indexed URL-only) if something links to it. Do instead: let Google crawl the page and add a noindex tag — that’s the only reliable way to remove it.

Trying to target the desktop or smartphone crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. separately in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. Both share the single Googlebot product tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., so a rule written for one applies to both. Do instead: if you need different behavior per device, read the HTTP user-agent request header in your own server-side logic — robots.txt can’t make that distinction.

Letting mobile and desktop versions of a page drift apart. Under mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost., the smartphone agent’s view is what gets indexed and ranked. Content, structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., metadata, or robots tags that only exist on desktop are effectively invisible to Google. Do instead: keep both versions in sync, and check what the smartphone agent actually sees.

Relying on crawl-delay to slow GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. down. Google doesn’t process the non-standard crawl-delay directive at all — it’s a Bing-only lever. Do instead: return 429/500/503 if you genuinely need Google to back off, understanding that throttles the whole hostname and only holds for a day or two before sustained errors start dropping pages from the index.

Trusting the Googlebot user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string in your logs at face value. The UA header is trivially spoofed, and plenty of traffic that claims to be Googlebot isn’t. Do instead: verify with reverse + forward DNS or against Google’s published IP ranges before treating “Googlebot” traffic in your analytics as real.

Assuming JavaScript-only navigation (click handlers, no real <a href>) will get discovered and rendered like a normal link. Googlebot discovers URLs primarily through links, and renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. happens later in a separate, stateless queue that doesn’t fire arbitrary UI interactions. Do instead: expose every important path as a real <a href> link that works without JavaScript.

Treating crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. as a ranking lever. Getting crawled more often isn’t a scoreboard — it’s just the gate you have to pass through to be eligible to rank. Do instead: spend effort on content and technical health, not on trying to attract more crawl hits for their own sake.

Common Googlebot issues

A page is blocked in robots.txt but still shows up in search

• Cause: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. only stops crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. If other pages link to the blocked URL, Google can still index it URL-only (usually with no snippet/description).
• Fix: if you actually want it out of results, allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. in robots.txt and add a noindex tag instead — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. has to be able to read the page to see the tag. Confirm the fix with the robots-txt-tester tool and re-check the URL’s status once it’s recrawled.

JavaScript-rendered content isn’t showing up in the index

• Cause: crawling and renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. are separate steps. The fetched HTML gets queued for renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. — usually seconds, sometimes much longer — before a headless Chromium executes the JavaScript. Content that depends on click-only or non-<a href> navigation may never render at all, since the Web Rendering Service doesn’t fire arbitrary UI interactions.
• Fix: check whether the content exists in the raw HTML response versus only after JS execution, and make sure important paths are real <a href> links. The render-gap tool is built for exactly this check — it shows the gap between what’s fetched and what actually renders.

Server logs show a lot of “Googlebot” traffic that seems off

• Cause: the Googlebot user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string is trivially spoofed. A meaningful share of traffic claiming to be Googlebot in raw logs is impostors.
• Fix: verify with reverse-DNS (should resolve to a hostname ending in googlebot.com, google.com, or googleusercontent.com) followed by forward-DNS back to the same IP, or match against Google’s published IP ranges (common-crawlers.json) for bulk log checks — see the Scripts tab for both methods, or run a log pull through the log-file-analyzer tool, which flags spoofed Googlebot hits automatically.

Crawl rate dropped suddenly

• Cause: Googlebot’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is algorithmic and self-throttling — sustained 429, 500, 503, or timeout responses tell it to back off, and that throttle applies hostname-wide, not just to the erroring URLs.
• Fix: check recent server error rates and response codes with the http-status-checker tool or your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. If errors persist more than a day or two, expect pages to start dropping from the index, not just slower crawling. Fixing the underlying 5xx/429 source is what restores the crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. — there’s no separate “unthrottle” lever.

Content that only exists on desktop isn’t ranking

• Cause: under mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost., the smartphone agent’s view is what gets indexed and ranked. If content, structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., or metadata only exists on the desktop version, it’s effectively invisible to Google.
• Fix: compare what the mobile and desktop versions serve — use the mobile-friendly-tester tool to see what the smartphone agent renders, and bring the two versions back in sync.

Tools for working with Googlebot

• robots-txt-tester — check whether a specific URL is allowed or blocked for Googlebot before you assume your robots.txt is doing what you think.
• log-file-analyzer — parse your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. to see actual GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawl activity and flag traffic that claims to be Googlebot but fails IP/DNS verification.
• render-gap — compare what Googlebot fetches versus what actually renders after JavaScript executes, directly useful for the crawl-vs-render distinction this article covers.
• mobile-friendly-tester — check what the smartphone crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (the primary one under mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost.) sees on a page.
• http-status-checker — confirm the status codes your server is returning, since sustained 429/5xx responses are what actually throttle Googlebot’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor..

Third-party: Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. shows how Googlebot last crawled and rendered a specific URL, including which crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (mobile or desktop) fetched it. Screaming Frog can crawl a site renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. as Googlebot to compare raw HTML against rendered output at scale.

Quiz

Five questions to check what stuck about GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..

Googlebot

Googlebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.

Related: Crawler, Bingbot, User Agent, Crawling, Mobile-First Indexing

Googlebot

Googlebot is the generic name for Google’s web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the automated software that fetches pages from the web so Google can build and maintain its searchable indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. It exists in two main variants: Googlebot Smartphone, which simulates a user on a mobile device and handles the majority of crawls under mobile-first indexingGoogle's practice of using the mobile version of a page's content — crawled by Googlebot smartphone — for indexing and ranking. It is not a separate index and not a ranking boost., and Googlebot Desktop, which simulates a desktop user for a minority of crawls. Both obey the same Googlebot robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. product tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., so you can’t selectively block one in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. — you can only tell them apart from the HTTP user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. header.

Both variants run an evergreen version of Chromium, meaning Googlebot sees JavaScript and modern CSS roughly the way a current Chrome user would. RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., though, happens in a separate, later step from the initial fetch — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. are not the same thing. Googlebot runs on thousands of machines distributed across datacenters, egressing primarily from US IP addresses, and is just one of dozens of Google crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and fetchers (Googlebot-Image, Google-InspectionTool, AdsBot, GoogleOther, and more).

Two distinctions matter most: crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is not a ranking factor — being crawled more won’t lift your positions — and blocking Googlebot in robots.txt is not the same as deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist., since a blocked URL can still be indexed URL-only if other pages link to it. Because the user-agent string is trivially spoofed, real verification means a reverse-DNS lookup to *.googlebot.com, *.google.com, or *.googleusercontent.com with a forward-DNS confirmation, or matching against Google’s published IP ranges.

Related: Crawler, Bingbot, User Agent, Crawling, Mobile-First Indexing

Changelog