Googlebot

TL;DR — Googlebot is Google’s web crawler — the program that visits your pages, downloads them, and hands them off so Google can index and rank them. There are two of them (a smartphone one and a desktop one), and the smartphone one does most of the work. Being crawled is a requirement to show up in search — but getting crawled more doesn’t make you rank higher.

What Googlebot is

Google doesn’t browse the web the way you do. It sends out an automated program — a crawler, or bot — that visits pages, downloads what’s on them, and follows links to find more pages. That program is Googlebot. (Bing’s equivalent is Bingbot.)

I describe it simply in my Ahrefs guide to Googlebot: “Googlebot is the web crawler used by Google to gather the information needed and build a searchable index of the web.” Everything Google shows you in search results started with Googlebot fetching the page.

There are actually two Googlebots

Googlebot comes in two flavors:

• Googlebot Smartphone — pretends to be someone on a phone. This is the main one. Google mostly looks at the mobile version of your site (that’s called mobile-first indexing), so most crawls come from the smartphone bot.
• Googlebot Desktop — pretends to be someone on a desktop computer. It does a smaller share of the crawling.

Here’s the catch: in your robots.txt file (the file that tells bots where they can go), both share the same name — Googlebot. So you can’t tell robots.txt “let the desktop one in but not the mobile one.” It’s all-or-nothing.

Does Googlebot run JavaScript?

Yes. Googlebot uses a recent version of Chrome under the hood, so it can read modern, JavaScript-heavy sites. As I put it in my Googlebot guide: “Googlebot is evergreen, meaning it sees websites as users would in the latest Chrome browser.” But running that JavaScript happens a little later, in a separate step — not the instant your page is first fetched.

The two things people get wrong

1. Crawling is not ranking. Getting crawled more often won’t move you up the results. Crawling is just how Google finds and downloads your page — it’s a gate you have to get through, not a scoreboard.
2. Blocking Googlebot in robots.txt does not delete you from Google. It only stops Google from reading the page. If other sites link to it, the URL can still show up in results (just without a useful description). To actually keep a page out, you let Google crawl it and add a noindex tag.

Want the technical version — the exact user-agent strings, how to verify a real Googlebot, byte limits, and the rendering queue? Switch to the Advanced tab.

TL;DR — Googlebot is Google Search’s crawler, split into Smartphone (primary, mobile-first) and Desktop, which share one Googlebot robots.txt token — you can’t target them separately. It runs an evergreen Chromium and renders JavaScript in a separate, later queue (crawl ≠ render). Crawling is required to rank but is not a ranking signal, and a robots-blocked URL can still be indexed URL-only. Verify it by reverse + forward DNS to a Google domain or against Google’s published IP ranges — the user-agent is trivially spoofed. And “Googlebot” is really just the Search-facing slice of a much bigger crawling platform.

What Googlebot actually is

Google is precise about the name: “Googlebot is the generic name for two types of web crawlers used by Google Search.” Those two types are Googlebot Smartphone (“a mobile crawler that simulates a user on a mobile device”) and Googlebot Desktop (“a desktop crawler that simulates a user on desktop”).

It is not one little program running on one machine. “Googlebot runs on thousands of machines,” as I describe it in my Googlebot guide, “they determine how fast and what to crawl on websites,” distributed across datacenters worldwide but egressing primarily from US IP addresses. Discovery happens mostly through links — Google finds new URLs “primarily from links embedded in previously crawled pages” — plus sitemaps. (For the full discovery and scheduling picture, that’s the crawling hub’s job.)

Smartphone vs Desktop — and why it’s “smartphone-first”

Under mobile-first indexing, the smartphone crawler is the primary one. Google: “For most sites Google Search primarily indexes the mobile version of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawler, and a minority using the desktop crawler.” Mobile-first indexing has been complete for all sites since October 2023, so the practical rule is: if content isn’t visible to the smartphone agent, it isn’t indexed. Match your content, structured data, metadata, and robots tags across mobile and desktop.

The robots.txt gotcha: “Both crawler types obey the same product token (user agent token) in robots.txt, and so you cannot selectively target either Googlebot Smartphone or Googlebot Desktop using robots.txt.” The only way to differentiate is to read the HTTP user-agent request header in your own server-side logic. (For the mechanics of that header format, see mobile-first indexing and user-agent.)

The user-agent strings

The robots.txt product token for both is just Googlebot. The full UA strings differ:

Googlebot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Googlebot Smartphone:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

W.X.Y.Z is a placeholder for the current Chrome version, which moves as Googlebot’s evergreen Chromium is updated. Don’t trust the string on its own, though — it’s trivially spoofed (see verification below).

Evergreen Chromium and the rendering queue

This is the distinction that trips people up most: crawling and rendering are separate steps. Googlebot runs “an evergreen version of Chromium,” and it became evergreen back in May 2019 (jumping from the old Chrome 41 to current stable), which is why it now handles ES6+, IntersectionObserver, Web Components, and modern CSS. But it doesn’t execute your JavaScript the moment it fetches the HTML.

Google: “Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page. The page may stay on this queue for a few seconds, but it can take longer than that. Once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript.” The rendering service (WRS) behaves like a modern browser but with quirks worth knowing: it’s effectively stateless — local/session storage and cookies are cleared across page loads — it doesn’t fetch images or videos (to save bandwidth), caches aggressively (and may ignore your caching headers), and doesn’t support WebSockets or WebRTC. If your content only appears after a click or a JS-driven navigation that isn’t a real <a href> link, expect rendering trouble. (Depth lives in the rendering sibling.)

Byte limits

Googlebot doesn’t download an unlimited amount per URL. As of Google’s March 2026 Inside Googlebot update, it fetches roughly the first 2 MB of any individual URL (including the HTTP header) and up to 64 MB for a PDF. The figure is a moving target — Google’s own framing is that “this limit is not set in stone and may change over time as the web evolves and HTML pages grow in size,” and earlier docs listed 15 MB (which turned out to be the broader infrastructure default, not Search’s number). The practical point holds regardless: anything past the cutoff simply isn’t fetched — “to Googlebot, they simply don’t exist.” Keep critical content and markup above the bloat.

Worked failure: the canonical exists, but Googlebot never receives it

Imagine a product template returning 2.4 MB of HTML. An application serializes a huge product-state object and recommendation payload near the top of the document; the canonical, product description, structured data, and related-product links do not appear until roughly byte 2,180,000. A browser downloads the whole response, so View Source looks correct. Googlebot Search stops around its documented 2 MB limit, so those later signals do not exist in the fetched resource.

Diagnose the response in byte order, not only in the rendered DOM:

curl -sS -D response-headers.txt -o page.html https://example.com/product
wc -c response-headers.txt page.html
LC_ALL=C grep -abo 'rel="canonical"' page.html
LC_ALL=C grep -abo 'application/ld+json' page.html

The local byte counts are an approximation because delivery intermediaries and response handling can differ, but they answer the useful first question: are critical signals comfortably early, or are they sitting near or beyond the boundary? The fix is to remove or defer oversized inline data and emit essential metadata, primary content, and crawlable links early—not to move the same bloat around and hope the cutoff changes.

How Googlebot fetches — politely

• Crawl rate is algorithmic and self-throttling. “For most sites, Googlebot shouldn’t access your site more than once every few seconds on average.” It speeds up or backs off based on your server’s health.
• Status codes are the lever. Returning 429, 500, or 503 tells Googlebot to slow down — but that affects the entire hostname, not just the erroring URLs, and only works for a day or two before sustained errors start dropping pages from the index. John Mueller: “I’d only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts,” and “404s are generally fine & once discovered, Googlebot will retry them anyway.”
• crawl-delay is ignored. Google does not process the non-standard crawl-delay robots.txt directive at all. (Bing does honor it — one of the real Googlebot/Bingbot divergences.)
• Crawling tracks crawl demand, not a flat quota — capacity (what your server can take) plus demand (popularity and staleness). For most sites this is a non-issue; it only bites at real scale. The full treatment is in crawl budget.

Verifying it’s really Googlebot

The user-agent header is “often spoofed by other crawlers” — so it alone proves nothing. Google’s crawlers identify themselves three ways: the user-agent header, the source IP, and the reverse-DNS hostname of that IP. Two real verification methods:

1. Manual (one-off). Reverse-DNS the source IP; confirm it resolves to a hostname ending in googlebot.com, google.com, or googleusercontent.com (the mask looks like crawl-***-***-***-***.googlebot.com); then forward-DNS that hostname and confirm it returns the original IP.
2. Automatic (at scale). Match the IP against Google’s published CIDR ranges. Google has split these from the old single googlebot.json into several JSON files by crawler category — the Googlebot one is https://www.gstatic.com/ipranges/common-crawlers.json (the legacy googlebot.json URL still redirects to the same data).

Both are in the Scripts tab, for macOS/Linux and Windows. Why bother? Plenty of traffic lies about being Googlebot, so logs that “show Googlebot” can be largely impostors — verify before you trust.

Googlebot is one bot in a fleet

“Googlebot” is genuinely a bit of a misnomer. Gary Illyes, March 2026: “I mean, calling it Googlebot, that’s a misnomer,” and “Googlebot is not our crawler infrastructure.” The infrastructure underneath, in his words, is “software as a service, if you like. SaaS” — a shared platform many Google products draw from. What you see in your logs is the Search slice of it: “When you see Googlebot in your server logs, you are just looking at Google Search.” He also notes there are “dozens, if not hundreds of different crawlers,” most too small to bother documenting.

The named ones you’ll actually meet alongside Googlebot include Googlebot-Image, Googlebot-Video, and Googlebot-News (which share Googlebot’s strings/tokens), Storebot-Google, and Google-InspectionTool (powers the URL Inspection and Rich Results tools). Two behave unusually: AdsBot ignores the global * robots.txt rule (a Disallow: / under User-agent: * still won’t stop it), and Google-Safety ignores robots.txt entirely. The AI-related crawlers — Google-Extended (controls Gemini training; not a ranking signal) and GoogleOther (R&D crawls, offloaded from Googlebot) — exist too, but the AI crawlers sibling covers those in depth, so I’ll point there rather than duplicate.

How to control Googlebot

Three controls, three different effects:

• robots.txt stops crawling, not indexing. Use it to keep bots out of low-value URL spaces — never as a deindexing tool.
• noindex stops indexing — but Googlebot must be allowed to crawl the page to see the tag in the first place.
• Password protection blocks access entirely.

Which brings us to the single most misunderstood Googlebot fact: “There’s a difference between crawling and indexing; blocking Googlebot from crawling a page doesn’t prevent the URL of the page from appearing in search results.” A robots-blocked URL can still get indexed URL-only if something links to it. To actually remove a page, allow crawling and add noindex. I’ve written this up in detail in Indexed, though blocked by robots.txt.

For the wider pipeline Googlebot lives inside — URL discovery, the crawl scheduler, rendering, and the crawl-vs-index-vs-rank distinctions — see the crawling hub and How Search Works.

AI summary

A condensed take on the Advanced version:

• Googlebot = Google Search’s web crawler, in two variants: Smartphone (primary, under mobile-first indexing) and Desktop. They share one Googlebot robots.txt token, so you can’t target them separately — only via the HTTP user-agent header.
• Mobile-first since Oct 2023: if content isn’t visible to the smartphone agent, it isn’t indexed.
• Evergreen Chromium, separate rendering queue: Googlebot runs current Chromium but executes JavaScript later, in a stateless render step (storage/cookies cleared; no images/video fetched; aggressive caching). Crawl ≠ render.
• Byte limit ~2 MB per URL (PDFs 64 MB) as of 2026 — content beyond the cutoff isn’t fetched. The exact figure is “not set in stone.”
• Polite, algorithmic fetching: 429/5xx say “slow down” (hostname-wide, short-term); crawl-delay is ignored (Bing honors it).
• Crawling ≠ ranking, and robots-block ≠ deindex — a blocked URL can still be indexed URL-only. Remove pages with noindex (crawling allowed), not robots.txt.
• Verify by reverse + forward DNS to *.googlebot.com/*.google.com/ *.googleusercontent.com, or against Google’s published IP ranges (common-crawlers.json) — the user-agent is trivially spoofed.
• “Googlebot” is a fleet, not one program — the Search-facing slice of a larger crawling platform, alongside Googlebot-Image/Video/News, Google-InspectionTool, AdsBot (ignores *), Google-Safety (ignores robots.txt), and the AI crawlers (Google-Extended, GoogleOther — see the AI crawlers sibling).

Official documentation

Primary-source documentation from the search engines.

Google

• Googlebot — the canonical doc: the two crawler types, UA strings, byte limits, and crawl/index controls. Start here.
• Overview of Google crawlers and fetchers — every Google user-agent, the three crawler categories, and the published IP ranges.
• Google’s common crawlers — the full UA-string and robots-token reference for Googlebot and its siblings.
• Verify Google crawlers and fetchers — reverse/forward DNS and the JSON IP-range files.
• Mobile-first indexing best practices — why Smartphone is primary and what to keep consistent.
• JavaScript SEO basics — the evergreen Chromium renderer and the rendering queue.
• How HTTP status codes affect Google’s crawlers — how 2xx/3xx/4xx/429/5xx change crawl behavior.
• Inside Googlebot (March 2026) — the current byte limits and “Googlebot is just Google Search” framing.

Bing / Microsoft (for contrast)

• Bingbot crawl control — Bing’s manual crawl scheduling and its crawl-delay support, which Googlebot doesn’t have.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what Googlebot is

• “Googlebot is the generic name for two types of web crawlers used by Google Search.” — Google Search Central docs. Jump to quote
• “For most sites Google Search primarily indexes the mobile version of the content. As such the majority of Googlebot crawl requests will be made using the mobile crawler, and a minority using the desktop crawler.” Jump to quote
• “Google uses the mobile version of a site’s content, crawled with the smartphone agent, for indexing and ranking.” Jump to quote

Google — rendering and byte limits

• “While Google Search runs JavaScript with an evergreen version of Chromium, there are a few things that you can optimize.” Jump to quote
• “Googlebot queues all pages with a 200 HTTP status code for rendering, unless a robots meta tag or header tells Google not to index the page.” Jump to quote
• “Googlebot crawls the first 2MB of a supported file type, and the first 64MB of a PDF file.” Jump to quote

Google — crawl vs index

• “There’s a difference between crawling and indexing; blocking Googlebot from crawling a page doesn’t prevent the URL of the page from appearing in search results.” — Google Search Central docs. Jump to quote

Gary Illyes, Google (reported via Search Engine Journal and Google’s March 2026 blog post)

• “I mean, calling it Googlebot, that’s a misnomer.” … “Googlebot is not our crawler infrastructure.” … “it’s software as a service, if you like. SaaS.” Read the coverage
• “When you see Googlebot in your server logs, you are just looking at Google Search.” Read the coverage

John Mueller, Google (via Search Engine Journal’s Reddit coverage)

• “I’d only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts.” … “404s are generally fine & once discovered, Googlebot will retry them anyway.” Read the coverage

Googlebot readiness checklist

A quick pass to confirm Googlebot can find, fetch, render, and correctly handle your pages:

• Important content is reachable via real <a href> links (not click-only or JS-driven navigation the renderer can’t follow).
• Mobile and desktop versions match — same content, structured data, metadata, and robots tags (mobile-first means the smartphone agent’s view is what counts).
• Critical content and markup sit above the ~2 MB fetch cutoff; PDFs under 64 MB.
• robots.txt doesn’t block anything you want indexed — and you’re not using it to try to deindex (that’s noindex’s job, with crawling allowed).
• Server returns fast, stable responses — minimal 5xx/429/timeouts, since those throttle crawling hostname-wide.
• You’re not relying on crawl-delay (Google ignores it).
• Server logs are checked with verified Googlebot (reverse + forward DNS or IP ranges) — not raw user-agent matches, which impostors spoof.
• You understand a robots-blocked URL can still be indexed URL-only if linked to.

Googlebot — cheat sheet

The two crawler types

robots.txt product token (both): Googlebot

Other Google crawlers you’ll see in logs

Fast facts

• Fetch limit: ~2 MB per URL (PDFs 64 MB), as of 2026 — beyond it isn’t fetched. (“Not set in stone.”)
• Renders with evergreen Chromium, in a separate, later queue (crawl ≠ render).
• crawl-delay: ignored by Google (Bing honors it).
• 429/5xx: temporary “slow down” — affects the whole hostname.
• Verify with reverse + forward DNS or published IP ranges — never the UA string alone.
• Crawling is not a ranking factor; robots-block is not deindexing.

Verify a bot is really Googlebot

The user-agent header is trivially spoofed, so confirm with a reverse-DNS lookup (must end in a Google domain) followed by a forward-DNS lookup (must resolve back to the same IP).

macOS / Linux

# 1) Reverse-DNS the IP from your logs — it should end in
#    googlebot.com, google.com, or googleusercontent.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward-DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

Windows

nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

If the reverse lookup doesn’t end in a Google domain, or the forward lookup doesn’t match the original IP, it isn’t Googlebot.

Verify at scale against Google’s IP ranges

For large-scale checks, match source IPs against Google’s published CIDR ranges instead of doing per-request DNS. Google split the old single googlebot.json into several files by category — Googlebot lives in common-crawlers.json:

# Fetch the current Googlebot (common crawlers) ranges
curl -s https://www.gstatic.com/ipranges/common-crawlers.json
# The legacy URL still works and redirects to the same data:
# https://developers.google.com/static/search/apis/ipranges/googlebot.json

Load the CIDR prefixes from that JSON and test each logged IP for membership before trusting any “Googlebot” hit.

Resources worth your time

My related writing

• What Is Googlebot & How Does It Work? — my full Ahrefs guide on Googlebot, with the crawl/control/verify detail.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — how Googlebot stacks up against the AI crawlers by share and speed.
• JavaScript SEO Issues & Best Practices — the rendering side of what Googlebot does.
• Indexed, though blocked by robots.txt — why a robots-blocked URL can still be indexed.
• The Beginner’s Guide to Technical SEO — where Googlebot fits in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• r/TechSEO — the community for crawl/index debugging.
• Googlebot: What it is, how it works & how to optimize (Search Engine Land) — thorough practical guide with SEL’s editorial depth; good for a second opinion on the basics.
• Google explains how crawling works in 2026 (Barry Schwartz, Search Engine Land) — solid write-up of Gary Illyes’s March 2026 Inside Googlebot post, including the 2 MB byte-limit clarification.
• Google Says They Deploy Hundreds of Undocumented Crawlers (Search Engine Journal) — covers Gary Illyes’s “calling it Googlebot is a misnomer” remarks in depth.
• Is Google’s Two Waves of Indexing Over? (Onely) — Martin Splitt’s on-record conversation about the crawl-then-render queue and how the gap is shrinking; still the clearest industry treatment of WRS timing.
• Google Reveals JavaScript Rendering Secrets at Chrome Dev Summit 2018 (Lumar) — detailed notes on WRS stateless behavior (cookies/localStorage cleared, no image fetches, WebSockets unsupported) that are still relevant today.
• From Googlebot to GPTBot: Who’s crawling your site? (Cloudflare blog) — Cloudflare Radar data on crawler share and speed showing Googlebot leading “good bot” traffic.
• Was That Really a Google Bot Crawling My Site? (Imperva Research) — data-backed look at Googlebot impersonation; context for why DNS verification matters.

Stats worth citing

• Googlebot is the fastest crawler on the web. From my read of Cloudflare Radar data: “Googlebot is the fastest crawler on the web according to Cloudflare Radar, with Ahrefsbot being the 2nd fastest.” Source
• AI bots are closing in on search bots. From my Cloudflare Radar analysis, search-engine crawlers (Googlebot among them) still crawl the most, but AI bots are a clear #2 and on pace to overtake them within a couple of years. Source
• Byte limit: ~2 MB per URL (PDFs 64 MB) — Google’s documented Googlebot fetch limit as of 2026, with the explicit caveat that the figure isn’t fixed. Source

Googlebot mistakes to avoid

Using robots.txt to try to deindex a page. robots.txt stops crawling, not indexing. A robots-blocked URL can still show up in results (indexed URL-only) if something links to it. Do instead: let Google crawl the page and add a noindex tag — that’s the only reliable way to remove it.

Trying to target the desktop or smartphone crawler separately in robots.txt. Both share the single Googlebot product token, so a rule written for one applies to both. Do instead: if you need different behavior per device, read the HTTP user-agent request header in your own server-side logic — robots.txt can’t make that distinction.

Letting mobile and desktop versions of a page drift apart. Under mobile-first indexing, the smartphone agent’s view is what gets indexed and ranked. Content, structured data, metadata, or robots tags that only exist on desktop are effectively invisible to Google. Do instead: keep both versions in sync, and check what the smartphone agent actually sees.

Relying on crawl-delay to slow Googlebot down. Google doesn’t process the non-standard crawl-delay directive at all — it’s a Bing-only lever. Do instead: return 429/500/503 if you genuinely need Google to back off, understanding that throttles the whole hostname and only holds for a day or two before sustained errors start dropping pages from the index.

Trusting the Googlebot user-agent string in your logs at face value. The UA header is trivially spoofed, and plenty of traffic that claims to be Googlebot isn’t. Do instead: verify with reverse + forward DNS or against Google’s published IP ranges before treating “Googlebot” traffic in your analytics as real.

Assuming JavaScript-only navigation (click handlers, no real <a href>) will get discovered and rendered like a normal link. Googlebot discovers URLs primarily through links, and rendering happens later in a separate, stateless queue that doesn’t fire arbitrary UI interactions. Do instead: expose every important path as a real <a href> link that works without JavaScript.

Treating crawl frequency as a ranking lever. Getting crawled more often isn’t a scoreboard — it’s just the gate you have to pass through to be eligible to rank. Do instead: spend effort on content and technical health, not on trying to attract more crawl hits for their own sake.

Common Googlebot issues

A page is blocked in robots.txt but still shows up in search

• Cause: robots.txt only stops crawling, not indexing. If other pages link to the blocked URL, Google can still index it URL-only (usually with no snippet/description).
• Fix: if you actually want it out of results, allow crawling in robots.txt and add a noindex tag instead — Googlebot has to be able to read the page to see the tag. Confirm the fix with the robots-txt-tester tool and re-check the URL’s status once it’s recrawled.

JavaScript-rendered content isn’t showing up in the index

• Cause: crawling and rendering are separate steps. The fetched HTML gets queued for rendering — usually seconds, sometimes much longer — before a headless Chromium executes the JavaScript. Content that depends on click-only or non-<a href> navigation may never render at all, since the Web Rendering Service doesn’t fire arbitrary UI interactions.
• Fix: check whether the content exists in the raw HTML response versus only after JS execution, and make sure important paths are real <a href> links. The render-gap tool is built for exactly this check — it shows the gap between what’s fetched and what actually renders.

Server logs show a lot of “Googlebot” traffic that seems off

• Cause: the Googlebot user-agent string is trivially spoofed. A meaningful share of traffic claiming to be Googlebot in raw logs is impostors.
• Fix: verify with reverse-DNS (should resolve to a hostname ending in googlebot.com, google.com, or googleusercontent.com) followed by forward-DNS back to the same IP, or match against Google’s published IP ranges (common-crawlers.json) for bulk log checks — see the Scripts tab for both methods, or run a log pull through the log-file-analyzer tool, which flags spoofed Googlebot hits automatically.

Crawl rate dropped suddenly

• Cause: Googlebot’s crawl rate is algorithmic and self-throttling — sustained 429, 500, 503, or timeout responses tell it to back off, and that throttle applies hostname-wide, not just to the erroring URLs.
• Fix: check recent server error rates and response codes with the http-status-checker tool or your server logs. If errors persist more than a day or two, expect pages to start dropping from the index, not just slower crawling. Fixing the underlying 5xx/429 source is what restores the crawl rate — there’s no separate “unthrottle” lever.

Content that only exists on desktop isn’t ranking

• Cause: under mobile-first indexing, the smartphone agent’s view is what gets indexed and ranked. If content, structured data, or metadata only exists on the desktop version, it’s effectively invisible to Google.
• Fix: compare what the mobile and desktop versions serve — use the mobile-friendly-tester tool to see what the smartphone agent renders, and bring the two versions back in sync.

Tools for working with Googlebot

• robots-txt-tester — check whether a specific URL is allowed or blocked for Googlebot before you assume your robots.txt is doing what you think.
• log-file-analyzer — parse your server logs to see actual Googlebot crawl activity and flag traffic that claims to be Googlebot but fails IP/DNS verification.
• render-gap — compare what Googlebot fetches versus what actually renders after JavaScript executes, directly useful for the crawl-vs-render distinction this article covers.
• mobile-friendly-tester — check what the smartphone crawler (the primary one under mobile-first indexing) sees on a page.
• http-status-checker — confirm the status codes your server is returning, since sustained 429/5xx responses are what actually throttle Googlebot’s crawl rate.

Third-party: Google Search Console’s URL Inspection tool shows how Googlebot last crawled and rendered a specific URL, including which crawler (mobile or desktop) fetched it. Screaming Frog can crawl a site rendering as Googlebot to compare raw HTML against rendered output at scale.

Quiz

Five questions to check what stuck about Googlebot.

Googlebot

Googlebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.

Related: Crawler, Bingbot, User Agent, Crawling, Mobile-First Indexing

Googlebot

Googlebot is the generic name for Google’s web crawler — the automated software that fetches pages from the web so Google can build and maintain its searchable index. It exists in two main variants: Googlebot Smartphone, which simulates a user on a mobile device and handles the majority of crawls under mobile-first indexing, and Googlebot Desktop, which simulates a desktop user for a minority of crawls. Both obey the same Googlebot robots.txt product token, so you can’t selectively block one in robots.txt — you can only tell them apart from the HTTP user-agent header.

Both variants run an evergreen version of Chromium, meaning Googlebot sees JavaScript and modern CSS roughly the way a current Chrome user would. Rendering, though, happens in a separate, later step from the initial fetch — crawling and rendering are not the same thing. Googlebot runs on thousands of machines distributed across datacenters, egressing primarily from US IP addresses, and is just one of dozens of Google crawlers and fetchers (Googlebot-Image, Google-InspectionTool, AdsBot, GoogleOther, and more).

Two distinctions matter most: crawling is not a ranking factor — being crawled more won’t lift your positions — and blocking Googlebot in robots.txt is not the same as deindexing, since a blocked URL can still be indexed URL-only if other pages link to it. Because the user-agent string is trivially spoofed, real verification means a reverse-DNS lookup to *.googlebot.com, *.google.com, or *.googleusercontent.com with a forward-DNS confirmation, or matching against Google’s published IP ranges.

Related: Crawler, Bingbot, User Agent, Crawling, Mobile-First Indexing

Changelog