User Agent

TL;DR — A user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is a little line of text every browser and every botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. sends with each request to say “here’s who I am.” Google’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. says it’s GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.; Bing’s says bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.. In robots.txt you don’t write the whole line — you use a short name (a “tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.”) like Googlebot. And here’s the catch: that line is just text, so anyone can fake it. The only real way to know a bot is who it claims to be is to check where its request actually came from.

What a user agent is

Every time your browser loads a page, it sends along a short text label that says what it is — something like “I’m Chrome on a Mac.” That label is the user agent, and it travels in an HTTP header on every request. Servers can read it and react to it.

Important catch up front: the client fills in that label itself. Nothing checks it. It’s a claim, not a credential — so a user agent that says “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” is not the same thing as a request that’s actually verified as Googlebot.

Crawlers do the same thing. When Googlebot fetches your page, it sends a user agent that includes Googlebot. When BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. fetches it, the user agent includes bingbot. That’s how a bot announces itself in your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened..

The string vs. the short name

There are really two things people mean by “user agent,” and mixing them up causes a lot of confusion:

• The user-agent string is the full line in the request header. Googlebot’s is long and looks a lot like a browser.
• The user-agent token is the short name you use in robots.txt to target a bot — like Googlebot or bingbot. The token is just a piece of the full string, not the whole thing.

So when you write a rule in robots.txt, you use the short token:

User-agent: Googlebot
Disallow: /private/

You don’t paste the giant browser-looking string in there.

You can’t trust the string

This is the one thing to remember. The user-agent line is plain text, so anything can fake it. Any script can claim to be Googlebot in a single line of code — and plenty do, to sneak past blocks. Google itself says the Googlebot header is “often spoofed.”

That means you should never decide who gets access to your site based only on the user agent. If you actually need to confirm a visitor is the real Googlebot (say, you’re reading your logs), you verify by checking where the request came from — not what it says it is. The Advanced tab walks through exactly how.

A few gotchas

• robots.txt user-agent names are case-insensitive — Googlebot and googlebot are the same.
• Blocking everything with User-agent: * does not block all of Google’s bots — its ad crawlers and safety crawler ignore the wildcard.
• Showing one version of a page to a crawler and a different one to real people is cloaking, and Google treats it as spam.

Want the full picture — the token tables, every Google and Bing crawler, the exact verification commands, and the cloaking rules — switch to the Advanced tab.

TL;DR — A user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is the HTTP request header any client sends to identify itself; it’s optional, client-filled metadata, not authenticated identity. Its value is the user-agent string. Separate from that is the user-agent tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. (product token) used in robots.txt — RFC 9309 says it SHOULD be a substring of the string, a strong convention with documented exceptions (Google-Extended has no request string at all). Matching is case-insensitive, version numbers/wildcards in the token line are ignored, the most-specific group wins, and same-token groups merge but never merge with *. The string is trivially spoofed — Google calls its own “often spoofed” — so verify by reverse + forward DNS (behind any proxy/CDN, use the real client IP) against googlebot.com/google.com/googleusercontent.com for Google or search.msn.com for Bing, or match published IP ranges — and even a verified request only proves a request arrived, not that the page was indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., retrieved, or used for AI training. AdsBot and Google-Safety ignore User-agent: *. Chrome is also freezing detail out of browser UA strings (User-Agent reduction); Client Hints are the structured but opt-in replacement, and neither substitutes for crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. verification. User-agent adaptation can be legitimate, but deceptively showing crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. materially different content can be cloaking.

The header, the string, and the token

Three things, and keeping them straight is most of this topic.

• The header. User-Agent is an HTTP request header. Every client sends it: your browser, curl, a crawler, a bot. Per RFC 9110 (the core HTTP semantics standard), it’s an optional field the client fills in — client-supplied descriptive metadata, not an authenticated identity the server has verified.
• The string. The header’s value — a freeform line describing the software, version, renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. engine, and sometimes the OS.
• The token. The short identifier used in robots.txt User-agent: lines to target a crawler — Googlebot, bingbot, Google-Extended.

The relationship is the part that trips people up. RFC 9309 (the formal Robots Exclusion Protocol standard) says the token “SHOULD be a substring of the identification string that the crawler sends… in the case of HTTP, the product token SHOULD be a substring in the User-Agent header.” That’s a SHOULD, not a MUST — a strong convention the standard recommends, not a hard requirement every crawler is mechanically bound to. Google-Extended (below) is the clearest example of a documented exception to it. Don’t read the substring rule as universal just because Google follows it for most of its own tokens. The token is part of the string when a provider does supply one; you target the token in robots.txt and read the string in your logs.

Google’s own framing of how its bots identify themselves is useful here: “Google’s crawlers identify themselves through three things: the HTTP user-agent request header, the source IP address of the request, and the reverse DNS hostname of the source IP.” Note that the user-agent is only one of the three — the other two are how you actually verify it.

Google-Extended: a token with no string

The cleanest illustration of token ≠ string is Google-Extended. It controls whether Google can use your content for Gemini training and groundingGrounding is anchoring an AI model's answer to source documents it retrieves at the moment you ask — not to the patterns frozen into its weights during training. Retrieval-Augmented Generation (RAG) is the most common way to do it. — and it has no dedicated HTTP request user-agent string at all. The crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. itself is done with existing GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. strings; Google-Extended exists only as a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. control token. You’ll never see “Google-Extended” in a request header in your logs.

The practical consequence: blocking Google-Extended affects only AI-training use of your content — it does not stop GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. you for Search. They’re separate decisions controlled by separate tokens. (For the broader picture of the bots reading your site, see AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. and crawler.)

Googlebot’s user-agent strings

Googlebot is “evergreen” — it runs on a recent version of Chrome, and the Chrome version in its string updates periodically (it has since December 2019). That’s why the version appears as a W.X.Y.Z placeholder:

Googlebot Smartphone (mobile):

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Two things to internalize. First, don’t hardcode the version — W.X.Y.Z changes, and matching on it will break. Match the stable token Googlebot instead. Second, you can’t separate mobile from desktop in robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. Both variants share the one Googlebot token, so a robots.txt rule applies to both.

Google’s crawler tokens

Google runs a whole family of crawlers and fetchers, each with its own token. The ones you’ll meet most:

And the ones that break the usual rules — the special-case crawlers that ignore User-agent: *:

• AdsBot (AdsBot-Google) and AdsBot Mobile (AdsBot-Google-Mobile) — they don’t obey the wildcard. To block them you must name them explicitly.
• AdSense (Mediapartners-Google) — same; ignores the global *.
• Google-Safety — used for malware/abuse detection; it ignores robots.txt entirely.

The implication is the one people miss: User-agent: * does not block AdsBot or Google-Safety. If you “block all bots” with a wildcard and assume AdsBot is gone, it isn’t. (This is exactly the kind of surprise that lands a page in indexed though blocked by robots.txtA Google Search Console Page Indexing status: the URL was excluded from indexing because your robots.txt disallows crawling it. Usually intentional and benign — robots.txt blocks crawling, not indexing. territory — see robots.txt for the full control story.)

Bingbot’s user-agent strings

Bing rebuilt BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s string in 2022 to reflect that it renders with Microsoft Edge. The current strings:

Bingbot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

Bingbot Mobile:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The robots.txt token is just bingbot. The thing to watch: post-2022, Bingbot’s string looks almost exactly like a real Chrome/Edge browser — the only tell is the bingbot/2.0 fragment inside it. If you have any logic that filters or detects bots by UA, that change matters.

How robots.txt actually matches a token

A few rules govern which group of rules a crawler obeys (per Google’s robots.txt spec and RFC 9309):

• Most-specific match wins. Google “determines the correct group of rules by finding… the group with the most specific user agent that matches the crawler’s user agent.” A Googlebot group beats a * group for Googlebot.
• Same-token groups merge — but never with *. Multiple groups naming the same agent are combined into one. A specific-agent group and the * group are not merged; * is only the fallback when nothing specific matches.
• Case-insensitive. Field name and value both — Googlebot, googlebot, GOOGLEBOT are equivalent.
• Version numbers and wildcards in the token line are ignored. Per Google, “both googlebot/1.2 and googlebot* are equivalent to googlebot.” You cannot write User-agent: Googlebot* to match a family — the * there does nothing.

So a User-agent: line takes a token and matches it as a plain (case-insensitive) substring of the crawler’s identity — no version pinning, no wildcards inside it.

Why you can’t trust the string — and how to verify

The user-agent string is freeform text. Anything can set it. One line of curl will claim to be Googlebot, and plenty of tools and malicious bots do exactly that to slip past blocks. Google says so in its own Googlebot docs: “the HTTP user-agent request header used by Googlebot is often spoofed by other crawlers.” As I’ve put it in my Googlebot guide, “Many SEO tools and some malicious bots will pretend to be Googlebot. This may allow them to access websites that try to block them.”

So never make an access or content decision on the string alone. Verify instead.

One prerequisite before either method: get the real source IP. If your site sits behind a reverse proxy, load balancer, or CDN, the address in your default access log may be the proxy’s IP, not the crawler’s — you need the original client IP (usually forwarded in a header like X-Forwarded-For, configured correctly at your proxy) or neither verification method below means anything.

Method 1 — reverse + forward DNS (best for spot checks). Google’s two steps:

1. “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command. Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.”
2. “Run a forward DNS lookup on the domain name retrieved in step 1… Verify that it’s the same as the original accessing IP address from your logs.”

For Bingbot, the same two-step dance, but the hostname must end in search.msn.com (not a Bing-branded domain — a common surprise). Commands are in the Scripts tab.

Method 2 — published IP ranges (best at scale). Google doesn’t publish a static allowlist for hardcoding (“these IP address ranges can change”), but it does publish machine-readable CIDR JSON files you can match against (common-crawlers.json and the broader crawler files). Bing now publishes its ranges too. I built a Googlebot IP verification tool for exactly this — paste in IPs and it classifies them. Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. has a built-in “Verify Bingbot” tool as well.

DNS is better for a one-off log check; IP-range matching is better for verifying at volume. Use whichever fits — but use one of them. And treat both the expected hostnames and the range files as current as of today, not permanent — Google and Bing have changed these paths before (the IP-range JSON files moved and were renamed since this article was first written), so re-check the live verification doc if a lookup that used to work stops matching.

A UA match is not proof of downstream outcomes

Even a fully verified request — real Googlebot IP, forward-confirmed reverse DNS, everything checks out — only proves one thing: that request reached your server. It’s tempting to round that up into a much bigger claim, but each of these is a separate fact requiring separate evidence:

• Request received — a request with that user agent hit your server. (What log verification actually proves.)
• Identity confirmed — the request really came from the crawler it claims to be. (What reverse DNS / IP-range matching adds on top.)
• Content fetched and rendered — the crawler successfully rendered the page (no errors, no blocked resources). Not guaranteed just because a request landed.
• Indexed — the URL made it into the search index. A successful fetch doesn’t guarantee indexing.
• Used for retrieval, citation, or training — for AI crawlers especially (Google-Extended, GPTBot, and the rest), a crawl is not proof your content was retrieved for a specific answer, cited, or used in model training. Those are separate, mostly unobservable steps downstream of the crawl.

A verified Googlebot hit in your logs is real signal — just don’t stretch it further than what it actually shows.

TIP Verify the IP before trusting a crawler user agent

The verifier combines published ranges with forward-confirmed reverse DNS where the operator supports them. Some bots publish neither, so an unverifiable result is not proof of identity.

Paste the source IP or a log line into my Googlebot Verifier before allowlisting, blocking, or serving different content to a claimed crawler. Googlebot Verifier Free

1. Take the source IP from server logs, not from a user-supplied header.
2. Compare the claim with current published ranges and forward-confirmed reverse DNS.
3. Base access decisions on the evidence tier and recheck rules as operator ranges change.

Tap to enlarge

Web Bot Auth: where verification is heading

In 2026 Google began experimenting with Web Bot Auth — “an experimental cryptographic protocol used to authenticate requests sent by bots.” The idea is to “move beyond easily spoofed headers to a verified identity and decouple agent identity from IP addresses.” Bots cryptographically sign their requests; sites verify the signature against Google’s published public keys, and signed requests carry a Signature-Agent header. Google’s own caveat matters: “We don’t sign every request of a particular agent. Be sure that you fall back to the established methods of bot verification.” So it’s additive, not a replacement — reverse DNS and IP ranges remain your baseline today.

Browsers are getting harder to parse from the UA string too

Everything above is about crawlers, but the same “don’t over-trust the string” lesson applies to browsers, and it’s getting stronger. Chrome has been rolling out User-Agent reduction: freezing or coarsening parts of its UA string (full browser version, OS version, device model) instead of reporting them exactly, so the string can’t be used to fingerprint a specific user. Google’s own framing: “The granularity and abundance of detail can lead to user identification. The default availability of this information can lead to covert tracking.” Practically, that means UA-string parsing for exact browser/OS/device version — analytics, device detection, bug triage — is increasingly unreliable and will only get more so.

The replacement Chrome recommends is User-Agent Client Hints (UA-CH): structured data the browser sends only when a server explicitly asks for it. Low-entropy hints (browser brand, major version, mobile flag) go out by default; high-entropy hints (exact version, platform version, device model) require the server to opt in via an Accept-CH response header first — an explicit negotiation, not a broadcast. Two caveats before you lean on it: it’s a Chrome/Chromium-family mechanism, not something every browser sends, and even where it’s supported, “the value may be blank, not returned, or populated with a varying value.” Client Hints solve the browser-string problem; they are not a crawler-verification mechanism — Google and Bing still verify their own crawlers via DNS and IP ranges, not Client Hints.

User-agent targeting and cloaking

The tempting move — “detect Googlebot by its UA and serve it something special” — is both technically fragile and a policy violation.

Fragile, because Google doesn’t crawl with one UA. You’d have to correctly handle Googlebot (mobile and desktop), Google-InspectionTool, AdsBot, GoogleOther, and more, from rotating IPs — practically impossible to whitelist cleanly.

A policy violation, because serving different content to a crawler than to users is cloaking: “presenting different content to users and search engines with the intent to manipulate search rankings and mislead users.” The penalty ranges from algorithmic demotion to full deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist.. Note the line: legitimate adaptation (responsive layouts, content negotiation) is fine — it’s swapping the content itself between bots and users that crosses into cloaking.

For where the user agent sits in the bigger pipeline, see crawling (the hub) and crawler. For controlling what those bots are allowed to fetch, see robots.txt.

AI summary

A condensed take on the Advanced version:

• Three things, kept straight: the User-Agent request header — optional, client-filled metadata per RFC 9110, not authenticated identity; its value, the user-agent string; and the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. used in robots.txt. Per RFC 9309 the token SHOULD be a substring of the string — a strong convention, not a universal rule.
• Some tokens have no string. Google-Extended is the documented exception: it exists only as a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. control (Gemini training); crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. uses normal GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. strings. Blocking it doesn’t affect Search indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer./BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. strings are evergreen — the Chrome version shows as W.X.Y.Z and changes; match the stable token (Googlebot, bingbot), never the version. Mobile and desktop Googlebot share one token. Post-2022 BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. looks like a real browser except for the bingbot/2.0 fragment.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. matching: most-specific group wins; same-token groups merge but never with *; matching is case-insensitive; version numbers and wildcards in the User-agent: line are ignored (Googlebot* = Googlebot).
• AdsBot and Google-Safety ignore User-agent: * — block them by name or not at all.
• The string is trivially spoofed (Google calls its own “often spoofed”). Get the real client IP first (proxies/CDNs can mask it in your logs), then verify by reverse + forward DNS (googlebot.com/google.com/googleusercontent.com; Bing → search.msn.com) or published IP ranges — files Google has renamed/moved before, so re-check the live doc if a lookup stops matching. Never trust the string for access control.
• A verified request still isn’t proof of everything downstream. Request received, identity confirmed, content rendered, page indexed, and content retrieved/cited/trained-on are separate claims needing separate evidence — a log hit proves the first, nothing else automatically.
• Web BotA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. Auth (2026, experimental) signs requests cryptographically — additive, with DNS/IP still the fallback.
• Browser UA strings are getting harder to parse too: Chrome’s User-Agent reduction freezes exact version/OS/device detail out of the string; Client Hints are the structured, opt-in replacement — but they’re Chrome-specific and don’t substitute for crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. verification.
• Serving different content by UA is cloaking — a spam-policy violation, and fragile because Google crawls with many UAs.

Official documentation

Primary-source documentation from the search engines and the standard.

Google

• Overview of Google crawlers and fetchers (user agents) — the three identification signals and the full crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. list.
• Google’s common crawlers — the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. + user-agent-string table.
• Google Special-Case Crawlers — AdsBot, Mediapartners-Google, Google-Safety and the * exceptions.
• Verify Requests from Google Crawlers and Fetchers — the two-step DNS method and the IP-range files.
• What Is Googlebot — the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. strings and the “often spoofed” note.
• How Google Interprets the robots.txt Specification — token matching, case-insensitivity, ignored wildcards.
• Authenticating Requests with Web Bot Auth (Experimental) — the cryptographic signing protocol.
• Updating the user agent of Googlebot (2019) — why the string shows W.X.Y.Z.
• Spam Policies — Cloaking — the definition and the penalty.

Bing / Microsoft

• Announcing user-agent change for Bingbot (Apr 2022) — the current desktop + mobile strings (Fabrice Canel).
• How to Verify that Bingbot is Bingbot (Aug 2012) — the *.search.msn.com reverse-DNS method.
• Bing Webmaster Tools — Verify Bingbot — the built-in verification tool.

The standard

• RFC 9110 — HTTP Semantics, §10.1.5 User-Agent — the core HTTP spec: User-Agent is an optional, client-supplied field, not authenticated identity.
• RFC 9309: Robots Exclusion Protocol — the formal SHOULD-level definition of the product token as a substring of the user-agent string.
• MDN — User-Agent header — the HTTP syntax of the header itself.

Browser UA strings

• Chrome Privacy Sandbox — User-Agent reduction — what gets frozen/coarsened in Chrome’s UA string, and why.
• Chrome for Developers — User-Agent Client Hints — low- vs. high-entropy hints and the Accept-CH opt-in.

Quotes from the source

On-the-record statements from Google, Bing, and the RFC. Each link is a deep link that jumps to the quoted passage on the source page.

Google — how crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. identify themselves

• “Google’s crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. identify themselves through three things: the HTTP user-agent request header, the source IP address of the request, and the reverse DNS hostname of the source IP.” — Google Search Central docs. Jump to quote

Google — the string is spoofed

• “The HTTP user-agent request header used by GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is often spoofed by other crawlers.” — Google Search Central docs. Jump to quote

Google — verifying by DNS

• “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command. Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.” Jump to quote
• “Google doesn’t post a public list of IP addresses for website owners to allowlist because these IP address ranges can change.” Jump to quote

Google — robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. matching

• “All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot).” — Google’s robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. spec. Jump to quote

Google — Web Bot Auth

• “An experimental cryptographic protocol used to authenticate requests sent by bots.” Jump to quote
• “We don’t sign every request of a particular agent. Be sure that you fall back to the established methods of bot verification.” Jump to quote

Google — cloaking

• “Cloaking refers to the practice of presenting different content to users and search engines with the intent to manipulate search rankings and mislead users.” — Google Search Essentials, Spam Policies. Jump to quote

RFC 9309 — token is a substring of the string

• “The product token SHOULD be a substring of the identification string that the crawler sends to the service. For example, in the case of HTTP, the product token SHOULD be a substring in the User-Agent header.” Jump to quote

Patrick Stox — on spoofing

• “Many SEO tools and some malicious bots will pretend to be GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. This may allow them to access websites that try to block them.” — from my Googlebot guide on Ahrefs. Read it

Crawler → token → string → verify

The reference table. The tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. is what you put in robots.txt; the verify hostname is what a genuine request reverse-resolves to.

robots.txt matching rules at a glance

Fast facts

• Token = substring of the UA string (RFC 9309). Not the whole string.
• Mobile and desktop Googlebot share one token — you can’t split them in robots.txt.
• The Chrome version in the string is W.X.Y.Z — it changes; never hardcode it.
• The UA string is trivially spoofed — verify by DNS or IP, never trust the string.

Verify a bot by reverse + forward DNS

The user-agent string can be faked in one line of curl. Confirm a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is genuine by checking the IP it actually came from. The pattern is the same for Google and Bing — only the expected hostname differs.

Get the right IP first: if requests pass through a reverse proxy, load balancer, or CDN before hitting your server, your default logs may show the proxy’s address, not the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.’s. Use the true client IP (from a correctly configured forwarding header) before running either check below.

macOS / Linux

# --- Googlebot ---
# 1) Reverse DNS the IP from your logs — must end in googlebot.com, google.com, or googleusercontent.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# --- Bingbot ---
# Reverse DNS must end in search.msn.com, then forward-confirm back to the IP
host 157.55.39.1
host <the-hostname-it-returned>

Windows

:: Googlebot
nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

:: Bingbot
nslookup 157.55.39.1
nslookup <the-hostname-it-returned>

If the reverse lookup doesn’t end in the expected domain — googlebot.com / google.com / googleusercontent.com for Google, search.msn.com for Bing — or the forward lookup doesn’t match the original IP, it isn’t the real bot, no matter what the user-agent string says.

Match against published IP ranges (at scale)

For verifying lots of hits, skip per-request DNS and match the IP against the engine’s published CIDR ranges instead. Google publishes machine-readable JSON:

https://developers.google.com/static/crawling/ipranges/common-crawlers.json
https://developers.google.com/static/crawling/ipranges/special-crawlers.json
https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json

Pull the file, build the CIDR set, and test each logged IP for membership. (I built a Googlebot IP verification tool that does this for you.) Bing publishes its ranges too, and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. has a built-in “Verify BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.” check.

User-agent sanity checklist

Before you write UA rules or act on a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. in your logs:

• You’re targeting the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. in robots.txt (e.g. Googlebot), not pasting the full UA string.
• No version numbers or * inside the User-agent: line — they’re ignored (Googlebot* does nothing).
• You haven’t assumed User-agent: * blocks AdsBot or Google-Safety — it doesn’t; name them explicitly if you need to.
• If you blocked Google-Extended, you understand it only affects AI-training use — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. still crawls and indexes for Search.
• You’re not keying access control or content on the raw UA string — it’s spoofable.
• Any “is this really GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.?” check goes through reverse + forward DNS (googlebot.com / google.com / googleusercontent.com) or the published IP ranges — Bing through search.msn.com.
• No version-pinned UA matching anywhere — the Chrome W.X.Y.Z portion changes.
• You’re not serving different content to a bot UA than to users (that’s cloaking).

Tools for working with user agents

Three of my own free tools cover the three tasks people actually come to this topic for: confirming a claimed botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is real, seeing which user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are actually hitting your site, and checking what AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are allowed to do.

Googlebot Verifier — the tool for the exact problem this article keeps coming back to: the user-agent string is just text, so you can’t trust it on its own. Paste in an IP address from your logs, pick which crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. it claims to be (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., and others), and it runs the reverse + forward DNS check and the published-IP-range match for you, then returns a tiered verdict — reverse-DNS confirmed, in published ranges, spoofed, unverifiable, or not a known crawler. Got a whole day of hits to check instead of one IP? Paste up to 500 IPs or raw log lines into the bulk box.

Log File Analyzer — for the question “which user agents are actually crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. my site?” Drop in a server access log (nginx, Apache, IIS/W3C, or JSON) and it parses it entirely in your browser, breaking crawl activity down by bot and by section, flagging status-code waste, splitting AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. from search crawlers, and — the part that matters most for this topic — running a spoofer report that names requests claiming to be a known crawler’s user-agent tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. without the IP to back it up.

AI-Crawler Access Checker — for the newer family of tokens that don’t behave like GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. or bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.. Enter a URL and it checks your robots.txt against every major AI-crawler user-agent token (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and more), shows the exact rule that wins for each, and flags whether an llms.txt exists. Useful for confirming a token like Google-Extended is doing what you think it’s doing — since, as covered above, it has no request string of its own to spot in your logs.

Prompts for user-agent tasks

Two prompts built around the specific traps this topic sets — spoofing and robots.txt tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. syntax — not generic “audit my SEO” filler. Paste your own data into the placeholders.

Prompt 1 — triage a batch of user-agent strings from your logs for signs of spoofing

Paste a column of raw user-agent strings pulled from your access log (not IPs — this prompt can’t verify identity, only spot inconsistencies in the string itself):

Here is a list of raw User-Agent strings from my server access log, one per
line. For each one:
1. Say which crawler token it claims to be (e.g. Googlebot, bingbot,
   GPTBot), or "no recognizable token" if none.
2. Flag anything internally inconsistent for that claimed crawler — e.g. a
   claimed Googlebot string missing "compatible; Googlebot" or the
   "+http://www.google.com/bot.html" URL, a claimed bingbot string missing
   "bingbot/2.0", or a Chrome version that looks hand-typed rather than a
   real evergreen build.
3. Remind me that this is a text-pattern check only — it cannot confirm
   identity. Real verification requires reverse+forward DNS or matching
   against the crawler's published IP ranges.

[paste user-agent strings here]

Prompt 2 — check a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. for token-matching mistakes

Paste your full robots.txt file:

Review this robots.txt file for user-agent token mistakes:
1. Flag any User-agent line that includes a version number or a wildcard
   inside the token (e.g. "Googlebot/1.2" or "Googlebot*") — these are
   ignored, not matched as a family.
2. Check whether User-agent: * is being relied on to block AdsBot-Google,
   AdsBot-Google-Mobile, Mediapartners-Google, or Google-Safety — these
   ignore the wildcard and need their own named group if I want them
   blocked.
3. Note any duplicate groups for the same token that could be merged, and
   confirm token matching here is case-insensitive so I don't need
   near-duplicate groups for casing variants.
4. List which named groups exist and which of Google's/Bing's common
   crawler tokens (Googlebot, Googlebot-Image, Google-Extended, bingbot)
   have no explicit group at all, so I know they're falling through to *.

[paste robots.txt here]

Test yourself: user agent

Five quick questions on the header, the string, the tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., and how to verify a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. is real. Pick an answer for each, then check.

Resources worth your time

My related writing

• What Is Googlebot & How Does It Work? — the full GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. UA strings, verification methods, and my IP verification tool.
• Indexed, though blocked by robots.txt — where UA-based blocking and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. blocking collide.
• Robots.txt and SEO: Everything You Need to Know — how user-agent groups and rules actually work.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — the changing cast of user agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. in your logs.

Official / standards

• RFC 9110 — HTTP Semantics, §10.1.5 User-Agent — the base definition: optional, client-supplied metadata.
• RFC 9309: Robots Exclusion Protocol — the formal SHOULD-level product-tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.-as-substring definition.
• Google’s Overview of crawlers and fetchers and Verify Google crawlers.
• Chrome Privacy Sandbox — User-Agent reduction and User-Agent Client Hints — why browser UA strings are getting harder to parse, and what replaces them.

From others

• John Mueller — Bots that impersonate Googlebot — on spoofing and why reverse DNS is the answer.
• MDN — User-Agent header — the HTTP spec view of the header.
• r/TechSEO — the community for crawl/log debugging.
• Web Bot Auth: Google’s new experimental method to validate authentic bots (Search Engine Land, Barry Schwartz, May 2026) — the best news-desk summary of how cryptographic botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. signing works and what it means in practice.
• Google-Agent user agent identifies AI agent traffic in server logs (Search Engine Land) — covers the new user-triggered fetcher that ignores robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. and uses Web Bot Auth.
• Google Is Testing New Bot Authorization Standard (Search Engine Journal) — broader industry context on the IETF standard and which companies (Amazon, Cloudflare, Akamai, OpenAI) are backing it.
• Announcing future user-agents for Bingbot (Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. Blog, Fabrice Canel, Dec 2019) — the original announcement of BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s shift to Edge-based renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. before the 2022 rollout.
• Microsoft list of Bingbot IP addresses released (Search Engine Land) — coverage of Bing’s decision to publish IP ranges for at-scale bot verification.

User Agent

A user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.

Related: Crawler, Googlebot, Bingbot, Robots.txt

User Agent

A user agent is an HTTP request header field that every client — a browser, a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., a bot, or a command-line tool like curl — sends to identify itself to the server. Per RFC 9110, it’s an optional, client-supplied field — descriptive metadata, not an authenticated identity the server has verified. The value of that header is the user-agent string: a freeform line describing the software, its version, the renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. engine, and sometimes the operating system. For a search engine crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index., it’s the primary way the bot announces who it is on every request it makes.

There’s a second, easily-confused concept: the user-agent tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. (also called the “product token”). That’s the short identifier used in robots.txt User-agent: lines to target a specific crawler — Googlebot, bingbot, Google-Extended. Per RFC 9309, the token SHOULD be a substring of the full user-agent string — a strong convention, not an absolute rule, and it is not the full string. Some tokens, like Google-Extended, have no dedicated request-header string at all — they exist only as a robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. control.

The critical caveat: a user-agent string is just text, and anything can fake it. Anyone can send User-Agent: Googlebot/2.1 in one line of curl, and Google’s own docs note the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. header is “often spoofed.” So you never trust the string for access control. To confirm a bot is genuinely GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. or BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., you verify by reverse DNS (and a forward-confirmation lookup) — Googlebot resolves to googlebot.com, google.com, or googleusercontent.com; BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. to search.msn.com — or by matching the IP against the engine’s published ranges.

Related: Crawler, Googlebot, Bingbot, Robots.txt

Changelog