Utilisateur Agent

TL;DR — A utilisateur agent is a little line of text every navigateur and every bot sends with chaque requête to dire “here’s who I am.” Google’s robot d’exploration dit it’s Googlebot; Bing’s dit bingbot. In robots.txt vous don’t écrire the whole line — vous utiliser a short nom (a “token”) comme Googlebot. And here’s the catch: que line is simplement text, so anyone peut fake it. The seulement réel façon to know a bot is who it claims to be is to vérifier où its requête en réalité came from.

Ce que a utilisateur agent is

Every temps votre navigateur loads une page, it sends along a short text étiquette que dit Ce que c’est — something comme “I’m Chrome on a Mac.” Que étiquette is the utilisateur agent, and it travels in an HTTP header on every requête. Servers peut lire it and react to it.

Important catch up front: the client fills in que étiquette itself. Nothing checks it. It’s a claim, pas a credential — so a utilisateur agent que dit “Googlebot” n’est pas the même chose as une requête that’s en réalité verified as Googlebot.

Robots d’exploration do the même chose. Quand Googlebot récupère votre page, it sends a utilisateur agent que inclut Googlebot. Quand Bingbot récupère it, the utilisateur agent inclut bingbot. That’s how a bot announces itself in votre serveur logs.

The string vs. the short nom

Là are really two choses personnes mean by “user agent,” and mixing les up causes a lot of confusion:

• The user-agent string is the complet line in la requête header. Googlebot’s is long and semble a lot comme a navigateur.
• The user-agent token is the short nom vous utiliser in robots.txt to target a bot — comme Googlebot or bingbot. The token is simplement a piece of the complet string, pas the whole chose.

So quand vous écrire a rule in robots.txt, vous utiliser the short token:

User-agent: Googlebot
Disallow: /private/

Vous don’t paste the giant browser-looking string in là.

Vous pouvez’t trust the string

Ce is the un chose to remember. The user-agent line is plain text, so anything peut fake it. Quelconque script peut claim to be Googlebot in a unique line of code — and plenty do, to sneak past blocks. Google itself dit the Googlebot header is “souvent spoofed.”

Que signifie vous devez jamais decide who obtient accès to votre site fondé seulement on the utilisateur agent. Si vous en réalité besoin to confirmer a visitor is the réel Googlebot (dire, you’re reading votre logs), vous vérifier by checking où la requête came from — pas ce que it dit it is. The Avancé tab walks via exactly how.

A few gotchas

• robots.txt user-agent noms are case-insensitive — Googlebot and googlebot are the même.
• Blocking everything with User-agent: * fait pas block tout of Google’s bots — its ad robots d’exploration and safety robot d’exploration ignore the wildcard.
• Showing un version of une page to a robot d’exploration and a différent un to réel personnes is cloaking, and Google treats it as spam.

Vouloir the complet picture — the token tables, every Google and Bing robot d’exploration, the exact verification commands, and the cloaking rules — switch to the Avancé tab.

TL;DR — A utilisateur agent is the HTTP requête header quelconque client sends to identifier itself; it’s optional, client-filled metadata, pas authenticated identity. Its valeur is the user-agent string. Separate from que is the user-agent token (product token) utilisé in robots.txt — RFC 9309 dit it DEVRAIT be a substring of the string, a strong convention with documented exceptions (Google-Extended has aucun requête string at tout). Matching is case-insensitive, version numbers/wildcards in the token line are ignored, the most-specific groupe wins, and same-token groupes merge but jamais merge with *. The string is trivially spoofed — Google calls its propre “often spoofed” — so vérifier by reverse + forward DNS (behind quelconque proxy/CDN, utiliser the réel client IP) contre googlebot.com/google.com/googleusercontent.com pour Google or search.msn.com pour Bing, or match publié IP ranges — and même a verified requête seulement proves une requête arrived, pas que lune page was indexé, retrieved, or utilisé pour AI training. AdsBot and Google-Safety ignore User-agent: *. Chrome is aussi freezing detail out of navigateur UA strings (User-Agent reduction); Client Hints are the structured but opt-in replacement, and neither substitutes pour robot d’exploration verification. User-agent adaptation peut be legitimate, but deceptively showing robots d’exploration materially différent content peut be cloaking.

The header, the string, and the token

Three choses, and keeping les straight is la plupart of ce topic.

• The header. User-Agent is an HTTP requête header. Every client sends it: votre navigateur, curl, a robot d’exploration, a bot. Per RFC 9110 (the core HTTP semantics standard), it’s an optional field the client fills in — client-supplied descriptive metadata, pas an authenticated identity le serveur has verified.
• The string. The header’s valeur — a freeform line describing the software, version, rendering engine, and parfois the OS.
• The token. The short identifier utilisé in robots.txt User-agent: lines to target a robot d’exploration — Googlebot, bingbot, Google-Extended.

The relationship is the partie que trips personnes up. RFC 9309 (the formal Robots Exclusion Protocol standard) dit the token “DEVRAIT be a substring of the identification string que the robot d’exploration sends… in the cas of HTTP, the product token DEVRAIT be a substring in the User-Agent header.” That’s a SHOULD, pas a MUST — a strong convention the standard recommends, pas a hard requirement every robot d’exploration is mechanically bound to. Google-Extended (ci-dessous) is the clearest exemple of a documented exception to it. Don’t lire the substring rule as universal simplement parce que Google follows it pour la plupart of its propre tokens. The token is partie of the string quand a provider fait supply un; vous target the token in robots.txt and lire the string in votre logs.

Google’s propre framing of how its bots identifier themselves is utile ici: “Google’s robots d’exploration identifier themselves via three choses: the HTTP user-agent requête header, the source IP adresse of la requête, and the reverse DNS hostname of the source IP.” Remarque que the user-agent is seulement un of the three — the autre two are how vous en réalité vérifier it.

Google-Extended: a token with aucun string

The cleanest illustration of token ≠ string is Google-Extended. It contrôle si Google peut utiliser votre content pour Gemini training and grounding — and it has aucun dedicated HTTP requête user-agent string at tout. The exploration itself is fait with existing Googlebot strings; Google-Extended exists seulement as a robots.txt contrôler token. You’ll jamais voir “Google-Extended” in une requête header in votre logs.

The practical consequence: blocking Google-Extended affecte seulement AI-training utiliser of votre content — it fait pas arrêter Googlebot from exploration and indexation vous pour Search. They’re separate decisions controlled by separate tokens. (Pour the broader picture of the bots reading votre site, voir AI robots d’exploration and robot d’exploration.)

Googlebot’s user-agent strings

Googlebot is “evergreen” — it runs on a recent version of Chrome, and the Chrome version in its string updates periodically (it has since December 2019). That’s pourquoi the version apparaît as a W.X.Y.Z placeholder:

Googlebot Smartphone (mobile):

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

Googlebot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; Googlebot/2.1; +http://www.google.com/bot.html) Chrome/W.X.Y.Z Safari/537.36

Two choses to internalize. Premier, don’t hardcode the version — W.X.Y.Z changements, and matching on it va break. Match the stable token Googlebot à la place. Second, vous pouvez’t separate mobile from desktop in robots.txt. Les deux variants share the un Googlebot token, so a robots.txt rule s’applique to les deux.

Google’s robot d’exploration tokens

Google runs a whole family of robots d’exploration and fetchers, chaque with its propre token. The ones you’ll meet la plupart:

And the ones que break the usual rules — the special-case robots d’exploration que ignore User-agent: *:

• AdsBot (AdsBot-Google) and AdsBot Mobile (AdsBot-Google-Mobile) — ils don’t obey the wildcard. To block les vous doit nom les explicitly.
• AdSense (Mediapartners-Google) — même; ignores the global *.
• Google-Safety — utilisé pour malware/abuse detection; it ignores robots.txt entirely.

The implication is the un personnes miss: User-agent: * ne fait pas block AdsBot or Google-Safety. Si vous “block all bots” with a wildcard and assume AdsBot is gone, it isn’t. (Ce is exactly the kind of surprise que lands une page in indexé though blocked by robots.txt territory — voir robots.txt pour the complet contrôler story.)

Bingbot’s user-agent strings

Bing rebuilt Bingbot’s string in 2022 to reflect que it renders with Microsoft Edge. The current strings:

Bingbot Desktop:

Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36

Bingbot Mobile:

Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm)

The robots.txt token is simplement bingbot. The chose to watch: post-2022, Bingbot’s string semble almost exactly comme a réel Chrome/Edge navigateur — the seulement tell is the bingbot/2.0 fragment à l’intérieur it. Si vous have quelconque logic que filters or detects bots by UA, que modifier matters.

How robots.txt en réalité matches a token

A few rules govern qui groupe of rules a robot d’exploration obeys (per Google’s robots.txt spec and RFC 9309):

• Most-specific match wins. Google “determines the correct groupe of rules by finding… the groupe with the la plupart spécifique utilisateur agent que matches the crawler’s utilisateur agent.” A Googlebot groupe beats a * groupe pour Googlebot.
• Same-token groupes merge — but jamais with *. Multiple groupes naming the même agent are combined into un. A specific-agent groupe and the * groupe are pas merged; * is seulement the fallback quand nothing spécifique matches.
• Case-insensitive. Field nom and valeur les deux — Googlebot, googlebot, GOOGLEBOT are equivalent.
• Version numbers and wildcards in the token line are ignored. Per Google, “both googlebot/1.2 and googlebot* are equivalent to googlebot.” Vous ne peut pas écrire User-agent: Googlebot* to match a family — the * là fait nothing.

So a User-agent: line takes a token and matches it as a plain (case-insensitive) substring of the crawler’s identity — aucun version pinning, aucun wildcards à l’intérieur it.

Pourquoi vous pouvez’t trust the string — and how to vérifier

The user-agent string is freeform text. Anything peut définir it. Un line of curl va claim to be Googlebot, and plenty of outils and malicious bots do exactly que to slip past blocks. Google dit so in its propre Googlebot docs: “the HTTP user-agent requête header utilisé by Googlebot is souvent spoofed by autre robots d’exploration.” As I’ve put it in my Googlebot guide, “Nombreux SEO outils and some malicious bots va pretend to be Googlebot. Ce may autoriser les to accès websites que essayer to block les.”

So jamais faire an accès or content decision on the string alone. Vérifier à la place.

Un prerequisite avant soit méthode: obtenir the réel source IP. Si votre site sits behind a reverse proxy, charger balancer, or CDN, the adresse in votre par défaut accès log may be the proxy’s IP, pas the crawler’s — vous besoin the original client IP (usually forwarded in a header comme X-Forwarded-For, configuré correctement at votre proxy) or neither verification méthode ci-dessous signifie anything.

Méthode 1 — reverse + forward DNS (meilleur pour spot checks). Google’s two steps:

1. “Run a reverse DNS lookup on the accessing IP adresse from votre logs, en utilisant the host command. Vérifier que the domain nom is soit googlebot.com, google.com, or googleusercontent.com.”
2. “Run a forward DNS lookup on the domain nom retrieved in step 1… Vérifier que it’s the même as the original accessing IP adresse from votre logs.”

Pour Bingbot, the même two-step dance, but the hostname doit fin in search.msn.com (pas a Bing-branded domain — a courant surprise). Commands are in the Scripts tab.

Méthode 2 — publié IP ranges (meilleur at scale). Google doesn’t publish a static allowlist pour hardcoding (“these IP address ranges can change”), but it fait publish machine-readable CIDR JSON fichiers vous pouvez match contre (common-crawlers.json and the broader robot d’exploration fichiers). Bing now publishes its ranges aussi. I construit a Googlebot IP verification outil pour exactly ce — paste in IPs and it classifies les. Bing Webmaster Outils has a built-in “Vérifier Bingbot” outil as bien.

DNS is meilleur pour a one-off log vérifier; IP-range matching is meilleur pour verifying at volume. Utiliser whichever fits — but utiliser un of les. And treat les deux the attendu hostnames and the range fichiers as current as of today, pas permanent — Google and Bing have modifié ces paths avant (the IP-range JSON fichiers déplacé and were renamed since ce article was premier written), so re-check the live verification doc si a lookup que utilisé to fonctionner arrête matching.

A UA match n’est pas proof of downstream outcomes

Même a entièrement verified requête — réel Googlebot IP, forward-confirmed reverse DNS, everything checks out — seulement proves un chose: que requête reached votre serveur. It’s tempting to round que up into a beaucoup bigger claim, but chaque of ces is a separate fact requiring separate evidence:

• Requête reçu — une requête with que utilisateur agent hit votre serveur. (Ce que log verification en réalité proves.)
• Identity confirmed — la requête really came from the robot d’exploration it claims to be. (Ce que reverse DNS / IP-range matching adds on top.)
• Content récupéré and rendered — the robot d’exploration successfully rendered lune page (aucun errors, aucun blocked resources). Pas guaranteed simplement parce que une requête landed.
• Indexé — l’URL made it into the search index. A successful récupérer doesn’t guarantee indexation.
• Utilisé pour retrieval, citation, or training — pour AI robots d’exploration surtout (Google-Extended, GPTBot, and the rest), a explorer n’est pas proof votre content was retrieved pour a spécifique réponse, cited, or utilisé in model training. Ceux are separate, mostly unobservable steps downstream of the explorer.

A verified Googlebot hit in votre logs is réel signal — simplement don’t stretch it plus loin que ce que it en réalité montre.

Web Bot Auth: où verification is heading

In 2026 Google began experimenting with Web Bot Auth — “an experimental cryptographic protocol utilisé to authenticate requêtes sent by bots.” The idea is to “déplacer au-delà easily spoofed headers to a verified identity and decouple agent identity from IP addresses.” Bots cryptographically sign leur requêtes; sites vérifier the signature contre Google’s publié public keys, and signed requêtes carry a Signature-Agent header. Google’s propre caveat matters: “We don’t sign every requête of a particulier agent. Be certain que vous fall back to the established méthodes of bot verification.” So it’s additive, pas a replacement — reverse DNS and IP ranges remain votre baseline today.

Navigateurs are getting harder to parse from the UA string aussi

Everything ci-dessus is à propos de robots d’exploration, but the même “don’t over-trust the string” lesson s’applique to navigateurs, and it’s getting stronger. Chrome has been rolling out User-Agent reduction: freezing or coarsening parts of its UA string (complet navigateur version, OS version, device model) au lieu de reporting les exactly, so the string can’t be utilisé to fingerprint a spécifique utilisateur. Google’s propre framing: “The granularity and abundance of detail peut lead to utilisateur identification. The par défaut availability of ce information peut lead to covert tracking.” Practically, que signifie UA-string parsing pour exact navigateur/OS/device version — analytics, device detection, bug triage — is increasingly unreliable and va seulement obtenir plus so.

The replacement Chrome recommends is User-Agent Client Hints (UA-CH): structured données le navigateur sends seulement quand a server explicitly demande pour it. Low-entropy hints (navigateur brand, major version, mobile flag) go out by par défaut; high-entropy hints (exact version, platform version, device model) exiger le serveur to opt in via an Accept-CH réponse header premier — an explicit negotiation, pas a broadcast. Two caveats avant vous lean on it: it’s a Chrome/Chromium-family mechanism, pas something every navigateur sends, and même où it’s pris en charge, “the valeur may be blank, pas renvoyé, or populated with a varying valeur.” Client Hints solve le navigateur-string problem; ils ne sont pas a crawler-verification mechanism — Google and Bing encore vérifier leur propre robots d’exploration via DNS and IP ranges, pas Client Hints.

User-agent targeting and cloaking

The tempting déplacer — “detect Googlebot by its UA and serve it something special” — is les deux technically fragile and a policy violation.

Fragile, parce que Google doesn’t explorer with un UA. You’d have to correctement handle Googlebot (mobile and desktop), Google-InspectionTool, AdsBot, GoogleOther, and plus, from rotating IPs — practically impossible to whitelist cleanly.

A policy violation, parce que serving différent content to a robot d’exploration que to utilisateurs is cloaking: “presenting différent content to utilisateurs and moteur de recherches with the intent to manipulate search rankings and mislead utilisateurs.” The penalty ranges from algorithmic demotion to complet deindexing. Remarque the line: legitimate adaptation (responsive layouts, content negotiation) is fine — it’s swapping le contenu itself entre bots and utilisateurs que crosses into cloaking.

Pour où the utilisateur agent sits in the bigger pipeline, voir exploration (the hub) and robot d’exploration. Pour controlling ce que ceux bots are allowed to récupérer, voir robots.txt.

AI summary

A condensed prendre on the Avancé version:

• Three choses, kept straight: the User-Agent requête header — optional, client-filled metadata per RFC 9110, pas authenticated identity; its valeur, the user-agent string; and the token utilisé in robots.txt. Per RFC 9309 the token DEVRAIT be a substring of the string — a strong convention, pas a universal rule.
• Some tokens have aucun string. Google-Extended is the documented exception: it exists seulement as a robots.txt contrôler (Gemini training); exploration uses normal Googlebot strings. Blocking it doesn’t affecter Search indexation.
• Googlebot/Bingbot strings are evergreen — the Chrome version montre as W.X.Y.Z and changements; match the stable token (Googlebot, bingbot), jamais the version. Mobile and desktop Googlebot share un token. Post-2022 Bingbot semble comme a réel navigateur except pour the bingbot/2.0 fragment.
• robots.txt matching: most-specific groupe wins; same-token groupes merge but jamais with *; matching is case-insensitive; version numbers and wildcards in the User-agent: line are ignored (Googlebot* = Googlebot).
• AdsBot and Google-Safety ignore User-agent: * — block les by nom or pas at tout.
• The string is trivially spoofed (Google calls its propre “often spoofed”). Obtenir the réel client IP premier (proxies/CDNs peut mask it in votre logs), alors vérifier by reverse + forward DNS (googlebot.com/google.com/googleusercontent.com; Bing → search.msn.com) or publié IP ranges — fichiers Google has renamed/déplacé avant, so re-check the live doc si a lookup arrête matching. Jamais trust the string pour accès contrôler.
• A verified requête encore isn’t proof of everything downstream. Requête reçu, identity confirmed, content rendered, page indexé, and content retrieved/cited/trained-on are separate claims needing separate evidence — a log hit proves the premier, nothing sinon automatically.
• Web Bot Auth (2026, experimental) signs requêtes cryptographically — additive, with DNS/IP encore the fallback.
• Navigateur UA strings are getting harder to parse aussi: Chrome’s User-Agent reduction freezes exact version/OS/device detail out of the string; Client Hints are the structured, opt-in replacement — but they’re Chrome-specific and don’t substitute pour robot d’exploration verification.
• Serving différent content by UA is cloaking — a spam-policy violation, and fragile parce que Google crawls with nombreux UAs.

Documentation officielle

Primary-source documentation from the moteur de recherches and the standard.

Google

• Overview of Google robots d’exploration and fetchers (utilisateur agents) — the three identification signals and the complet robot d’exploration liste.
• Google’s courant robots d’exploration — the token + user-agent-string table.
• Google Special-Case Robots d’exploration — AdsBot, Mediapartners-Google, Google-Safety and the * exceptions.
• Vérifier Requêtes from Google Robots d’exploration and Fetchers — the two-step DNS méthode and the IP-range fichiers.
• Ce que Is Googlebot — the Googlebot strings and the “often spoofed” remarque.
• How Google Interprets the robots.txt Specification — token matching, case-insensitivity, ignored wildcards.
• Authenticating Requêtes with Web Bot Auth (Experimental) — the cryptographic signing protocol.
• Updating the utilisateur agent of Googlebot (2019) — pourquoi the string montre W.X.Y.Z.
• Spam Policies — Cloaking — the definition and the penalty.

Bing / Microsoft

• Announcing user-agent modifier pour Bingbot (Apr 2022) — the current desktop + mobile strings (Fabrice Canel).
• How to Vérifier que Bingbot is Bingbot (Aug 2012) — the *.search.msn.com reverse-DNS méthode.
• Bing Webmaster Outils — Vérifier Bingbot — the built-in verification outil.

The standard

• RFC 9110 — HTTP Semantics, §10.1.5 User-Agent — the core HTTP spec: User-Agent is an optional, client-supplied field, pas authenticated identity.
• RFC 9309: Robots Exclusion Protocol — the formal SHOULD-level definition of the product token as a substring of the user-agent string.
• MDN — User-Agent header — the HTTP syntax of the header itself.

Navigateur UA strings

• Chrome Privacy Sandbox — User-Agent reduction — ce que obtient frozen/coarsened in Chrome’s UA string, and pourquoi.
• Chrome pour Developers — User-Agent Client Hints — low- vs. high-entropy hints and the Accept-CH opt-in.

Quotes from the source

On-the-record statements from Google, Bing, and the RFC. Chaque lien is a deep lien que jumps to the quoted passage on the source page.

Google — how robots d’exploration identifier themselves

• “Google’s crawlers identify themselves through three things: the HTTP user-agent request header, the source IP address of the request, and the reverse DNS hostname of the source IP.” — Recherche Google Central docs. Jump to quote

Google — the string is spoofed

• “The HTTP user-agent request header used by Googlebot is often spoofed by other crawlers.” — Recherche Google Central docs. Jump to quote

Google — verifying by DNS

• “Run a reverse DNS lookup on the accessing IP address from your logs, using the host command. Verify that the domain name is either googlebot.com, google.com, or googleusercontent.com.” Jump to quote
• “Google doesn’t post a public list of IP addresses for website owners to allowlist because these IP address ranges can change.” Jump to quote

Google — robots.txt token matching

• “All non-matching text is ignored (for example, both googlebot/1.2 and googlebot* are equivalent to googlebot).” — Google’s robots.txt spec. Jump to quote

Google — Web Bot Auth

• “An experimental cryptographic protocol used to authenticate requests sent by bots.” Jump to quote
• “We don’t sign every request of a particular agent. Be sure that you fall back to the established methods of bot verification.” Jump to quote

Google — cloaking

• “Cloaking refers to the practice of presenting different content to users and search engines with the intent to manipulate search rankings and mislead users.” — Recherche Google Essentials, Spam Policies. Jump to quote

RFC 9309 — token is a substring of the string

• “The product token SHOULD be a substring of the identification string that the crawler sends to the service. For example, in the case of HTTP, the product token SHOULD be a substring in the User-Agent header.” Jump to quote

Patrick Stox — on spoofing

• “Many SEO tools and some malicious bots will pretend to be Googlebot. This may allow them to access websites that try to block them.” — from my Googlebot guide on Ahrefs. Lire it

Robot d’exploration → token → string → vérifier

The référence table. The token is ce que vous put in robots.txt; the vérifier hostname is ce que a genuine requête reverse-resolves to.

robots.txt matching rules En un coup d’œil

Fast facts

• Token = substring of the UA string (RFC 9309). Pas the whole string.
• Mobile and desktop Googlebot share un token — vous pouvez’t split les in robots.txt.
• The Chrome version in the string is W.X.Y.Z — it changements; jamais hardcode it.
• The UA string is trivially spoofed — vérifier by DNS or IP, jamais trust the string.

Vérifier a bot by reverse + forward DNS

The user-agent string peut be faked in un line of curl. Confirmer a bot is genuine by checking the IP it en réalité came from. The pattern is the même pour Google and Bing — seulement the attendu hostname differs.

Obtenir the correct IP premier: si requêtes réussir via a reverse proxy, charger balancer, or CDN avant hitting votre serveur, votre par défaut logs may montrer the proxy’s adresse, pas the crawler’s. Utiliser the vrai client IP (from a correctement configuré forwarding header) avant running soit vérifier ci-dessous.

macOS / Linux

# --- Googlebot ---
# 1) Reverse DNS the IP from your logs — must end in googlebot.com, google.com, or googleusercontent.com
host 66.249.66.1
# → 1.66.249.66.in-addr.arpa domain name pointer crawl-66-249-66-1.googlebot.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host crawl-66-249-66-1.googlebot.com
# → crawl-66-249-66-1.googlebot.com has address 66.249.66.1

# --- Bingbot ---
# Reverse DNS must end in search.msn.com, then forward-confirm back to the IP
host 157.55.39.1
host <the-hostname-it-returned>

Windows

:: Googlebot
nslookup 66.249.66.1
nslookup crawl-66-249-66-1.googlebot.com

:: Bingbot
nslookup 157.55.39.1
nslookup <the-hostname-it-returned>

Si the reverse lookup doesn’t fin in the attendu domain — googlebot.com / google.com / googleusercontent.com pour Google, search.msn.com pour Bing — or the forward lookup doesn’t match the original IP, it isn’t the réel bot, aucun matter ce que the user-agent string dit.

Match contre publié IP ranges (at scale)

Pour verifying lots of hits, skip per-request DNS and match the IP contre the engine’s publié CIDR ranges à la place. Google publishes machine-readable JSON:

https://developers.google.com/static/crawling/ipranges/common-crawlers.json
https://developers.google.com/static/crawling/ipranges/special-crawlers.json
https://developers.google.com/static/crawling/ipranges/user-triggered-fetchers.json

Pull the fichier, construire the CIDR définir, and tester chaque logged IP pour membership. (I construit a Googlebot IP verification outil que fait ce pour vous.) Bing publishes its ranges aussi, and Bing Webmaster Outils has a built-in “Vérifier Bingbot” vérifier.

User-agent sanity checklist

Avant vous écrire UA rules or act on a bot in votre logs:

• You’re targeting the token in robots.txt (e.g. Googlebot), pas pasting the complet UA string.
• Aucun version numbers or * à l’intérieur the User-agent: line — they’re ignored (Googlebot* ne fait pashing).
• Vous haven’t assumed User-agent: * blocks AdsBot or Google-Safety — it doesn’t; nom les explicitly si vous devez.
• Si vous blocked Google-Extended, vous comprendre it seulement affecte AI-training utiliser — Googlebot encore crawls and indexes pour Search.
• You’re pas keying accès contrôler or content on the raw UA string — it’s spoofable.
• Quelconque “is this really Googlebot?” vérifier goes via reverse + forward DNS (googlebot.com / google.com / googleusercontent.com) or the publié IP ranges — Bing via search.msn.com.
• Aucun version-pinned UA matching anywhere — the Chrome W.X.Y.Z portion changements.
• You’re pas serving différent content to a bot UA que to utilisateurs (that’s cloaking).

Outils pour working with utilisateur agents

Three of my propre free outils cover the three tasks personnes en réalité come to ce topic pour: confirming a claimed bot is réel, seeing qui utilisateur agents are en réalité hitting votre site, and checking ce que AI robots d’exploration are allowed to do.

Googlebot Verifier — the outil pour the exact problem ce article garde coming back to: the user-agent string is simplement text, so vous pouvez’t trust it on its propre. Paste in an IP adresse from votre logs, pick qui robot d’exploration it claims to be (Googlebot, Bingbot, and others), and it runs the reverse + forward DNS vérifier and the published-IP-range match pour vous, alors renvoie a tiered verdict — reverse-DNS confirmed, in publié ranges, spoofed, unverifiable, or pas a connu robot d’exploration. Got a whole day of hits to vérifier au lieu de un IP? Paste up to 500 IPs or raw log lines into the bulk box.

Log Fichier Analyzer — pour the question “qui utilisateur agents are en réalité exploration my site?” Drop in a server accès log (nginx, Apache, IIS/W3C, or JSON) and it parses it entirely in votre navigateur, breaking explorer activity bas by bot and by section, flagging status-code waste, splitting AI robots d’exploration from search robots d’exploration, and — the partie que matters la plupart pour ce topic — running a spoofer report que noms requêtes claiming to be a connu crawler’s user-agent token sans the IP to back it up.

AI-Crawler Accès Checker — pour the newer family of tokens que don’t behave comme Googlebot or bingbot. Enter une URL and it checks votre robots.txt contre every major AI-crawler user-agent token (GPTBot, ClaudeBot, PerplexityBot, Google-Extended, and plus), montre the exact rule que wins pour chaque, and flags si an llms.txt exists. Utile pour confirming a token comme Google-Extended is doing ce que vous think it’s doing — since, as covered ci-dessus, it has aucun requête string of its propre to spot in votre logs.

Prompts pour user-agent tasks

Two prompts construit autour the spécifique traps ce topic sets — spoofing and robots.txt token syntax — pas generic “audit my SEO” filler. Paste votre propre données into the placeholders.

Prompt 1 — triage a batch of user-agent strings from votre logs pour signs of spoofing

Paste a column of raw user-agent strings pulled from votre accès log (pas IPs — ce prompt can’t vérifier identity, seulement spot inconsistencies in the string itself):

Here is a list of raw User-Agent strings from my server access log, one per
line. For each one:
1. Say which crawler token it claims to be (e.g. Googlebot, bingbot,
   GPTBot), or "no recognizable token" if none.
2. Flag anything internally inconsistent for that claimed crawler — e.g. a
   claimed Googlebot string missing "compatible; Googlebot" or the
   "+http://www.google.com/bot.html" URL, a claimed bingbot string missing
   "bingbot/2.0", or a Chrome version that looks hand-typed rather than a
   real evergreen build.
3. Remind me that this is a text-pattern check only — it cannot confirm
   identity. Real verification requires reverse+forward DNS or matching
   against the crawler's published IP ranges.

[paste user-agent strings here]

Prompt 2 — vérifier a robots.txt pour token-matching mistakes

Paste votre complet robots.txt fichier:

Review this robots.txt file for user-agent token mistakes:
1. Flag any User-agent line that includes a version number or a wildcard
   inside the token (e.g. "Googlebot/1.2" or "Googlebot*") — these are
   ignored, not matched as a family.
2. Check whether User-agent: * is being relied on to block AdsBot-Google,
   AdsBot-Google-Mobile, Mediapartners-Google, or Google-Safety — these
   ignore the wildcard and need their own named group if I want them
   blocked.
3. Note any duplicate groups for the same token that could be merged, and
   confirm token matching here is case-insensitive so I don't need
   near-duplicate groups for casing variants.
4. List which named groups exist and which of Google's/Bing's common
   crawler tokens (Googlebot, Googlebot-Image, Google-Extended, bingbot)
   have no explicit group at all, so I know they're falling through to *.

[paste robots.txt here]

Testez vos connaissances: utilisateur agent

Five rapide questions on the header, the string, the token, and how to vérifier a bot is réel. Pick an réponse pour chaque, alors vérifier.

Ressources utiles

My connexe writing

• Ce que Is Googlebot & How Fait It Fonctionner? — the complet Googlebot UA strings, verification méthodes, and my IP verification outil.
• Indexé, though blocked by robots.txt — où UA-based blocking and robots.txt blocking collide.
• Robots.txt and SEO: Everything Vous devez Know — how user-agent groupes and rules en réalité fonctionner.
• Meet the Nouveau Web Robots d’exploration: AI Bots Are Closing in on Moteur de recherche Bots — the modification cast of utilisateur agents in votre logs.

Official / standards

• RFC 9110 — HTTP Semantics, §10.1.5 User-Agent — the base definition: optional, client-supplied metadata.
• RFC 9309: Robots Exclusion Protocol — the formal SHOULD-level product-token-as-substring definition.
• Google’s Overview of robots d’exploration and fetchers and Vérifier Google robots d’exploration.
• Chrome Privacy Sandbox — User-Agent reduction and User-Agent Client Hints — pourquoi navigateur UA strings are getting harder to parse, and ce que replaces les.

From others

• John Mueller — Bots que impersonate Googlebot — on spoofing and pourquoi reverse DNS is the réponse.
• MDN — User-Agent header — the HTTP spec view of the header.
• r/TechSEO — the community pour explorer/log debugging.
• Web Bot Auth: Google’s nouveau experimental méthode to validate authentic bots (Moteur de recherche Land, Barry Schwartz, May 2026) — the meilleur news-desk summary of how cryptographic bot signing fonctionne and Ce que cela signifie En pratique.
• Google-Agent utilisateur agent identifies AI agent trafic in server logs (Moteur de recherche Land) — covers the nouveau user-triggered fetcher que ignores robots.txt and uses Web Bot Auth.
• Google Is Testing Nouveau Bot Authorization Standard (Moteur de recherche Journal) — broader industry context on the IETF standard and qui companies (Amazon, Cloudflare, Akamai, OpenAI) are backing it.
• Announcing future user-agents pour Bingbot (Bing Webmaster Blog, Fabrice Canel, Dec 2019) — the original announcement of Bingbot’s shift to Edge-based rendering avant the 2022 rollout.
• Microsoft liste of Bingbot IP addresses released (Moteur de recherche Land) — coverage of Bing’s decision to publish IP ranges pour at-scale bot verification.

Journal des modifications