Bingbot

TL;DR — BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. is Microsoft Bing’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the bot that visits your pages, downloads them, and adds them to the Bing search indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. It works a lot like GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. Blocking it prevents Bing from crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. the affected URLs and can also affect products or partners that use Bing’s search infrastructure, so verify the impact against current Microsoft documentation.

What Bingbot is

When you see bingbot in your server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., that’s Microsoft’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. visiting your site. Just like GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. does for Google, Bingbot follows links, reads sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., downloads your pages, and hands them off to be added to Bing’s index — the database Bing pulls from when someone searches.

It launched back on October 1, 2010, taking over from an older bot called MSNBot. The name you’ll see in robots.txt and your logs is simply bingbot.

Why it matters more than you’d think

Here’s the part that surprises people. Bing’s slice of the search market is small, so it’s tempting to ignore Bingbot entirely. But the Bing index Bingbot builds doesn’t only feed Bing. It also powers:

• Yahoo Search
• DuckDuckGo
• Ecosia
• Microsoft Edge’s default search
• Microsoft Copilot

So if you block Bingbot, you’re not just disappearing from Bing — you’re cutting yourself off from all of those at once. That’s a much bigger deal than “3% market share” makes it sound.

Can Bingbot read JavaScript?

Yes. There used to be a worry that Bing couldn’t handle JavaScript-heavy sites, but Bingbot now runs on the same Chromium engine that powers Microsoft Edge, so it renders pages the way a modern browser does. If your content shows up in a current version of Edge or Chrome, Bingbot can generally see it too.

How to help (or stop) Bingbot

• To let it in: make sure your important pages are linked, submit a sitemapThe Google Search Console report where you submit sitemaps and watch how Google processes them — type, last read date, status, and how many URLs were discovered. It confirms Google read your list; it doesn't prove anything got indexed. in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility., and don’t accidentally block them in robots.txt.
• To slow it down: Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. has a feature called Crawl Control that lets you set how fast Bingbot crawls, hour by hour.
• To speed things up: there’s a “push” option called IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. that lets you tell Bing the instant a page changes, instead of waiting for the next crawl.

Want the technical version — the user-agent strings, how to verify a request is really Bingbot, the other Microsoft crawlers, and how Bingbot feeds Copilot? Switch to the Advanced tab.

TL;DR — BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. is Microsoft Bing’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (robots.txt tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory.: bingbot), live since October 1, 2010 when it replaced MSNBot. It’s evergreen on the Chromium-based Edge engine, so it renders modern JavaScript. Its real reach is broader than traffic attributed directly to bing.com because Microsoft uses its search infrastructure across products and partnerships. Verify it using Bing’s documented verification method rather than trusting the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string. Manage the crawl with robots.txt, Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility., and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — Bing’s push protocol, co-created with Yandex.

From MSNBot to Bingbot

Bingbot is Microsoft’s primary web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the program that discovers, fetches, and processes pages to build and refresh the Bing indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. It went live on October 1, 2010, replacing the legacy MSNBot. That transition came with a robots.txt rule that still matters if you have old directives lying around: Microsoft’s announcement said that “as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, the BingBot directives will take precedence.” So MSNBot rules don’t quietly override your Bingbot rules — it’s the other way around.

Bingbot powers more than Bing

This is the single most important reframe on this page. Bing’s direct search-market share is small, which leads a lot of SEOs to skip it. But the index Bingbot builds also feeds Yahoo Search, DuckDuckGo, Ecosia, Edge’s default search, and Microsoft Copilot — and historically it powered ChatGPT Search too. The Yahoo dependency goes all the way back to the 2010 launch, when Microsoft warned site owners: “To keep your content in the Bing index, please ensure that BingBot has the same access to your site as given to MSNBot.”

The practical consequence: a Disallow against bingbot doesn’t cost you “just Bing.” It cascades across that whole ecosystem at once. Whether you care about Bing specifically or not, Bingbot is the gateway to a meaningful chunk of the searchable web — and increasingly to AI answers.

The user-agent string (and why it’s evergreen)

Bingbot’s user-agent contains the bingbot/2.0 token plus a Chromium-based Chrome/W.X.Y.Z version number, where W.X.Y.Z is substituted with the current stable Microsoft Edge build (Microsoft’s own example: 100.0.4896.127) — that substitution is what makes the string evergreen. Per Microsoft’s April 2022 announcement of the finalized strings, the desktop UA is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 and the mobile UA is Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm). Microsoft’s earlier, December 2019 preview of these strings had also proposed an extra trailing Edg/W.X.Y.Z fragment, but that fragment isn’t in the final strings Microsoft published in 2022 and confirmed fully live in February 2023 — if you’re matching bingbot’s UA in code, don’t require it. The older legacy string was the bare Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm).

“Evergreen” means exactly that the version number tracks the latest stable Edge automatically. Practically, this killed the old two-tier crawl/render gap: because Bingbot runs the same Chromium engine family as modern Edge and Chrome, it renders JavaScript the way a current browser does. Bing also treats good-faith dynamic renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. as a legitimate option for heavy JS sites — it isn’t cloaking in their view.

Verifying Bingbot — the user-agent isn’t enough

User-agent strings are trivially spoofed, so a request claiming to be Bingbot proves nothing on its own. The reliable check is two-step reverse DNS:

1. Reverse (PTR) lookup the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. IP from your logs. A genuine Bingbot IP resolves to a hostname ending in search.msn.com — the pattern is msnbot-<ip-with-dashes>.search.msn.com (so 157.55.39.1 → msnbot-157-55-39-1.search.msn.com).
2. Forward (A) lookup that hostname. It must resolve back to the original IP.

If either step fails, it isn’t Bingbot. Microsoft does publish an IP list at bing.com/toolbox/bingbot.json (28 IPv4 prefixes, last generated January 2024), but the company’s own guidance is to prefer reverse DNS over the IP list, because the addresses can change at any time. Treat the JSON as a rough reference, not a firewall allow-list. (The same reverse-DNS-over-IP-list logic applies to GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. — see the verification scripts in the Scripts tab.)

How Bingbot discovers and crawls

Discovery is link-driven. Fabrice Canel, Microsoft’s longtime Bingbot lead, describes the approach as following importance: “We are guided by key pages that are important on the internet and we follow links to understand what’s next.” The scale is enormous — in a 2019 interview Canel said “Every day, my team discovers more than 100 billion new URLs never seen before, while useless URLs parameters are ignored.” (A 2020 figure put new-URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched. at around 70 billion per day; both are real numbers from different dates, so it’s worth keeping the year attached when you cite either one.)

Two behaviors set Bingbot apart from how Google describes its own crawler:

• Pre-filtering before fetch. Bingbot makes a cost/value judgment before it even downloads a URL. As Bing has put it, “Pages that are deemed to have absolutely no potential for being useful in satisfying a user’s search query in Bing results are not retained.”
• URL memory. Bingbot remembers URLs essentially forever: “Bingbot retains every URL in memory and comes back and recrawls intermittently, even if all links to it have been removed.” That’s why a page can still see Bingbot hits long after you’ve stripped every internal linkAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. to it.

Underneath all of it is a politeness-first posture. Canel: “Our default crawl policy is to be as polite as possible when crawling the web.” And his candid take on the problem itself: “Crawling right is a fascinating engineering problem that hasn’t fully been solved yet.”

Other Microsoft crawlers

Bingbot isn’t the only Microsoft bot you’ll see:

• AdIdxBot (adidxbot) — the Bing Ads quality-control crawler. The catch worth knowing: it shares crawl budgetThe number of URLs an engine will crawl in a timeframe. with Bingbot, so heavy ad crawling can eat into the budget for your organic pages.
• BingPreview — generates page snapshots and preview thumbnails.
• MicrosoftPreview — link previews in Teams and Office.
• BingVideoPreview — video previews shown only in Bing. Like the others it has desktop and mobile variants, and it identifies itself against +https://aka.ms/microsoftbots rather than the bingbot.htm URL the older crawlers use.
• MSNBot / MSNBot-Media — the legacy crawler, retired October 1, 2010.

Controlling Bingbot’s crawl

Three levers, from blunt to surgical:

• robots.txt — the bingbot token controls crawling. Use it to keep the bot out of low-value spaces; it isn’t a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. tool.
• Crawl Control (Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility.) — lets you set Bingbot’s hourly crawl rate, so you can slow it during peak traffic and let it run faster off-peak. This is a genuine Google/Bing divergence: Google retired its crawl-rate slider in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance., while Bing still gives you an active control.
• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — instead of waiting to be re-crawled, you push the change. It’s a protocol “created by Microsoft Bing and Yandex, allowing websites to easily notify search engines whenever their website content is created, updated, or deleted.” One ping reaches every participating engine. I won’t duplicate the full treatment here — see the IndexNow deep dive for setup, the key file, and the 10,000-URL-per-request limit.

The strategic point I keep coming back to: Bing fixed a reputation for aggressive crawling by getting efficient, and IndexNow is the centerpiece of that. As I’ve written in my analysis of AI and search engine bot crawling, “It’s estimated that 53% of crawler traffic is wasted effort, and IndexNow can help cut that number down to almost nothing.” Pushing changes beats trying to coax more exploratory crawling.

Bingbot, Copilot, and the AI opt-out gap

Because the Bing index feeds Microsoft Copilot, Bingbot is effectively an AI crawler too. Canel’s guidance for showing up well there is concrete: “schema markupSchema markup is code that uses the schema.org vocabulary to label what your content means so search engines can understand it and show rich results. It's most often written in JSON-LD, and it's not a direct ranking factor. helps Microsoft’s LLMs understand your content,” and “Gen AIs value fresh content in particular, partly as a reference check of their LLM training data. Use the API at indexnow.org to push that information as it’s published or updated.”

There’s a wrinkle competitor articles tend to skip. Google offers a Google-Extended token that lets you opt out of AI training without affecting search indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. By most third-party reads there’s no equivalent separate opt-out for Bing — meaning blocking bingbot to stay out of Copilot training would also drop you from Bing search and everything downstream of it. I’d treat that as an observation about how the controls currently line up rather than a settled Microsoft policy, and check the current documentation before acting on it.

For where Bingbot sits in the bigger picture — the crawler family, Googlebot, the user-agent concept, AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls., robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., and IndexNow — the crawling hub ties it all together.

AI summary

A condensed take on the Advanced version:

• BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. = Microsoft Bing’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. robots.txt tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. bingbot; live since October 1, 2010 when it replaced MSNBot (and BingBot directives take precedence over leftover MSNBot ones).
• Reach > market share. The indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. Bingbot builds also powers Yahoo, DuckDuckGo, Ecosia, Edge default search, and Copilot — so blocking bingbot cascades far beyond Bing.
• Evergreen renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM.. Runs on the Chromium-based Edge engine; UA carries the bingbot/2.0 token plus a Chrome/W.X.Y.Z version number substituted with the current stable Edge build (no separate Edg/ fragment in the final 2022 strings). It renders modern JavaScript; good-faith dynamic renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. isn’t cloaking.
• Verify by two-step reverse DNS: PTR to msnbot-*.search.msn.com, then a forward lookup that matches the IP. User-agentsA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. are spoofable; prefer reverse DNS over the bingbot.json IP list (which changes).
• How it crawls: follows “key pages” and links, discovers ~100 billion new URLs a day (2019; ~70 billion in a 2020 figure), pre-filters URLs with “no potential” before fetching, and keeps every URL in memory — recrawling even after all links are removed.
• Other Microsoft bots: AdIdxBot (shares Bingbot’s crawl budgetThe number of URLs an engine will crawl in a timeframe.), BingPreview, MicrosoftPreview, BingVideoPreview; MSNBot is retired.
• Control it with robots.txt, Crawl Control (hourly rate — Google removed its equivalent), and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. (push, co-created with Yandex). Patrick’s framing: ~53% of crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. traffic is wasted, and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. cuts that toward nothing.
• AI angle: Bingbot feeds Copilot; schema and fresh contentContent freshness is how recent or up-to-date a page is — by its original publish date, its last substantive revision, or the currency of the facts inside it. It only helps rankings when the query itself benefits from recent results (Query Deserves Freshness), and cosmetic date changes with no real update don't count. help; there’s no clear Bing equivalent to Google-Extended, so blocking for AI training also drops search (treat as observation, verify current docs).

Official documentation

Primary-source documentation, mostly from Microsoft/Bing, with a Google contrast link. Heads up: many blogs.bing.com posts and the bing.com/webmasters/help pages are JavaScript-rendered or redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., so confirm details in a browser if a link bounces.

Bing / Microsoft

• Which crawlers does Bing use? — the official list of Microsoft botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and their user-agent tokensA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. (BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., AdIdxBot, and the preview crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.).
• Announcing user-agent change for Bing crawler bingbot — Microsoft’s own finalized desktop and mobile UA strings (confirmed fully live February 2023); this is the source for the exact strings quoted above.
• Bingbot IP list (bingbot.json) — the published IPv4 prefixes. Use as a reference only; Microsoft recommends reverse DNS instead.
• Verify Bingbot tool — Microsoft’s online lookup for confirming a crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. IP is genuinely Bingbot.
• Bing Webmaster Tools — Crawl Control — set Bingbot’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. by hour.
• IndexNow / indexnow.org — the push protocol for instantly signaling created, updated, or deleted URLs.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s “crawl efficiency” philosophy.

Google (for contrast)

• Overview of Google crawlers and fetchers — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s user-agents and IP ranges, the parallel to Bing’s setup. Note the divergences: Google retired its manual crawl-rate control, doesn’t use IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. for general pages, and offers Google-Extended to opt out of AI training separately from search.

Quotes from the source

On-the-record statements about BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. — mostly from Fabrice Canel, Microsoft’s Bingbot lead. Each link is a deep link that jumps to the quoted passage.

Fabrice Canel, Microsoft Bing — discovery and scale

• “We are guided by key pages that are important on the internet and we follow links to understand what’s next.” Jump to quote
• “Every day, my team discovers more than 100 billion new URLs never seen before, while useless URLs parameters are ignored.” (Search Engine Journal interview, 2019.) Jump to quote
• “CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. right is a fascinating engineering problem that hasn’t fully been solved yet.” (SEJ, 2019.) Jump to quote
• “Our default crawl policy is to be as polite as possible when crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. the web.” (SEJ, 2019.) Jump to quote

Microsoft Bing — how Bingbot handles URLs

• “Bingbot retains every URL in memory and comes back and recrawlsCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. intermittently, even if all links to it have been removed.” Jump to quote
• “Pages that are deemed to have absolutely no potential for being useful in satisfying a user’s search query in Bing results are not retained.” Jump to quote

Fabrice Canel, Microsoft Bing — efficiency, schema, and AI

• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” Jump to quote
• “schema markupSchema markup is code that uses the schema.org vocabulary to label what your content means so search engines can understand it and show rich results. It's most often written in JSON-LD, and it's not a direct ranking factor. helps Microsoft’s LLMsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). understand your content.” (SMX Munich, 2025.) Jump to quote
• “Gen AIs value fresh contentContent freshness is how recent or up-to-date a page is — by its original publish date, its last substantive revision, or the currency of the facts inside it. It only helps rankings when the query itself benefits from recent results (Query Deserves Freshness), and cosmetic date changes with no real update don't count. in particular, partly as a reference check of their LLM training data. Use the API at indexnowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it..org to push that information as it’s published or updated.” Jump to quote

Microsoft Bing — the 2010 launch and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it.

• “as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file, the BingBot directives will take precedence.” (Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. blog, 2010.)
• “To keep your content in the Bing indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., please ensure that BingBot has the same access to your site as given to MSNBot.” (Bing Webmaster blog, 2010.)
• “IndexNow is a new protocol created by Microsoft Bing and Yandex, allowing websites to easily notify search engines whenever their website content is created, updated, or deleted.” (Bing Webmaster blog, 2021.)

Patrick Stox (me) — Bing’s efficiency turnaround

• “There was a lot of blowback on Bing in the past because of their aggressive crawling.” Jump to quote
• “It’s estimated that 53% of crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. traffic is wasted effort, and IndexNow can help cut that number down to almost nothing.” Jump to quote

What should I do about Bingbot?

BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. decisions branch on two things: whether a request is genuinely Bingbot, and whether you want to slow it down, speed it up, or block it outright. Click through it.

Bingbot & Microsoft crawlers — cheat sheet

User-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings

W.X.Y.Z = the current stable Microsoft Edge version, substituted into the Chrome/ tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. (evergreen — it updates itself). These are the final strings Microsoft published in April 2022 and confirmed fully live in February 2023; an earlier December 2019 preview had also proposed a trailing Edg/W.X.Y.Z fragment, but it isn’t part of the finalized UA — don’t require it in botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.-matching code.

The Microsoft crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. family

Fast facts

• Launched October 1, 2010, replacing MSNBot.
• If both MSNBot and BingBot directives exist in robots.txt, BingBot wins.
• Verify via reverse DNS to *.search.msn.com, not the user-agent and not the IP list (IPs change).
• IP list: bing.com/toolbox/bingbot.json — 28 IPv4 prefixes, generated Jan 2024.
• Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. sets the hourly crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. (Google removed its equivalent).
• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. = push changed URLs; co-created with Yandex.

Verify a bot is really Bingbot

A request that says it’s BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. proves nothing — the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string is trivially spoofed. As Microsoft puts it, the fact that a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. uses the exact Bingbot user-agent doesn’t make it Bingbot. Confirm it with a two-step reverse + forward DNS check. A genuine Bingbot IP reverse-resolves to a hostname ending in search.msn.com (pattern msnbot-<ip-with-dashes>.search.msn.com), and that hostname must forward-resolve back to the same IP.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in search.msn.com
host 157.55.39.1
# → 1.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-1.search.msn.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host msnbot-157-55-39-1.search.msn.com
# → msnbot-157-55-39-1.search.msn.com has address 157.55.39.1

Windows

nslookup 157.55.39.1
nslookup msnbot-157-55-39-1.search.msn.com

If the reverse lookup doesn’t end in search.msn.com, or the forward lookup doesn’t return the original IP, it isn’t Bingbot — drop or rate-limit it. You can also use Microsoft’s online Verify Bingbot tool, and the published bingbot.json IP list as a rough cross-check — but Microsoft is explicit that the IPs can change, so DNS is the source of truth, not a hardcoded allow-list. (Same approach as verifying GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.; see the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. hub’s Scripts tab for the Google version.)

Bingbot readiness checklist

A quick pass to confirm you’re working with BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., not against it:

• robots.txt doesn’t block bingbot from anything you want in Bing (remember: that also gates Yahoo, DuckDuckGo, Ecosia, and Copilot).
• No stale MSNBot directives fighting your Bingbot rules (BingBot wins, but clean them up).
• XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. submitted in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility., listing only canonical, indexable URLs with accurate lastmod.
• BotA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. verification uses reverse + forward DNS, not the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. or a hardcoded IP list.
• If you firewall or allow-list bots, you’re verifying by DNS — not pinning the bingbot.json IPs (they change).
• Crawl Control reviewed if Bingbot traffic strains your server at peak hours.
• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. wired up so created/updated/deleted URLs are pushed, not waited on.
• JS-dependent content is reachable and renders (Bingbot is evergreen, but real <a href> links still beat click-only navigation).
• Aware that AdIdxBot shares Bingbot’s crawl budgetThe number of URLs an engine will crawl in a timeframe. if you run Bing Ads.
• Before blocking bingbot for AI reasons, confirmed there’s no separate Copilot opt-out — blocking it currently drops you from Bing search too.

The mental models

1. Reach, not market share. Bing’s direct search share is small, so it’s tempting to judge BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. by that number alone. The reframe that matters: the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. Bingbot builds is the backend for Yahoo Search, DuckDuckGo, Ecosia, Edge’s default search, and Microsoft Copilot. A Disallow: bingbot rule doesn’t cost you “3% of search” — it costs you every one of those surfaces at once. Judge Bingbot decisions by ecosystem reach, not by Bing’s standalone number.

2. Verify by DNS, not by user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.. A request’s user-agent string is just a header the client sets — anyone can type bingbot/2.0 into a curl command. The only claim that can’t be faked cheaply is a DNS record pointing back to Microsoft’s own search.msn.com domain, checked both directions (reverse then forward). Treat every other signal — user-agent, the published IP list — as a hint, not proof.

3. Three levers, blunt to surgical. robots.txt is the blunt instrument: it keeps Bingbot out of a path entirely, but it’s a crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. control, not a deindexingDeindexing means getting a URL to stop appearing in Google's search results. There's no single delete button — the right method depends on whether you own the page, whether removal is temporary or permanent, and whether the content should still exist. one. Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. is the dial: it shapes when Bingbot hits you hardest, hour by hour, without shutting anything out. IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. is the scalpel: it doesn’t wait for Bingbot to come back around at all — it pushes a single URL the moment it changes. Reach for the least blunt lever that solves the actual problem.

4. Google retired the dial; Bing kept it. Google removed its crawl-rate limiter from Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.. Bing’s Crawl Control still gives you an active hourly rate control. If you’re used to treating crawl-rate management as “a thing you can’t really do anymore,” that’s true for GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. but not for Bingbot — it’s a real, current lever worth using when Bingbot traffic strains your server at peak hours.

The standing KPI: total Bing-ecosystem referral traffic

Bing’s own market-share number understates what BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. is worth to you, because the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. it builds also feeds Yahoo Search, DuckDuckGo, Ecosia, and Microsoft Copilot. The honest KPI isn’t “Bing traffic” in isolation — it’s the combined referral traffic across that whole ecosystem, tracked over time.

Segment referral traffic by the full Bing-index family

• Metric — Sessions/clicks in your analytics tool where the referrer is bing.com, search.yahoo.com, duckduckgo.com, ecosia.org, or copilot.microsoft.com, summed and tracked as one group alongside (not instead of) your Google number.
• What it tells you — Whether blocking or under-serving Bingbot is actually costing you traffic beyond Bing itself. A dip that shows up across all five referrers at once is a Bingbot access problem, not a one-engine ranking problem.
• How to pull it — Segment your existing analytics tool (GA4, server logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened., or whatever you already run) by referrer domain for each of the five above, and pair it with Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Search Performance, which reports Bing’s own clicks and impressions directly (it can’t see Yahoo/DuckDuckGo/Ecosia downstream numbers, so analytics referrer data is the only place that combined view exists).
• Benchmark / realistic range — There’s no honest industry-wide number to target here — the ecosystem split varies enormously by audience and geography. Establish your own baseline from a few months of referrer data before you judge any single month’s move.
• Cadence — Monthly is enough for the trend; check weekly right after a robots.txt change, an IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. rollout, or anything else that touches Bingbot’s access.

Tools for verifying and controlling Bingbot

• Googlebot Verifier — my own tool. Paste an IP that claims to be BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. and it runs the forward-confirmed reverse DNS check for you, plus names the real network owner if it’s a spoofer. This is the practical version of the two-step DNS check in the Scripts tab, without touching a terminal.
• robots.txt Tester — my own tool. Check whether a specific URL is allowed or blocked for the bingbot tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. specifically (as distinct from * or GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.), see exactly which rule wins, and catch a leftover msnbot directive fighting your Bingbot rule before it ships.
• Verify Bingbot tool — Microsoft’s own online IP lookup, useful as a cross-check against GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. Verifier.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Crawl Control — set Bingbot’s hourly crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. directly; the active control Google retired the equivalent of.
• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. (indexnow.org) — push created/updated/deleted URLs instead of waiting for Bingbot’s next pass; one ping reaches every participating engine.

Resources worth your time

My related writing

• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — where I cover Bing’s aggressive-crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. past, the IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. turnaround, and the 53%-wasted-traffic stat.
• When Should You Worry About Crawl Budget? — includes Bing’s “crawl efficiency” framing and the note that “Bing also has an Indexing APIThe Google Indexing API lets site owners notify Google directly when a URL is added, updated, or removed. Google officially supports it only for pages with JobPosting or BroadcastEvent (livestream) structured data — not for general content. that’s available to everyone.”
• Bingbot (Ahrefs SEO glossary) — the short reference version.
• The Beginner’s Guide to Technical SEO — where crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• How Bingbot Works: Discovering, Crawling, Extracting & Indexing (SEJ) — the detailed Fabrice Canel walkthrough behind the URL-memory and pre-filtering points above.
• Why & How Bing Plans To Improve Its Crawler, Bingbot (SEJ, 2019) — the Canel interview with the 100 billion new URLs/day figure, the polite crawl policy quote, and the “fascinating engineering problem” line.
• Microsoft Bing / Copilot: Use Schema For Its LLMs (Search Engine Land, 2025) — Canel at SMX Munich on schema markupSchema markup is code that uses the schema.org vocabulary to label what your content means so search engines can understand it and show rich results. It's most often written in JSON-LD, and it's not a direct ranking factor. helping Microsoft LLMsA large language model (LLM) is a deep-learning model trained on massive text corpora to predict the next token and generate human-like text. LLMs use the transformer architecture and power AI search features like Google's AI Overviews (Gemini) and Bing Copilot (GPT-4). and fresh contentContent freshness is how recent or up-to-date a page is — by its original publish date, its last substantive revision, or the currency of the facts inside it. It only helps rankings when the query itself benefits from recent results (Query Deserves Freshness), and cosmetic date changes with no real update don't count. as a Copilot ranking signal.
• The five infrastructure gates behind crawl, render, and index (Search Engine Land) — Canel’s “less is more for SEO” crawl efficiency framework, explained gate by gate.
• How Bingbot and Googlebot choose pages (Kalicube) — Canel’s “guided by key pages” quote about URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched. philosophy, with GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. comparison.
• r/TechSEO — the community for crawl/index debugging.

Stats worth citing

• ~100 billion new URLs discovered per day — Fabrice Canel’s 2019 figure for how many never-before-seen URLs BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. finds daily (“more than 100 billion new URLs never seen before”). A separate 2020 figure put it around 70 billion — keep the date attached when you cite either. Source
• ~53% of crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. traffic is wasted effort — my estimate, and the case for IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it.: pushing changes “can help cut that number down to almost nothing.” Source
• 28 published IPv4 prefixes — the size of Microsoft’s official Bingbot IP list (bingbot.json, generated January 2024) — useful context, but Microsoft warns the addresses change, so verify by reverse DNS instead. Source

Test yourself: Bingbot

Five quick questions on BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., verification, and reach. Pick an answer for each, then check.

Bingbot

Bingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.

Related: Crawler, Googlebot, User Agent, Crawling, IndexNow

Bingbot

Bingbot is the automated crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. Microsoft uses to discover and download web pages for the Bing indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. It launched on October 1, 2010, replacing the older MSNBot, and its robots.txt user-agent tokenA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is bingbot. Since adopting the Chromium-based Microsoft Edge engine, it’s an evergreen crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — it renders JavaScript the same way a current browser does, and its user-agent string tracks the latest stable Edge version.

The thing most people underrate is its reach. Bing’s own search-market share is small, but the Bing index that Bingbot builds also feeds Yahoo Search, DuckDuckGo, Ecosia, Edge’s default search, and Microsoft Copilot. So a robots.txt rule that blocks bingbot doesn’t just cost you Bing — it cascades across that whole ecosystem.

Because user-agent strings are trivially spoofed, you verify a request is genuinely Bingbot with a two-step reverse DNS check: the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. IP’s PTR record should resolve to a hostname ending in search.msn.com, and a forward lookup of that hostname must return the original IP. Microsoft publishes an IP list at bing.com/toolbox/bingbot.json but recommends reverse DNS over it, since the addresses can change.

To manage the crawl, Bing gives site owners Crawl Control in Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. (set an hourly crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.) and co-created IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. with Yandex — a push protocol that lets you notify the engine the moment content changes rather than waiting to be re-crawled.

Related: Crawler, Googlebot, User Agent, Crawling, IndexNow

Changelog