Bingbot

TL;DR — Bingbot is Microsoft Bing’s crawler — the bot that visits your pages, downloads them, and adds them to the Bing search index. It works a lot like Googlebot. Blocking it prevents Bing from crawling the affected URLs and can also affect products or partners that use Bing’s search infrastructure, so verify the impact against current Microsoft documentation.

What Bingbot is

When you see bingbot in your server logs, that’s Microsoft’s crawler visiting your site. Just like Googlebot does for Google, Bingbot follows links, reads sitemaps, downloads your pages, and hands them off to be added to Bing’s index — the database Bing pulls from when someone searches.

It launched back on October 1, 2010, taking over from an older bot called MSNBot. The name you’ll see in robots.txt and your logs is simply bingbot.

Why it matters more than you’d think

Here’s the part that surprises people. Bing’s slice of the search market is small, so it’s tempting to ignore Bingbot entirely. But the Bing index Bingbot builds doesn’t only feed Bing. It also powers:

• Yahoo Search
• DuckDuckGo
• Ecosia
• Microsoft Edge’s default search
• Microsoft Copilot

So if you block Bingbot, you’re not just disappearing from Bing — you’re cutting yourself off from all of those at once. That’s a much bigger deal than “3% market share” makes it sound.

Can Bingbot read JavaScript?

Yes. There used to be a worry that Bing couldn’t handle JavaScript-heavy sites, but Bingbot now runs on the same Chromium engine that powers Microsoft Edge, so it renders pages the way a modern browser does. If your content shows up in a current version of Edge or Chrome, Bingbot can generally see it too.

How to help (or stop) Bingbot

• To let it in: make sure your important pages are linked, submit a sitemap in Bing Webmaster Tools, and don’t accidentally block them in robots.txt.
• To slow it down: Bing Webmaster Tools has a feature called Crawl Control that lets you set how fast Bingbot crawls, hour by hour.
• To speed things up: there’s a “push” option called IndexNow that lets you tell Bing the instant a page changes, instead of waiting for the next crawl.

Want the technical version — the user-agent strings, how to verify a request is really Bingbot, the other Microsoft crawlers, and how Bingbot feeds Copilot? Switch to the Advanced tab.

TL;DR — Bingbot is Microsoft Bing’s crawler (robots.txt token: bingbot), live since October 1, 2010 when it replaced MSNBot. It’s evergreen on the Chromium-based Edge engine, so it renders modern JavaScript. Its real reach is broader than traffic attributed directly to bing.com because Microsoft uses its search infrastructure across products and partnerships. Verify it using Bing’s documented verification method rather than trusting the user-agent string. Manage the crawl with robots.txt, Crawl Control in Bing Webmaster Tools, and IndexNow — Bing’s push protocol, co-created with Yandex.

From MSNBot to Bingbot

Bingbot is Microsoft’s primary web crawler — the program that discovers, fetches, and processes pages to build and refresh the Bing index. It went live on October 1, 2010, replacing the legacy MSNBot. That transition came with a robots.txt rule that still matters if you have old directives lying around: Microsoft’s announcement said that “as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txt file, the BingBot directives will take precedence.” So MSNBot rules don’t quietly override your Bingbot rules — it’s the other way around.

Bingbot powers more than Bing

This is the single most important reframe on this page. Bing’s direct search-market share is small, which leads a lot of SEOs to skip it. But the index Bingbot builds also feeds Yahoo Search, DuckDuckGo, Ecosia, Edge’s default search, and Microsoft Copilot — and historically it powered ChatGPT Search too. The Yahoo dependency goes all the way back to the 2010 launch, when Microsoft warned site owners: “To keep your content in the Bing index, please ensure that BingBot has the same access to your site as given to MSNBot.”

The practical consequence: a Disallow against bingbot doesn’t cost you “just Bing.” It cascades across that whole ecosystem at once. Whether you care about Bing specifically or not, Bingbot is the gateway to a meaningful chunk of the searchable web — and increasingly to AI answers.

The user-agent string (and why it’s evergreen)

Bingbot’s user-agent contains the bingbot/2.0 token plus a Chromium-based Chrome/W.X.Y.Z version number, where W.X.Y.Z is substituted with the current stable Microsoft Edge build (Microsoft’s own example: 100.0.4896.127) — that substitution is what makes the string evergreen. Per Microsoft’s April 2022 announcement of the finalized strings, the desktop UA is Mozilla/5.0 AppleWebKit/537.36 (KHTML, like Gecko; compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm) Chrome/W.X.Y.Z Safari/537.36 and the mobile UA is Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/W.X.Y.Z Mobile Safari/537.36 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm). Microsoft’s earlier, December 2019 preview of these strings had also proposed an extra trailing Edg/W.X.Y.Z fragment, but that fragment isn’t in the final strings Microsoft published in 2022 and confirmed fully live in February 2023 — if you’re matching bingbot’s UA in code, don’t require it. The older legacy string was the bare Mozilla/5.0 (compatible; bingbot/2.0; +http://www.bing.com/bingbot.htm).

“Evergreen” means exactly that the version number tracks the latest stable Edge automatically. Practically, this killed the old two-tier crawl/render gap: because Bingbot runs the same Chromium engine family as modern Edge and Chrome, it renders JavaScript the way a current browser does. Bing also treats good-faith dynamic rendering as a legitimate option for heavy JS sites — it isn’t cloaking in their view.

Verifying Bingbot — the user-agent isn’t enough

User-agent strings are trivially spoofed, so a request claiming to be Bingbot proves nothing on its own. The reliable check is two-step reverse DNS:

1. Reverse (PTR) lookup the crawling IP from your logs. A genuine Bingbot IP resolves to a hostname ending in search.msn.com — the pattern is msnbot-<ip-with-dashes>.search.msn.com (so 157.55.39.1 → msnbot-157-55-39-1.search.msn.com).
2. Forward (A) lookup that hostname. It must resolve back to the original IP.

If either step fails, it isn’t Bingbot. Microsoft does publish an IP list at bing.com/toolbox/bingbot.json (28 IPv4 prefixes, last generated January 2024), but the company’s own guidance is to prefer reverse DNS over the IP list, because the addresses can change at any time. Treat the JSON as a rough reference, not a firewall allow-list. (The same reverse-DNS-over-IP-list logic applies to Googlebot — see the verification scripts in the Scripts tab.)

How Bingbot discovers and crawls

Discovery is link-driven. Fabrice Canel, Microsoft’s longtime Bingbot lead, describes the approach as following importance: “We are guided by key pages that are important on the internet and we follow links to understand what’s next.” The scale is enormous — in a 2019 interview Canel said “Every day, my team discovers more than 100 billion new URLs never seen before, while useless URLs parameters are ignored.” (A 2020 figure put new-URL discovery at around 70 billion per day; both are real numbers from different dates, so it’s worth keeping the year attached when you cite either one.)

Two behaviors set Bingbot apart from how Google describes its own crawler:

• Pre-filtering before fetch. Bingbot makes a cost/value judgment before it even downloads a URL. As Bing has put it, “Pages that are deemed to have absolutely no potential for being useful in satisfying a user’s search query in Bing results are not retained.”
• URL memory. Bingbot remembers URLs essentially forever: “Bingbot retains every URL in memory and comes back and recrawls intermittently, even if all links to it have been removed.” That’s why a page can still see Bingbot hits long after you’ve stripped every internal link to it.

Underneath all of it is a politeness-first posture. Canel: “Our default crawl policy is to be as polite as possible when crawling the web.” And his candid take on the problem itself: “Crawling right is a fascinating engineering problem that hasn’t fully been solved yet.”

Other Microsoft crawlers

Bingbot isn’t the only Microsoft bot you’ll see:

• AdIdxBot (adidxbot) — the Bing Ads quality-control crawler. The catch worth knowing: it shares crawl budget with Bingbot, so heavy ad crawling can eat into the budget for your organic pages.
• BingPreview — generates page snapshots and preview thumbnails.
• MicrosoftPreview — link previews in Teams and Office.
• BingVideoPreview — video previews shown only in Bing. Like the others it has desktop and mobile variants, and it identifies itself against +https://aka.ms/microsoftbots rather than the bingbot.htm URL the older crawlers use.
• MSNBot / MSNBot-Media — the legacy crawler, retired October 1, 2010.

Controlling Bingbot’s crawl

Three levers, from blunt to surgical:

• robots.txt — the bingbot token controls crawling. Use it to keep the bot out of low-value spaces; it isn’t a deindexing tool.
• Crawl Control (Bing Webmaster Tools) — lets you set Bingbot’s hourly crawl rate, so you can slow it during peak traffic and let it run faster off-peak. This is a genuine Google/Bing divergence: Google retired its crawl-rate slider in Search Console, while Bing still gives you an active control.
• IndexNow — instead of waiting to be re-crawled, you push the change. It’s a protocol “created by Microsoft Bing and Yandex, allowing websites to easily notify search engines whenever their website content is created, updated, or deleted.” One ping reaches every participating engine. I won’t duplicate the full treatment here — see the IndexNow deep dive for setup, the key file, and the 10,000-URL-per-request limit.

The strategic point I keep coming back to: Bing fixed a reputation for aggressive crawling by getting efficient, and IndexNow is the centerpiece of that. As I’ve written in my analysis of AI and search engine bot crawling, “It’s estimated that 53% of crawler traffic is wasted effort, and IndexNow can help cut that number down to almost nothing.” Pushing changes beats trying to coax more exploratory crawling.

Bingbot, Copilot, and the AI opt-out gap

Because the Bing index feeds Microsoft Copilot, Bingbot is effectively an AI crawler too. Canel’s guidance for showing up well there is concrete: “schema markup helps Microsoft’s LLMs understand your content,” and “Gen AIs value fresh content in particular, partly as a reference check of their LLM training data. Use the API at indexnow.org to push that information as it’s published or updated.”

There’s a wrinkle competitor articles tend to skip. Google offers a Google-Extended token that lets you opt out of AI training without affecting search indexing. By most third-party reads there’s no equivalent separate opt-out for Bing — meaning blocking bingbot to stay out of Copilot training would also drop you from Bing search and everything downstream of it. I’d treat that as an observation about how the controls currently line up rather than a settled Microsoft policy, and check the current documentation before acting on it.

For where Bingbot sits in the bigger picture — the crawler family, Googlebot, the user-agent concept, AI crawlers, robots.txt, and IndexNow — the crawling hub ties it all together.

AI summary

A condensed take on the Advanced version:

• Bingbot = Microsoft Bing’s crawler. robots.txt token bingbot; live since October 1, 2010 when it replaced MSNBot (and BingBot directives take precedence over leftover MSNBot ones).
• Reach > market share. The index Bingbot builds also powers Yahoo, DuckDuckGo, Ecosia, Edge default search, and Copilot — so blocking bingbot cascades far beyond Bing.
• Evergreen rendering. Runs on the Chromium-based Edge engine; UA carries the bingbot/2.0 token plus a Chrome/W.X.Y.Z version number substituted with the current stable Edge build (no separate Edg/ fragment in the final 2022 strings). It renders modern JavaScript; good-faith dynamic rendering isn’t cloaking.
• Verify by two-step reverse DNS: PTR to msnbot-*.search.msn.com, then a forward lookup that matches the IP. User-agents are spoofable; prefer reverse DNS over the bingbot.json IP list (which changes).
• How it crawls: follows “key pages” and links, discovers ~100 billion new URLs a day (2019; ~70 billion in a 2020 figure), pre-filters URLs with “no potential” before fetching, and keeps every URL in memory — recrawling even after all links are removed.
• Other Microsoft bots: AdIdxBot (shares Bingbot’s crawl budget), BingPreview, MicrosoftPreview, BingVideoPreview; MSNBot is retired.
• Control it with robots.txt, Crawl Control (hourly rate — Google removed its equivalent), and IndexNow (push, co-created with Yandex). Patrick’s framing: ~53% of crawler traffic is wasted, and IndexNow cuts that toward nothing.
• AI angle: Bingbot feeds Copilot; schema and fresh content help; there’s no clear Bing equivalent to Google-Extended, so blocking for AI training also drops search (treat as observation, verify current docs).

Official documentation

Primary-source documentation, mostly from Microsoft/Bing, with a Google contrast link. Heads up: many blogs.bing.com posts and the bing.com/webmasters/help pages are JavaScript-rendered or redirect, so confirm details in a browser if a link bounces.

Bing / Microsoft

• Which crawlers does Bing use? — the official list of Microsoft bots and their user-agent tokens (Bingbot, AdIdxBot, and the preview crawlers).
• Announcing user-agent change for Bing crawler bingbot — Microsoft’s own finalized desktop and mobile UA strings (confirmed fully live February 2023); this is the source for the exact strings quoted above.
• Bingbot IP list (bingbot.json) — the published IPv4 prefixes. Use as a reference only; Microsoft recommends reverse DNS instead.
• Verify Bingbot tool — Microsoft’s online lookup for confirming a crawling IP is genuinely Bingbot.
• Bing Webmaster Tools — Crawl Control — set Bingbot’s crawl rate by hour.
• IndexNow / indexnow.org — the push protocol for instantly signaling created, updated, or deleted URLs.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s “crawl efficiency” philosophy.

Google (for contrast)

• Overview of Google crawlers and fetchers — Googlebot’s user-agents and IP ranges, the parallel to Bing’s setup. Note the divergences: Google retired its manual crawl-rate control, doesn’t use IndexNow for general pages, and offers Google-Extended to opt out of AI training separately from search.

Quotes from the source

On-the-record statements about Bingbot — mostly from Fabrice Canel, Microsoft’s Bingbot lead. Each link is a deep link that jumps to the quoted passage.

Fabrice Canel, Microsoft Bing — discovery and scale

• “We are guided by key pages that are important on the internet and we follow links to understand what’s next.” Jump to quote
• “Every day, my team discovers more than 100 billion new URLs never seen before, while useless URLs parameters are ignored.” (Search Engine Journal interview, 2019.) Jump to quote
• “Crawling right is a fascinating engineering problem that hasn’t fully been solved yet.” (SEJ, 2019.) Jump to quote
• “Our default crawl policy is to be as polite as possible when crawling the web.” (SEJ, 2019.) Jump to quote

Microsoft Bing — how Bingbot handles URLs

• “Bingbot retains every URL in memory and comes back and recrawls intermittently, even if all links to it have been removed.” Jump to quote
• “Pages that are deemed to have absolutely no potential for being useful in satisfying a user’s search query in Bing results are not retained.” Jump to quote

Fabrice Canel, Microsoft Bing — efficiency, schema, and AI

• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” Jump to quote
• “schema markup helps Microsoft’s LLMs understand your content.” (SMX Munich, 2025.) Jump to quote
• “Gen AIs value fresh content in particular, partly as a reference check of their LLM training data. Use the API at indexnow.org to push that information as it’s published or updated.” Jump to quote

Microsoft Bing — the 2010 launch and IndexNow

• “as of October 1st, if we detect separate sets of custom directives for both MSNBot and BingBot in your robots.txt file, the BingBot directives will take precedence.” (Bing Webmaster blog, 2010.)
• “To keep your content in the Bing index, please ensure that BingBot has the same access to your site as given to MSNBot.” (Bing Webmaster blog, 2010.)
• “IndexNow is a new protocol created by Microsoft Bing and Yandex, allowing websites to easily notify search engines whenever their website content is created, updated, or deleted.” (Bing Webmaster blog, 2021.)

Patrick Stox (me) — Bing’s efficiency turnaround

• “There was a lot of blowback on Bing in the past because of their aggressive crawling.” Jump to quote
• “It’s estimated that 53% of crawler traffic is wasted effort, and IndexNow can help cut that number down to almost nothing.” Jump to quote

What should I do about Bingbot?

Bingbot decisions branch on two things: whether a request is genuinely Bingbot, and whether you want to slow it down, speed it up, or block it outright. Click through it.

Bingbot & Microsoft crawlers — cheat sheet

User-agent strings

W.X.Y.Z = the current stable Microsoft Edge version, substituted into the Chrome/ token (evergreen — it updates itself). These are the final strings Microsoft published in April 2022 and confirmed fully live in February 2023; an earlier December 2019 preview had also proposed a trailing Edg/W.X.Y.Z fragment, but it isn’t part of the finalized UA — don’t require it in bot-matching code.

The Microsoft crawler family

Fast facts

• Launched October 1, 2010, replacing MSNBot.
• If both MSNBot and BingBot directives exist in robots.txt, BingBot wins.
• Verify via reverse DNS to *.search.msn.com, not the user-agent and not the IP list (IPs change).
• IP list: bing.com/toolbox/bingbot.json — 28 IPv4 prefixes, generated Jan 2024.
• Crawl Control in Bing Webmaster Tools sets the hourly crawl rate (Google removed its equivalent).
• IndexNow = push changed URLs; co-created with Yandex.

Verify a bot is really Bingbot

A request that says it’s Bingbot proves nothing — the user-agent string is trivially spoofed. As Microsoft puts it, the fact that a bot uses the exact Bingbot user-agent doesn’t make it Bingbot. Confirm it with a two-step reverse + forward DNS check. A genuine Bingbot IP reverse-resolves to a hostname ending in search.msn.com (pattern msnbot-<ip-with-dashes>.search.msn.com), and that hostname must forward-resolve back to the same IP.

macOS / Linux

# 1) Reverse DNS the IP from your logs — it should end in search.msn.com
host 157.55.39.1
# → 1.39.55.157.in-addr.arpa domain name pointer msnbot-157-55-39-1.search.msn.com

# 2) Forward DNS that hostname back — it must resolve to the same IP
host msnbot-157-55-39-1.search.msn.com
# → msnbot-157-55-39-1.search.msn.com has address 157.55.39.1

Windows

nslookup 157.55.39.1
nslookup msnbot-157-55-39-1.search.msn.com

If the reverse lookup doesn’t end in search.msn.com, or the forward lookup doesn’t return the original IP, it isn’t Bingbot — drop or rate-limit it. You can also use Microsoft’s online Verify Bingbot tool, and the published bingbot.json IP list as a rough cross-check — but Microsoft is explicit that the IPs can change, so DNS is the source of truth, not a hardcoded allow-list. (Same approach as verifying Googlebot; see the crawling hub’s Scripts tab for the Google version.)

Bingbot readiness checklist

A quick pass to confirm you’re working with Bingbot, not against it:

• robots.txt doesn’t block bingbot from anything you want in Bing (remember: that also gates Yahoo, DuckDuckGo, Ecosia, and Copilot).
• No stale MSNBot directives fighting your Bingbot rules (BingBot wins, but clean them up).
• XML sitemap submitted in Bing Webmaster Tools, listing only canonical, indexable URLs with accurate lastmod.
• Bot verification uses reverse + forward DNS, not the user-agent or a hardcoded IP list.
• If you firewall or allow-list bots, you’re verifying by DNS — not pinning the bingbot.json IPs (they change).
• Crawl Control reviewed if Bingbot traffic strains your server at peak hours.
• IndexNow wired up so created/updated/deleted URLs are pushed, not waited on.
• JS-dependent content is reachable and renders (Bingbot is evergreen, but real <a href> links still beat click-only navigation).
• Aware that AdIdxBot shares Bingbot’s crawl budget if you run Bing Ads.
• Before blocking bingbot for AI reasons, confirmed there’s no separate Copilot opt-out — blocking it currently drops you from Bing search too.

The mental models

1. Reach, not market share. Bing’s direct search share is small, so it’s tempting to judge Bingbot by that number alone. The reframe that matters: the index Bingbot builds is the backend for Yahoo Search, DuckDuckGo, Ecosia, Edge’s default search, and Microsoft Copilot. A Disallow: bingbot rule doesn’t cost you “3% of search” — it costs you every one of those surfaces at once. Judge Bingbot decisions by ecosystem reach, not by Bing’s standalone number.

2. Verify by DNS, not by user-agent. A request’s user-agent string is just a header the client sets — anyone can type bingbot/2.0 into a curl command. The only claim that can’t be faked cheaply is a DNS record pointing back to Microsoft’s own search.msn.com domain, checked both directions (reverse then forward). Treat every other signal — user-agent, the published IP list — as a hint, not proof.

3. Three levers, blunt to surgical. robots.txt is the blunt instrument: it keeps Bingbot out of a path entirely, but it’s a crawling control, not a deindexing one. Crawl Control in Bing Webmaster Tools is the dial: it shapes when Bingbot hits you hardest, hour by hour, without shutting anything out. IndexNow is the scalpel: it doesn’t wait for Bingbot to come back around at all — it pushes a single URL the moment it changes. Reach for the least blunt lever that solves the actual problem.

4. Google retired the dial; Bing kept it. Google removed its crawl-rate limiter from Search Console. Bing’s Crawl Control still gives you an active hourly rate control. If you’re used to treating crawl-rate management as “a thing you can’t really do anymore,” that’s true for Googlebot but not for Bingbot — it’s a real, current lever worth using when Bingbot traffic strains your server at peak hours.

The standing KPI: total Bing-ecosystem referral traffic

Bing’s own market-share number understates what Bingbot is worth to you, because the index it builds also feeds Yahoo Search, DuckDuckGo, Ecosia, and Microsoft Copilot. The honest KPI isn’t “Bing traffic” in isolation — it’s the combined referral traffic across that whole ecosystem, tracked over time.

Segment referral traffic by the full Bing-index family

• Metric — Sessions/clicks in your analytics tool where the referrer is bing.com, search.yahoo.com, duckduckgo.com, ecosia.org, or copilot.microsoft.com, summed and tracked as one group alongside (not instead of) your Google number.
• What it tells you — Whether blocking or under-serving Bingbot is actually costing you traffic beyond Bing itself. A dip that shows up across all five referrers at once is a Bingbot access problem, not a one-engine ranking problem.
• How to pull it — Segment your existing analytics tool (GA4, server logs, or whatever you already run) by referrer domain for each of the five above, and pair it with Bing Webmaster Tools — Search Performance, which reports Bing’s own clicks and impressions directly (it can’t see Yahoo/DuckDuckGo/Ecosia downstream numbers, so analytics referrer data is the only place that combined view exists).
• Benchmark / realistic range — There’s no honest industry-wide number to target here — the ecosystem split varies enormously by audience and geography. Establish your own baseline from a few months of referrer data before you judge any single month’s move.
• Cadence — Monthly is enough for the trend; check weekly right after a robots.txt change, an IndexNow rollout, or anything else that touches Bingbot’s access.

Tools for verifying and controlling Bingbot

• Googlebot Verifier — my own tool. Paste an IP that claims to be Bingbot and it runs the forward-confirmed reverse DNS check for you, plus names the real network owner if it’s a spoofer. This is the practical version of the two-step DNS check in the Scripts tab, without touching a terminal.
• robots.txt Tester — my own tool. Check whether a specific URL is allowed or blocked for the bingbot token specifically (as distinct from * or Googlebot), see exactly which rule wins, and catch a leftover msnbot directive fighting your Bingbot rule before it ships.
• Verify Bingbot tool — Microsoft’s own online IP lookup, useful as a cross-check against Googlebot Verifier.
• Bing Webmaster Tools — Crawl Control — set Bingbot’s hourly crawl rate directly; the active control Google retired the equivalent of.
• IndexNow (indexnow.org) — push created/updated/deleted URLs instead of waiting for Bingbot’s next pass; one ping reaches every participating engine.

Resources worth your time

My related writing

• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — where I cover Bing’s aggressive-crawling past, the IndexNow turnaround, and the 53%-wasted-traffic stat.
• When Should You Worry About Crawl Budget? — includes Bing’s “crawl efficiency” framing and the note that “Bing also has an Indexing API that’s available to everyone.”
• Bingbot (Ahrefs SEO glossary) — the short reference version.
• The Beginner’s Guide to Technical SEO — where crawlers fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• How Bingbot Works: Discovering, Crawling, Extracting & Indexing (SEJ) — the detailed Fabrice Canel walkthrough behind the URL-memory and pre-filtering points above.
• Why & How Bing Plans To Improve Its Crawler, Bingbot (SEJ, 2019) — the Canel interview with the 100 billion new URLs/day figure, the polite crawl policy quote, and the “fascinating engineering problem” line.
• Microsoft Bing / Copilot: Use Schema For Its LLMs (Search Engine Land, 2025) — Canel at SMX Munich on schema markup helping Microsoft LLMs and fresh content as a Copilot ranking signal.
• The five infrastructure gates behind crawl, render, and index (Search Engine Land) — Canel’s “less is more for SEO” crawl efficiency framework, explained gate by gate.
• How Bingbot and Googlebot choose pages (Kalicube) — Canel’s “guided by key pages” quote about URL discovery philosophy, with Googlebot comparison.
• r/TechSEO — the community for crawl/index debugging.

Stats worth citing

• ~100 billion new URLs discovered per day — Fabrice Canel’s 2019 figure for how many never-before-seen URLs Bingbot finds daily (“more than 100 billion new URLs never seen before”). A separate 2020 figure put it around 70 billion — keep the date attached when you cite either. Source
• ~53% of crawler traffic is wasted effort — my estimate, and the case for IndexNow: pushing changes “can help cut that number down to almost nothing.” Source
• 28 published IPv4 prefixes — the size of Microsoft’s official Bingbot IP list (bingbot.json, generated January 2024) — useful context, but Microsoft warns the addresses change, so verify by reverse DNS instead. Source

Test yourself: Bingbot

Five quick questions on Bingbot, verification, and reach. Pick an answer for each, then check.

Bingbot

Bingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.

Related: Crawler, Googlebot, User Agent, Crawling, IndexNow

Bingbot

Bingbot is the automated crawler Microsoft uses to discover and download web pages for the Bing index. It launched on October 1, 2010, replacing the older MSNBot, and its robots.txt user-agent token is bingbot. Since adopting the Chromium-based Microsoft Edge engine, it’s an evergreen crawler — it renders JavaScript the same way a current browser does, and its user-agent string tracks the latest stable Edge version.

The thing most people underrate is its reach. Bing’s own search-market share is small, but the Bing index that Bingbot builds also feeds Yahoo Search, DuckDuckGo, Ecosia, Edge’s default search, and Microsoft Copilot. So a robots.txt rule that blocks bingbot doesn’t just cost you Bing — it cascades across that whole ecosystem.

Because user-agent strings are trivially spoofed, you verify a request is genuinely Bingbot with a two-step reverse DNS check: the crawling IP’s PTR record should resolve to a hostname ending in search.msn.com, and a forward lookup of that hostname must return the original IP. Microsoft publishes an IP list at bing.com/toolbox/bingbot.json but recommends reverse DNS over it, since the addresses can change.

To manage the crawl, Bing gives site owners Crawl Control in Bing Webmaster Tools (set an hourly crawl rate) and co-created IndexNow with Yandex — a push protocol that lets you notify the engine the moment content changes rather than waiting to be re-crawled.

Related: Crawler, Googlebot, User Agent, Crawling, IndexNow

Changelog