Baiduspider

TL;DR — BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). is Baidu’s web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the bot that visits your pages, downloads them for Baidu Search. If mainland China is an important market, Baiduspider belongs in your crawl monitoring. Check Baidu’s current webmaster documentation before relying on engine-specific behavior, especially for JavaScript renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. and crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. controls.

What Baiduspider is

Google doesn’t work inside mainland China. Baidu does — it’s the search engine most people there use, and Baiduspider is its crawler. When Baiduspider visits your site, it follows links, reads your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., downloads your pages, and hands them off to build Baidu’s indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. — the database Baidu pulls from when someone searches. If Baiduspider can’t reach and read a page, that page won’t show up in Baidu.

In Baidu’s own words (from its official English FAQ): “Baiduspider is Baidu search engine program which is used to visit pages on the internet and build information into Baidu index. This enables users to locate your site when they perform a search.” That’s the same job GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. does for Google and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. does for Bing.

Why it might matter to you

Baiduspider only really matters if you want to be foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. inside mainland China. Baidu still leads there while Google barely registers — so a China-facing site (or a global site with a China audience) needs Baiduspider the same way a Western site needs GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. If you don’t serve customers in China at all, Baiduspider is just another bot hitting your server, and some sites choose to block it to save resources.

The catch: Baiduspider is bad at JavaScript

This is the big one. Googlebot runs a modern version of Chrome and reads most JavaScript-heavy sites fine. Baiduspider does not — it’s widely reported to be poor at running JavaScript. If your important content or links only appear after JavaScript runs, Baidu may never see them, even if the same page ranks fine in Google. The safe move for a China-facing site: make sure your real content is in the plain HTML that comes back before any JavaScript runs.

How to help (or stop) Baiduspider

• To let it in: link to your important pages, submit a sitemapThe Google Search Console report where you submit sitemaps and watch how Google processes them — type, last read date, status, and how many URLs were discovered. It confirms Google read your list; it doesn't prove anything got indexed. in Baidu Search Resource Platform (Baidu’s version of Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.), and don’t accidentally block those pages in robots.txt.
• To keep it out of parts of your site: use robots.txt — Baidu says it follows the standard rules there.
• To slow it down: Baidu Search Resource Platform has a crawl-frequency area where you can see and cap how much Baiduspider crawls.

Want the technical version — the exact user-agent strings, how to verify a real Baiduspider, the robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. quirks, and the crawl-frequency controls? Switch to the Advanced tab.

TL;DR — BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). is Baidu’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the Chinese-market GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. It runs named variants sharing the Baiduspider tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. family (general PC/mobile, plus -image, -video, -news, -favo, -cpro, -ads), and a Baiduspider-render JavaScript variant that Baidu documents only in Chinese. Verify it by bidirectional DNS to *.baidu.com/*.baidu.jp — Baidu’s own FAQ warns about impersonators. Baidu claims it “strictly complies with robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. protocol,” with a named exception: Baiduspider-cpro and Baiduspider-ads run under commercial agreements and may follow different access rules; it also doesn’t document the non-standard crawl-delay as a reliable control. Because renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. behavior can change, server-rendered HTML is the conservative default for critical content. Verify current controls in Baidu Search Resource PlatformBaidu Webmaster Tools — officially the Baidu Search Resource Platform (百度搜索资源平台) at ziyuan.baidu.com — is Baidu's free webmaster console: verify a site, submit URLs and sitemaps (including the fast API push), and monitor indexing and crawl diagnostics. Since May 2022 registration needs a Chinese mobile number. (Ziyuan).

What Baiduspider actually is

Baiduspider (百度蜘蛛) is the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. operated by Baidu, the search engine that dominates inside mainland China, where Google is largely inaccessible behind the Great Firewall. Its job is the standard crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline: discover URLs, fetch pages, and build the index Baidu Search draws from. Baidu’s official English FAQ, “FAQs of Baiduspider” on help.baidu.com, is the rare piece of primary English-language documentation here, and it’s the source for most of the direct quotes below.

The asymmetry that frames everything: Baidu matters domestically, not globally. StatCounter’s June 2026 China snapshot put Baidu at 47.44%, Bing at 23.24%, Haosou at 14.02%, Sogou at 2.62%, and Google at just 2.28% (StatCounter, Search Engine Market Share China). Treat that as a snapshot, not a constant — StatCounter’s China numbers swing meaningfully month to month depending on device mix and methodology (Baidu has been reported anywhere from roughly 40% to 65% across different 2025–2026 snapshots), so re-check the live figure rather than quoting one month forever. The takeaway holds regardless: Baidu leads inside China while Google leads the rest of the world, which is why Baiduspider optimization is treated as its own international-SEO specialty rather than a GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. afterthought. Our market-specific SEO hub covers where Baidu fits alongside Yandex, Naver, and the other regional engines.

User-agent strings and crawler variants

Baidu’s English FAQ names the crawler variants directly. Reproduced verbatim (and yes, Baidu’s own copy repeats “Baiduspider” for both PC and mobile rather than giving distinct strings):

Name of Products | User-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. PC search | Baiduspider Mobile search | Baiduspider Image search | Baiduspider-image Video search | Baiduspider-video News search | Baiduspider-news Baidu bookmark | Baiduspider-favo Union baidu | Baiduspider-cpro Business search | Baiduspider-ads other search | Baiduspider

So the officially named variants in the English FAQ are Baiduspider-image, -video, -news, -favo (bookmark), -cpro (the “union”/ad-network crawler), and -ads (business search) — all sharing the Baiduspider family token. The literal general PC user-agent string, given on Baidu’s own site, is:

Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html)

The mobile UA embeds Baiduspider/2.0 inside an Android WebKit string. Every declared variant points to the same http://www.baidu.com/search/spider.html reference URL — the Baidu equivalent of Googlebot’s google.com/bot.html.

Baiduspider-render — the JS variant Baidu only documents in Chinese

Here’s a subtlety worth getting right, because a lot of write-ups get it wrong: the English FAQ table above does not list a “Baiduspider-render” variant, which has led some guides to call it merely third-party folklore. It isn’t. Baidu documents Baiduspider-render officially — just on its Chinese-language Ziyuan Academy page, “【官方说法】只需两步，正确识别百度蜘蛛（User-Agent）” (“[Official statement] Two steps to correctly identify Baiduspider”). That page splits UAs into three channels — 移动 (mobile), PC, and 小程序 (mini-program) — and lists an alternate render UA alongside the plain one:

Mozilla/5.0 (compatible; Baiduspider-render/2.0; +http://www.baidu.com/search/spider.html)

with a matching mobile string and a mini-program variant (Baiduspider-render/2.0;Smartapp). So Baiduspider-render is officially documented — just Chinese-only, absent from the English FAQ. What Baidu does not publish is a spec for what “-render” actually does differently; unlike Google, Baidu doesn’t document its JavaScript renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. capabilities. That gap is where the JS-rendering caveat below comes from.

How to verify Baiduspider (and not a spoofed bot)

The user-agent string is trivially spoofed — anyone can send a request that says Baiduspider. Baidu’s own FAQ warns about exactly this, describing “spammers or other trouble makers who pretend to be Baiduspider,” and its guidance is to verify with DNS, not the string. From the FAQ, verbatim:

We recommend using reverse DNS lookup to verify Baiduspider. Verification methods are different under linux/windows/os environments.

It gives per-OS commands — host on Linux, nslookup on Windows, dig -x on macOS — with worked examples, and the rule that matters:

The hostname of Baiduspider is *.baidu.com or *.baidu.jp. Others are fake hostnames.

The two *.baidu.com and *.baidu.jp domains are the quirk to remember — Baidu crawls from both a .com and a .jp reverse-DNS namespace, so a legitimate Baiduspider hostname can end in either. (One of Baidu’s own worked examples even resolves to BaiduMobaider-119-63-195-254.crawl.baidu.jp — note “BaiduMobaider,” not “Baiduspider,” in that hostname; that’s a real quirk in Baidu’s documentation, not a typo to silently correct.)

Baidu’s English FAQ spells out only the reverse-lookup half in its prose, but its Chinese Ziyuan Academy page is more explicit: it names the whole process “双向DNS解析认证” (bidirectional DNS resolution authentication) and breaks it into “第一步：DNS反查IP” (Step 1: reverse-lookup the IP) and “第二步：对域名运行正向DNS查找” (Step 2: run a forward DNS lookup on the hostname). That’s the complete, correct method — reverse-lookup the IP to confirm a *.baidu.com/*.baidu.jp hostname, then forward-lookup that hostname and confirm it resolves back to the same IP. It’s the same two-step approach you’d use to verify GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. or BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. — the commands are in the Scripts tab.

Baiduspider and robots.txt

Baidu makes an unusually strong, quotable compliance claim in its own FAQ:

Baidu strictly complies with robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. protocol.

And it supports the standard controls you’d expect. Per Baidu’s Ziyuan Academy material, Baiduspider supports wildcard matching with * and $ in robots.txt paths (parity with Google and Bing), and does exact, case-sensitive path matching. You can set per-user-agent rules — Baidu’s FAQ gives worked examples, including this one that blocks everything for general Baiduspider but allows the image crawler into /image/:

User-agent: Baiduspider
Disallow: /
User-agent: Baiduspider-image
Allow: /image/

(Baidu also notes that Baiduspider-video doesn’t currently support these rules.)

The named exception to “strict compliance”

The interesting tension: Baidu claims strict compliance but documents a named, official exception. Baiduspider-cpro (the ad-network crawler) and Baiduspider-ads (business search) operate under separate commercial agreements and ignore robots.txt by design. Verbatim from the FAQ: “Baiduspider-cpro will not work on the records set by robots.txt” and the same for -ads. So the honest framing isn’t the vague “Baidu doesn’t always respect robots.txt” you’ll see on aggregator sites — it’s that Baidu’s standard crawlers comply, with two contractually exempt ad crawlers as the one concrete, documented carve-out.

A separate, narrower point that’s easy to conflate: Baiduspider does not honor the non-standard crawl-delay directive. Search Engine Journal’s Abby Hamilton documented this in The Modern Guide To Robots.txt (Nov 2024) — her crawler comparison table lists “Baidu | Baiduspider | No” for crawl-delay support (Yandex’s YandexBotYandexBot is the main web crawler operated by Yandex, Russia's dominant search engine. It discovers and fetches pages for Yandex Search, is one of a family of Yandex robots, and supports a Yandex-only robots.txt directive (Clean-param) that Google and Bing don't have., by contrast, is a “Yes”). That’s a different claim from standard Allow/Disallow compliance, and worth keeping straight so you don’t accidentally contradict Baidu’s own robots.txt claim.

A real example: baidu.com’s own robots.txt

Baidu practices robots.txt segmentation on its own domain. Its live robots.txt gives its own Baiduspider a shorter disallow list than it gives Googlebot, MSNBot, Sogou, and Youdao (which additionally get /shifen/, /homepage/, and /cpro blocked), and blocks everything (Disallow: /) for any unnamed user-agent by default:

User-agent: Baiduspider
Disallow: /baidu
Disallow: /s?
Disallow: /ulink?
Disallow: /link?
Disallow: /home/news/data/
Disallow: /bh

One operational detail from the FAQ worth knowing when you’re editing rules: if you tighten robots.txt after Baidu has already indexed a site, “it usually takes 48 hours for the updated robots.txt to take effect.”

Does Baiduspider render JavaScript?

This is Baiduspider’s biggest practical limitation relative to Googlebot, and it’s worth being precise about what’s official and what isn’t. Baidu’s English FAQ doesn’t address JavaScript rendering at all — there’s no capability spec the way Google publishes one for Googlebot. What we have instead is consistent industry testing. Dan Taylor put it bluntly in Search Engine Journal’s Google vs Baidu comparison (Mar 2021): “Baidu is notoriously bad when it comes to crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. JavaScript, so make sure that all your important content and links are served in plain HTML on both the mobile and desktop versions of your website.”

That aligns with the position this site already takes. Our contentful SEOContentful SEO is the practice of making content delivered through the Contentful headless CMS crawlable, indexable, and rankable. Because Contentful serves content over APIs and renders no HTML itself, every SEO outcome is decided by the frontend that consumes it. coverage notes that Bing, Yandex, and Baidu “may not index [client-side-rendered JS] at all,” and that dynamic rendering is deprecated as a general Google technique. Put the three big crawlers on a spectrum: Googlebot does full evergreen-Chromium rendering, BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. does solid Chromium/Edge rendering, and Baiduspider is the weakest of the three at JavaScript — with plain, server-rendered HTML being the universally safe fallback across all of them.

Practical guidance for a China-facing site: default to server-side rendering or static HTML so critical content, links, and metadata exist in the initial response before any JavaScript runs. Treat client-side-only content as an indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. risk for Baidu specifically, independent of whether it ranks fine in Google. Frame SSR as an industry-consensus best practice here — not as an official Baidu recommendation, because no primary Baidu statement on JS rendering exists to cite.

Controlling Baiduspider’s crawl frequency

Baidu presents crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. as algorithmically self-adjusted, not something you set directly. From the FAQ, verbatim:

In order to ensure the search results cover most of your pages, Baiduspider must keep the crawling at a certain level. We have been trying our best to avoid increasing the loading to your servers, and to adjust the frequency based on combined factors, such as your server’s capability, your site’s quality and the update frequency of your site.

That’s the same server-capability-plus-demand logic Google and Bing describe. The FAQ names a feedback form (webmaster.baidu.com/feedback/index) as the official lever for reporting over-crawling — “If you find any unreasonable access from Baiduspider, please inform us.”

At the product level, Baidu Search Resource Platform (百度搜索资源平台, at ziyuan.baidu.com — the console historically called “Baidu Webmaster Tools”) is Baidu’s equivalent of Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility.. Its manual index confirms a “抓取频次” (Crawl Frequency) product area where site owners can view and cap Baiduspider’s crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. directly, alongside robots and crawl-diagnostics tools. Treat the feedback form as the official escalation path in Baidu’s English docs and the Ziyuan Crawl Frequency dashboard as the practical, UI-level control. (The exact in-product menu path sits behind a logged-in Ziyuan account, so confirm the current UI there before quoting specific steps.)

hreflang and other Google-standard signals

Signals that are standard for Google don’t all carry over to Baidu — and this is an industry/practitioner finding, not something either company documents. Google’s own localized-versions documentation is scoped entirely to Google Search and makes no claim about whether Baidu, Bing, or Yandex honor hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others.. On this site, our hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. guide states it plainly: Baidu doesn’t support hreflang at all — the fallback is to set an accurate content-language and <html lang> so Baidu can at least read the page’s language. That’s additive to my own Google-scoped hreflang work (the Ahrefs hreflang tags guide and the study of 374,756 domains, both of which discuss Google’s implementation only) rather than a contradiction of it.

No English rep culture — why the FAQ is unusually valuable

One thing that genuinely sets the Baidu ecosystem apart: unlike Google (Gary Illyes, John Mueller), Bing (Fabrice Canel), or Yandex, Baidu maintains no visible English-language public-rep Q&A culture. There’s no named spokesperson doing SEO community back-and-forth. Baidu’s clearest official English voice is the unsigned, institutional help.baidu.com FAQ. That’s why that FAQ — and the Chinese-language Ziyuan Academy pages behind it — are so valuable: most operational knowledge about Baiduspider in the English-speaking SEO world comes from practitioner testing (server-log analysis, DNS-verification exercises, robots.txt trial and error) rather than official rep commentary. Michael Bonfils summarized the broader lesson in Search Engine Land’s Baidu vs. Google: “SEO strategies that work for Google may not always translate directly to success on Baidu, China’s dominant search engine.”

For where Baiduspider sits in the bigger picture — the crawler family, Googlebot and Bingbot as peers, the user-agent concept, and robots.txt — the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. hub ties it together, and international SEO is the home for the China-market context.

AI summary

A condensed take on the Advanced version:

• BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). = Baidu’s web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — the Chinese-market GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.. Baidu dominates inside mainland China (StatCounter June 2026: Baidu ~47%, Google ~2%), so it’s an international-SEO specialty, not a GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. afterthought. That share swings month to month — treat it as a snapshot.
• Variants share the Baiduspider tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. family: general PC/mobile plus -image, -video, -news, -favo (bookmark), -cpro (ad network), -ads (business). General PC UA: Mozilla/5.0 (compatible; Baiduspider/2.0; +http://www.baidu.com/search/spider.html).
• Baiduspider-render is officially documented — but only on Baidu’s Chinese-language Ziyuan Academy page, absent from the English FAQ. What “-render” does differently is not spec’d.
• Verify by bidirectional DNS to *.baidu.com or *.baidu.jp (Baidu crawls from both): reverse-lookup the IP, then forward-lookup the hostname back to the same IP. The user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. is spoofable; Baidu warns about impersonators.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.: Baidu claims it “strictly complies,” supports */$ wildcards and case-sensitive matching — with a named exception: Baiduspider-cpro and Baiduspider-ads run under commercial agreements and ignore robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. Separately, Baiduspider does not honor the non-standard crawl-delay (per SEJ’s Abby Hamilton). Rule changes take ~48 hours to take effect.
• JavaScript is the biggest gap vs. Googlebot — no official capability spec; industry consensus (Dan Taylor: “notoriously bad… crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. JavaScript”) is that server-rendered/static HTML is the safe default for China-facing sites.
• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. is algorithmic (server capability + site quality + update frequency); report over-crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. via Baidu’s feedback form, and view/cap the rate in Baidu Search Resource PlatformBaidu Webmaster Tools — officially the Baidu Search Resource Platform (百度搜索资源平台) at ziyuan.baidu.com — is Baidu's free webmaster console: verify a site, submit URLs and sitemaps (including the fast API push), and monitor indexing and crawl diagnostics. Since May 2022 registration needs a Chinese mobile number. (Ziyuan) — Baidu’s Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. equivalent.
• hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. isn’t supported by Baidu (industry finding; neither company documents it either way) — fall back to content-language/<html lang>.
• No English rep culture: Baidu’s clearest official English voice is an unsigned FAQ, so practitioner testing carries most operational knowledge.

Official documentation

Primary-source documentation. Baidu’s is split across an English FAQ and Chinese-language Ziyuan (Baidu Search Resource PlatformBaidu Webmaster Tools — officially the Baidu Search Resource Platform (百度搜索资源平台) at ziyuan.baidu.com — is Baidu's free webmaster console: verify a site, submit URLs and sitemaps (including the fast API push), and monitor indexing and crawl diagnostics. Since May 2022 registration needs a Chinese mobile number.) pages; Google links are for the hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others.-intersection contrast.

Baidu

• FAQs of Baiduspider (help.baidu.com) — Baidu’s English-language official FAQ: what BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). is, the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. table, crawl-frequency policy, DNS verification with worked examples, and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. handling. The primary English source.
• 【官方说法】只需两步，正确识别百度蜘蛛（User-Agent）(ziyuan.baidu.com) — Baidu’s Chinese-language official article on identifying Baiduspider: the three UA channels (mobile/PC/mini-program), the Baiduspider-render UAs, and the “bidirectional DNS authentication” method spelled out in full.
• Baidu Search Resource Platform manual index (ziyuan.baidu.com) — the Ziyuan product manual indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. confirming the 抓取频次 (Crawl FrequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.), robots, and 抓取诊断 (Crawl Diagnostics) tool areas.
• baidu.com/robots.txt — Baidu’s own live robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., a real example of per-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. segmentation.

Google (for the hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others.-intersection contrast)

• Tell Google about localized versions of your page — Google’s hreflang docs, scoped entirely to Google Search; notably says nothing about whether Baidu, Bing, or Yandex honor hreflang.

Quotes from the source

Baidu keeps no named English-language spokesperson, so the “source” here is Baidu’s own unsigned FAQ, plus named industry commentary that stands in for the rep quotes you’d get from Google or Bing. Each link is a deep link that jumps to the quoted passage.

Baidu — “FAQs of BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan).” (help.baidu.com, English)

• “Baiduspider is Baidu search engine program which is used to visit pages on the internet and build information into Baidu indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. This enables users to locate your site when they perform a search.” Jump to quote
• “In order to ensure the search results cover most of your pages, Baiduspider must keep the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. at a certain level… to adjust the frequency based on combined factors, such as your server’s capability, your site’s quality and the update frequency of your site.” Jump to quote
• “We recommend using reverse DNS lookup to verify Baiduspider.” Jump to quote
• *“The hostname of Baiduspider is *.baidu.com or .baidu.jp. Others are fake hostnames.” Jump to quote
• “Baidu strictly complies with robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. protocol.” Jump to quote

Baidu — Ziyuan Academy (ziyuan.baidu.com, Chinese)

• Original: “二、双向DNS解析认证” — English paraphrase: “Step two: bidirectional DNS resolution authentication,” Baidu’s own name for verifying Baiduspider via a reverse lookup followed by a forward lookup back to the same IP. View the page

Dan Taylor, Search Engine Journal — on JavaScript

• “Baidu is notoriously bad when it comes to crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. JavaScript, so make sure that all your important content and links are served in plain HTML on both the mobile and desktop versions of your website.” (Mar 2021.) Jump to quote

Michael Bonfils, Search Engine Land — on strategy transfer

• “SEO strategies that work for Google may not always translate directly to success on Baidu, China’s dominant search engine.” Jump to quote

Abby Hamilton, Search Engine Journal — on crawl-delayCrawl-delay is a non-standard robots.txt directive that asks bots to wait between successive page fetches to ease server load. Google has ignored it since September 1, 2019 and Yandex dropped it on February 22, 2018; Bing and many other crawlers still honor it, but Bing interprets the value differently than the plain reading.

• Her crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. comparison table lists Baiduspider as not supporting the non-standard crawl-delay directive (“Baidu | Baiduspider | No”). (Nov 2024.) Read the coverage

Should I let Baiduspider crawl my site — and how do I handle it?

BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). is only worth the effort if you actually want China-market visibility. Click through it.

Verify a bot is really Baiduspider

Baidu’s own FAQ recommends reverse DNS to verify BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan)., because the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string is trivially spoofed. Do the full bidirectional check Baidu names on its Chinese Ziyuan page: reverse-lookup the IP (the hostname must end in *.baidu.com or *.baidu.jp), then forward-lookup that hostname back to the same IP.

macOS / Linux

# 1) Reverse-DNS the IP from your logs — it must end in baidu.com or baidu.jp
host 123.125.66.120
# → 120.66.125.123.in-addr.arpa domain name pointer Baiduspider-123-125-66-120.crawl.baidu.com

# 2) Forward-DNS that hostname back — it must resolve to the same IP
host Baiduspider-123-125-66-120.crawl.baidu.com
# → Baiduspider-123-125-66-120.crawl.baidu.com has address 123.125.66.120

macOS dig form (as Baidu’s FAQ gives it):

dig -x 123.125.66.120

Windows

nslookup 123.125.66.120
nslookup Baiduspider-123-125-66-120.crawl.baidu.com

If the reverse lookup doesn’t end in baidu.com or baidu.jp, or the forward lookup doesn’t return the original IP, it isn’t Baiduspider — drop or rate-limit it. (Same two-step approach as verifying GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.; see those articles’ Scripts tabs for the Google/Bing versions.)

Pull only Baiduspider hits from your server logs

Grab candidate Baiduspider lines from an access log, then verify each IP with the DNS check above — don’t trust the string alone.

# Extract lines whose user-agent claims to be Baiduspider, print unique IPs
grep -i 'baiduspider' /var/log/nginx/access.log \
  | awk '{print $1}' | sort -u

Match a hostname against Baidu’s allowed domains (regex)

Once you’ve reverse-resolved a hostname, this regex confirms it’s in one of Baidu’s two legitimate crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. namespaces (.baidu.com or .baidu.jp) rather than a look-alike:

# Passes only for hostnames ending in .baidu.com or .baidu.jp
echo "Baiduspider-123-125-66-120.crawl.baidu.com" \
  | grep -Eiq '\.baidu\.(com|jp)$' && echo "legit namespace" || echo "FAKE"

Beware look-alikes like baidu.com.evil.example — anchoring the pattern to the end of the string ($) is what stops those from passing.

A robots.txt starting point for Baiduspider

# Let Baiduspider crawl everything except low-value spaces
User-agent: Baiduspider
Disallow: /search
Disallow: /*?*sort=

# Block Baiduspider entirely (China not a target market)
# User-agent: Baiduspider
# Disallow: /

Remember: Disallow controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and rule changes take ~48 hours to take effect. The ad crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. Baiduspider-cpro and Baiduspider-ads ignore robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. by design.

Baiduspider — cheat sheet

User-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. strings and crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. variants

Every declared variant’s UA points to the same reference URL, http://www.baidu.com/search/spider.html — the BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). equivalent of GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s google.com/bot.html.

robots.txt compliance at a glance

Fast facts

• Verify with bidirectional DNS: reverse-lookup the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. IP to confirm a hostname ending in *.baidu.com or *.baidu.jp, then forward-lookup that hostname back to the same IP. The user-agent string alone proves nothing.
• robots.txt changes take about 48 hours to take effect on an already-indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. site.
• Baidu supports * and $ wildcard matching in robots.txt paths, and does exact, case-sensitive path matching.
• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. is algorithmically self-adjusted (server capability, site quality, update frequency) — view and cap it in Baidu Search Resource Platform (Ziyuan).
• Baiduspider’s biggest practical gap vs. GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. is weak JavaScript renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. — server-rendered or static HTML is the safe default for China-facing content.
• Baidu doesn’t support hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. — the fallback signal is an accurate content-language and <html lang>.

Baiduspider readiness checklist

A quick pass to confirm you’re working with BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan)., not accidentally against it — the same ground covered in the Advanced tab, in checklist form.

• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. doesn’t block Baiduspider (or the variant you care about) from anything you want foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. in Baidu search results.
• A sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. is submitted in Baidu Search Resource PlatformBaidu Webmaster Tools — officially the Baidu Search Resource Platform (百度搜索资源平台) at ziyuan.baidu.com — is Baidu's free webmaster console: verify a site, submit URLs and sitemaps (including the fast API push), and monitor indexing and crawl diagnostics. Since May 2022 registration needs a Chinese mobile number. (Ziyuan).
• BotA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. verification uses bidirectional DNS — reverse lookup to *.baidu.com/*.baidu.jp, then a forward lookup confirming the same IP — never the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string alone.
• Aware that Baiduspider-cpro and Baiduspider-ads ignore robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. by design, so blocking those two there won’t actually stop them.
• Any recent robots.txt tightening has had roughly 48 hours to propagate before you re-check indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. status.
• Critical content, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., and metadata all exist in the server-rendered HTML response — not injected only by client-side JavaScript.
• content-language and <html lang> are set correctly, since Baidu doesn’t support hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others..
• Crawl load reviewed in Ziyuan’s Crawl FrequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. (抓取频次) area if Baiduspider traffic is straining the server.
• Unreasonable or excessive crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. reported through Baidu’s feedback form (webmaster.baidu.com/feedback/index) rather than guessed at or blocked outright.
• No lingering reliance on Baidu MIP — the framework is retired (Cache phased out June 2020, full service discontinued by late September 2022).

Tools for working with Baiduspider

• DNS Checker — my own tool for the bidirectional DNS check Baidu’s own FAQ recommends. Enter the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. IP with record type PTR to run the reverse lookup (a genuine BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). hostname ends in .baidu.com or .baidu.jp), then enter that hostname with record type A to confirm the forward lookup resolves back to the same IP. It queries Cloudflare, Google, and Quad9 side by side so you can spot a resolver disagreeing.
• robots.txt Tester — my own tool for checking whether your robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. rules actually block or allow Baiduspider the way you intend. Paste your file, enter Baiduspider (or Baiduspider-image, Baiduspider-news, etc.) in the custom user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. field, and test it against real paths — it shows the exact rule that wins per URL. Remember that Baiduspider-cpro and Baiduspider-ads ignore robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. regardless of what the tester shows, since they run under separate commercial agreements.

Baiduspider myths and mistakes to avoid

Myth: “Any user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. containing ‘BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan).’ is really Baidu.” Why it’s wrong: the user-agent string is trivially spoofed, and Baidu’s own FAQ warns about “spammers or other trouble makers who pretend to be Baiduspider.” Do instead: run the bidirectional DNS check — reverse-lookup to a *.baidu.com/*.baidu.jp hostname, then forward-lookup back to the same IP — before you trust (or block based on) any “Baiduspider” hit.

Myth: “Baidu never follows robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., or is generally unreliable about it.” Why it’s wrong: Baidu’s FAQ states “Baidu strictly complies with robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. protocol” for its standard crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. The real exception is narrow and named — Baiduspider-cpro and Baiduspider-ads run under commercial agreements and ignore robots.txt — not a vague “Baidu ignores robots.” Do instead: cite the specific cpro/ads carve-out, and don’t conflate it with the separate fact that Baiduspider doesn’t honor the non-standard crawl-delay directive.

Myth: “Baiduspider renders JavaScript about as well as modern GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer..” Why it’s wrong: industry consensus (Dan Taylor called it “notoriously bad… crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. JavaScript”) is that Baiduspider lags well behind GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.’s evergreen Chromium; client-side-only content is a real indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. risk for Baidu even when it ranks fine in Google. Do instead: serve important content, links, and metadata in server-rendered or static HTML for China-facing sites — the safe default across Googlebot, BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share., and Baiduspider alike.

Myth: “hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. works the same for Baidu as it does for Google.” Why it’s wrong: Baidu doesn’t support hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. at all (an industry finding — Google’s own docs don’t comment on other engines either way). Do instead: fall back to accurate content-language and <html lang> so Baidu can at least read the page’s language; don’t rely on hreflang to steer Baidu.

Myth: “‘Baiduspider-render’ isn’t an officially documented variant.” Why it’s wrong: Baidu’s English FAQ table doesn’t list it, but Baidu’s Chinese-language Ziyuan Academy page documents Baiduspider-render/2.0 as an alternate PC/mobile UA and a mini-program variant. It’s official — just Chinese-only. Do instead: describe it as officially documented (Chinese-only), while noting Baidu publishes no spec for what “-render” actually does differently.

Myth: “Baidu MIP is still a live, recommended ranking lever.” Why it’s wrong: Baidu’s AMPAMP (Accelerated Mobile Pages) is an open-source web framework Google launched in 2015 to make mobile pages load near-instantly via restricted HTML/CSS/JS and CDN caching. It was never a ranking factor and, since June 2021, is no longer required for Top Stories.-like MIP (Mobile Instant Pages) framework is legacy. Per Jademond’s MIP glossary, the MIP Cache was phased out in June 2020 and the full MIP service was discontinued by late September 2022 — yet some current-dated agency content still promotes it as live, which is itself a source of confusion. Do instead: treat MIP as retired, and be skeptical of any recent guide that presents it as an active speed/ranking lever.

Test yourself: Baiduspider

Five quick questions on Baidu’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Pick an answer for each, then check.

Resources worth your time

My related writing

• Hreflang Tags: A Simple (But Complete) Guide — my Ahrefs guide (with Joshua Hardwick) on hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others.; Google-scoped, and the context for why Baidu’s lack of hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. support is a separate concern.
• Over 67% of Domains Using Hreflang Have Issues — my team’s study of 374,756 domains; Google-implementation focused, useful as the contrast to Baidu’s non-support.
• The Beginner’s Guide to Technical SEO — where crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and international signals fit in the bigger picture.

My speaking

• International SEO: The Weird Technical Parts (Pubcon Vegas 2019, SlideShare) — my talk on the technical edge cases of international SEOInternational SEO is the practice of optimizing a site so search engines understand which countries and/or languages it targets, and serve the right version to each user. It spans URL structure, hreflang, and on-page localization., the terrain Baidu-market work lives in.

From around the industry

• FAQs of Baiduspider (Baidu, help.baidu.com) — Baidu’s own English FAQ: UA table, DNS verification, robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. handling, and crawl-frequency policy. The primary English source.
• Google vs Baidu: Key Differences in SEO Strategy (Dan Taylor, Search Engine Journal) — the “notoriously bad at crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. JavaScript” assessment and the plain-HTML recommendation.
• Baidu vs. Google: Navigating the SEO landscape in 2024 (Michael Bonfils, Search Engine Land) — agency-founder view on why Google tactics don’t transfer directly to Baidu.
• The Modern Guide To Robots.txt (Abby Hamilton, Search Engine Journal) — the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. comparison table showing BaiduspiderBaiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan). doesn’t honor crawl-delay.
• A guide to web crawlers: What you need to know (James Allen, Search Engine Land) — Baiduspider’s place in a broader crawler roundup.
• Understanding Baidu Spiders, User Agents & robots.txt (Jademond Digital) — practitioner deep dive on the Baidu crawler variants and robots handling.
• Baidu MIP (glossary) (Jademond Digital) — the source for the MIP deprecation dates (Cache phased out June 2020; full service discontinued late September 2022).

Baiduspider

Baiduspider is the web crawler operated by Baidu, China's dominant search engine — the Chinese-market equivalent of Googlebot. It discovers, crawls, and indexes pages for Baidu Search, and you manage its access via robots.txt and Baidu's own console, Baidu Search Resource Platform (Ziyuan).

Related: Googlebot, Bingbot, Crawler

Baiduspider

Baiduspider (also written BaiduSpider) is the web crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. run by Baidu (百度), the search engine that dominates inside mainland China, where Google is largely inaccessible. It plays the same role GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. plays for Google: it discovers URLs, downloads pages, and builds the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. Baidu Search pulls from. Site owners manage it through robots.txt and through Baidu’s own webmaster console — Baidu Search Resource PlatformBaidu Webmaster Tools — officially the Baidu Search Resource Platform (百度搜索资源平台) at ziyuan.baidu.com — is Baidu's free webmaster console: verify a site, submit URLs and sitemaps (including the fast API push), and monitor indexing and crawl diagnostics. Since May 2022 registration needs a Chinese mobile number. (百度搜索资源平台, at ziyuan.baidu.com), historically called “Baidu Webmaster ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.'.”

Baidu runs several named crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. variants that share the Baiduspider tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. family — general web (PC and mobile), plus Baiduspider-image, -video, -news, -favo (bookmark), -cpro (ad network), and -ads (business search). A JavaScript-renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. variant, Baiduspider-render, is officially documented too — but only on Baidu’s Chinese-language help pages, not in its English FAQ.

Two things distinguish Baiduspider from GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.. First, its JavaScript renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. lags well behind: industry consensus is that heavy client-side content is a real indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. risk for Baidu specifically, so server-rendered HTML is the safe default for China-facing sites. Second, Baidu keeps no public English-language rep dialogue the way Google, Bing, and Yandex do — its clearest official English voice is an unsigned FAQ. As with the other engines, verify a bot claiming to be Baiduspider with bidirectional DNS (reverse lookup to *.baidu.com or *.baidu.jp, then a forward lookup back to the same IP) — the user-agent string alone is trivially spoofed.

Related: Googlebot, Bingbot, Crawler