YandexBot

TL;DR — YandexBot is the crawler for Yandex Search — the same job Googlebot does for Google and Bingbot does for Bing. It visits your pages, downloads them, and adds them to Yandex’s index. Whether you should care about it comes down to one question: do you have any audience or business in markets where Yandex is relevant? If yes, it matters. If no, it’s mostly just traffic in your logs.

What YandexBot is

When you see YandexBot in your server logs, that’s the crawler for Yandex — the search engine that dominates search in Russia the way Google dominates most of the rest of the world. Just like Googlebot and Bingbot, YandexBot follows links, reads sitemaps, downloads your pages, and hands them off to be added to Yandex’s search index.

The reason it gets its own article — instead of “just block all the non-Google bots” — is geography. Yandex is a rounding error worldwide, but in Russia it holds roughly 71% of the search market versus Google’s ~27% (StatCounter, June 2026). So if you sell to or serve people in Russia (and historically some nearby markets), YandexBot is your gateway to most of that search traffic.

How to spot it

The reliable identity check for YandexBot is a DNS lookup, not the user-agent string — here’s why, and how it works.

YandexBot’s user-agent contains the word YandexBot. Yandex’s own documentation lists the full string as:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268

But that string alone doesn’t prove anything: anyone can fake it. Scrapers and bad bots routinely pretend to be YandexBot. So the user-agent is just a filter for which log lines are worth checking — the actual proof is a reverse-then-forward DNS lookup confirming the request came from a real Yandex host, the same technique Google and Bing use for their own crawlers (covered in the Advanced tab).

Should you block it?

This is the real question most people are asking. A quick way to think about it:

• You have Russia/CIS-facing business? Don’t block it — you’d be cutting yourself out of the dominant search engine there.
• You have zero Russia-facing audience and the crawling is straining your server? Then blocking or slowing it is a reasonable call. You can tell it to stay out with a couple of lines in your robots.txt.

One caveat on the block side: a robots.txt rule is a request, not enforcement. Yandex’s own docs warn that some of its robots may ignore robots.txt directives, so if you need guaranteed exclusion — not just “please don’t” — block by verified IP at the server/firewall level instead (see the Advanced tab for which Yandex bots this affects).

One important gotcha, and it’s the same trap as Google: putting a page in robots.txt doesn’t remove it from Yandex’s search results — it just stops Yandex from reading the page. Yandex says this in its own docs, and adds a condition worth knowing: if you also block a page in robots.txt, Yandex “can’t index them and detect your instructions” — meaning a noindex tag only works if you let Yandex fetch the page to see it. Blocking crawling and adding noindex on the same URL cancels the noindex out. To actually keep a page out, allow crawling and add a noindex tag instead.

Want the technical version — the exact robots.txt tokens, Yandex’s unique Clean-param directive, how to verify a real YandexBot, and how it handles JavaScript? Switch to the Advanced tab.

TL;DR — YandexBot is Yandex Search’s main indexing crawler. Its robots.txt token YandexBot targets only the main indexing bot; Yandex targets the broader bot family. It supports a Yandex-only directive, Clean-param, that consolidates URL parameters — no Google/Bing equivalent — and it stopped honoring Crawl-delay on February 22, 2018 (use the Crawl rate tool instead; some SEO guides still wrongly claim Yandex supports Crawl-delay). JavaScript rendering is performed at the crawler’s discretion, so prefer SSR/pre-rendering for critical content. Disallow ≠ noindex (same trap as Google). Verify a real YandexBot by reverse-then-forward DNS to a yandex.ru/yandex.net/ yandex.com host — the same technique Google and Bing use — never the user-agent string alone.

What YandexBot actually is

YandexBot is the primary web crawler for Yandex, the Russian search engine. It discovers URLs, fetches pages, and feeds Yandex’s index — the same role Googlebot and Bingbot play for their engines. The user-agent string Yandex documents (on its “check that a robot belongs to Yandex” page) is:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268

Yandex adds a useful caveat next to it: because “the browser’s version may change,” it recommends not matching on a fixed Chrome version when you’re trying to identify the bot. Match the YandexBot token, not Chrome/81.0.4044.268.

Crucially, “YandexBot” is really just the main indexing member of a family of Yandex robots — YandexImages, YandexMetrika, YandexDirect, YandexMobileBot, YandexAccessibilityBot, YandexRenderResourcesBot, YandexCalendar, and more — each independently controllable in robots.txt. Many articles conflate “YandexBot” with “all Yandex crawlers,” which is imprecise.

One more worth knowing about: Yandex’s server-logs table now documents YandexAdditionalBot (and a near-duplicate token, YandexAdditional) as a robot that “helps process robots.txt to prevent page content from appearing in Search with Yandex AI responses,” applied to pages the main crawler has already indexed. Per that same table, it doesn’t take the general User-agent: * rules into account — so if you want to opt a page out of Yandex’s AI features specifically, you need an explicit User-agent: YandexAdditionalBot block, the same pattern other engines’ AI-crawler opt-outs use.

YandexBot vs. “Yandex” in robots.txt — they’re not the same token

This is Yandex’s most non-obvious robots.txt quirk, and it bites people migrating from Google-centric technical SEO. Start from Yandex’s own worked example — it makes the scope split explicit, comments included:

User-agent: YandexBot # will be used only by the main indexing bot
Disallow: /*id=

User-agent: Yandex # will be used by all Yandex bots
Disallow: /*sid= # except the main indexing bot

User-agent: * # will not be used by Yandex bots
Disallow: /cgi-bin

Read literally, that example’s own comments are the documentation:

• User-agent: YandexBot — used only by the main indexing bot.
• User-agent: Yandex — used by Yandex bots more broadly — but, per the example’s own comment on the second block, “except the main indexing bot.” The broader token isn’t universal even within the Yandex family.

Two things follow from Yandex’s rules here. First, precedence: “If the User-agent: Yandex string is detected, the User-agent: * string is ignored.” So a generic User-agent: * block won’t apply to Yandex bots if you’ve also written a Yandex block. Second — and this is the one that surprises security-minded readers — Yandex warns that “Some Yandex robots may ignore directives in robots.txt, including those for User-agent: Yandex.” Not every Yandex bot is guaranteed to obey a blanket rule, which is one more reason server-level verification and blocking matter for full exclusion.

Verifying it’s really YandexBot

Because the user-agent is spoofable, Yandex tells you to verify with DNS, exactly like Google and Bing do for their own crawlers. Yandex: “Some robots can disguise themselves as Yandex robots by indicating the relevant User Agent. You can check the authenticity of a robot using a reverse DNS lookup.” The documented method:

1. “Determine the IP address of the user agent in question using your server logs.”
2. “Use a reverse DNS lookup of the IP address to determine the host domain name.”
3. “Check whether the host belongs to Yandex. All Yandex robots have names ending in yandex.ru, yandex.net or yandex.com.” (If the host name has a different ending, it isn’t Yandex.)
4. “Make sure that the name is correct. Use a forward DNS lookup to get the IP address corresponding to the host name. It should match the IP address used in the reverse DNS lookup.”

And the fail condition, in Yandex’s words: “If the IP addresses do not match, it means that the host name is fake.” Yandex also mentions an official “IP address check tool” as an alternative to running the lookups by hand.

This is the same forward-confirmed reverse-DNS (FCrDNS) pattern all three major engines land on — Google verifies against googlebot.com/google.com/ googleusercontent.com, Bing against *.search.msn.com, and Yandex against yandex.ru/yandex.net/yandex.com. None of them treat a published IP list as trustworthy enough on its own. The commands are in the Scripts tab; the domain suffixes are the only thing that changes between engines. (For the Google and Bing versions, see the Googlebot and Bingbot siblings.)

Controlling YandexBot with robots.txt

Yandex recognizes a familiar core set of directives, each defined in its own docs:

• User-agent — “Indicates the robot to which the rules listed in robots.txt apply.”
• Disallow — “Prohibits crawling of sections or individual pages of the site.”
• Allow — “Allows indexing site sections or individual pages.”
• Sitemap — “Specifies the path to the Sitemap file that is posted on the site.”
• Clean-param — “Indicates to the robot that the page URL contains parameters (like UTM tags) that should be ignored when indexing it.” (Yandex-only — see below.)

A few file requirements worth knowing: the file must be “a TXT file named “robots”, robots.txt,” its size must not exceed 500 KB, and the server must return an HTTP 200 OK status for it to be read.

Disallow ≠ noindex — the same trap as Google

The single most misunderstood robots.txt fact carries straight over to Yandex, and Yandex states it plainly: “Pages restricted in robots.txt can participate in Yandex search. To remove pages from search, specify the noindex directive in the HTML code of the page or configure the HTTP header.” In other words, Disallow controls crawling, not indexing — a disallowed URL can still show up in Yandex’s results. This is the same conceptual trap Google has (I’ve written it up for Google in Indexed, though blocked by robots.txt), and the fix is identical: to actually remove a page, allow crawling and add noindex. Yandex spells out exactly why in the same section: “Do not restrict such pages in robots.txt, or the Yandex bot can’t index them and detect your instructions.” An index-control directive only works if the crawler can fetch the page to see it — Disallow and noindex on the same URL is a contradiction: the Disallow stops Yandex from ever reading the noindex tag, so the page stays exactly where it was.

Clean-param — Yandex’s unique parameter directive

Clean-param is the single most Yandex-specific directive for an audience used to Google and Bing, and it has no Google or Bing equivalent. Its purpose, per Yandex: “The Yandex robot uses this directive to avoid reloading duplicate information. This improves the robot’s efficiently and reduces the server load.” (That “efficiently” is a genuine typo on Yandex’s live page — I’m quoting it as-is rather than silently fixing it.)

The problem it solves: “The new parameter that doesn’t affect the page content may result in duplicate pages that should not be included in the search.” The syntax:

Clean-param: p0[&p1&p2&..&pn] [path]

Yandex’s own worked example — three URLs that differ only by a ref tracking parameter:

www.example.com/some_dir/get_book.pl?ref=site_1&book_id=123
www.example.com/some_dir/get_book.pl?ref=site_2&book_id=123
www.example.com/some_dir/get_book.pl?ref=site_3&book_id=123

…collapse to one canonical URL (www.example.com/some_dir/get_book.pl?book_id=123) with a single directive:

User-agent: Yandex
Clean-param: ref /some_dir/get_book.pl

Two details make Clean-param easy to get wrong. First, “The Clean-param directive does not require mandatory combination with the Disallow directive” — it stands on its own; you don’t Disallow the parameter URLs. Second, it’s intersectional: per Yandex it “is intersectional, so it can be specified anywhere in the file, regardless of the location.” Unlike Allow/Disallow, which are anchored to a path, Clean-param is a global directive you can drop anywhere in the file.

Yandex also notes it may handle some parameters automatically: “Parameters for analytics and tracking that don’t affect the page content may be automatically removed by the search engine if the algorithms determine that those parameters are insignificant.” But relying on Clean-param for the ones that matter to you is the deterministic move. This is the Yandex analog to how Google now leans on canonicalization signals (canonical tag, internal linking) since it deprecated its old URL Parameters tool — Yandex just gives you an explicit robots.txt directive where Google doesn’t.

Crawl-delay is dead (since Feb 2018)

If you’ve read that Crawl-delay works in Yandex’s robots.txt, that information is stale. Yandex’s own dedicated page is unambiguous: “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive.”

This is worth flagging because at least one widely-read SEO resource still says the opposite. Ahrefs’ robots.txt guide (authored by Joshua Hardwick, not me) currently states “Google no longer supports this directive, but Bing and Yandex do.” On the Yandex half, that’s contradicted by Yandex’s own current documentation. I’d treat Yandex’s dedicated, dated page as authoritative here — but the broader lesson is the useful one: verify a crawl-behavior claim against the engine’s live docs before you trust a secondhand guide, because these details drift and even good sources go stale. (Note the contrast with Bing, which does still honor crawl-delay — one of the real Bingbot/YandexBot divergences.)

The replacement is the Crawl rate setting in Yandex Webmaster, which lets you influence how fast YandexBot fetches your site. (Yandex’s Crawl-delay page is essentially one sentence plus a pointer to that setting.)

How YandexBot handles JavaScript

Yandex’s JavaScript rendering is explicitly labeled beta (β) by Yandex itself, and the default behavior is “at the bot’s discretion” — the bot “will independently determine whether to execute JavaScript code on the site’s pages.” When it does, it may “assess the quality and completeness of the content on the pages with and without JavaScript” and serve whichever version is likely more useful to the visitor.

There’s a meaningful tension worth surfacing. The 2023 Yandex source-code leak (exposed internal engineering docs, not an official statement — I’ll cover it more below) suggested a simpler picture. As Mike King wrote in his Search Engine Land analysis of the leak: “Yandex has no separate rendering system for JavaScript. They say this in their documentation and, although they have Webdriver-based system for visual regression testing called Gemini, they limit themselves to text-based crawl.” (That internal “Gemini” is a Yandex visual-regression testing tool — completely unrelated to Google’s Gemini AI model, despite the shared name. Worth disambiguating so nobody conflates the two.) And Dan Taylor’s Search Engine Journal write-up concluded there’s “nothing new to suggest Yandex can crawl JavaScript yet outside of already publicly documented processes.”

So the honest answer is neither “YandexBot renders JS just like Googlebot” nor “YandexBot never touches JavaScript.” It’s selective and beta, by Yandex’s own description. The leak’s “no separate rendering system” account is consistent with that framing, but it’s leaked internal material relayed through industry coverage, not a Yandex disclosure — so treat “architecturally simpler than Google” as a plausible reading of two consistent signals, not a documented fact. Don’t take either account on faith for a route that matters to you; test it directly:

• Rendered output: fetch the page with JavaScript disabled and compare it to the JS-rendered version. If the two differ meaningfully, don’t assume Yandex saw the rendered one.
• Resource access: confirm the JS, CSS, and API endpoints the page depends on aren’t blocked in robots.txt. Yandex’s YandexRenderResourcesBot fetches render-time resources, but (per Yandex’s own documentation of it) only for pages the main indexing bot can already reach — a blocked resource on an allowed page still won’t load.
• Delayed and interactive content: anything that loads after the DOMContentLoaded event or behind a click isn’t guaranteed to render. Yandex’s advanced rendering settings (window.YandexRotorSettings) exist specifically for sites where “content loads with a delay” — that’s a signal worth reading, not just a config option.

The practical takeaway: Yandex’s own docs recommend you “Prohibit rendering if SSR (Server-Side Rendering) or pre-rendering is implemented on the site,” and note that “Executing JavaScript code may create additional load on your server.” If you want reliable Yandex indexing of JS-heavy content, serve it server-side rather than testing your luck on client-side rendering.

For AJAX-style sites, Yandex says “When indexing an AJAX site, the Yandex bot scans the original URLs and executes JavaScript code on them” — and it has moved away from the old HTML-snapshot hack: if you still use the deprecated meta name="fragment" approach, “the bot will ignore it and index the original page.” Its modern recommendation mirrors Google’s: “If the links on AJAX pages use the # character, change the addresses to URLs without this character. For example, you may use the History API.”

What the 2023 source-code leak revealed about the crawler

In January 2023, Yandex’s internal source code leaked — a well-corroborated event covered across Search Engine Land, Search Engine Journal, and others. Treat this as leaked internal documentation, not a Yandex statement, but it exposed real detail about how the crawler works. Per Mike King’s SEL analysis: “Yandex’s documentation discusses a dual-distributed crawler system. One for real-time crawling called the ‘Orange Crawler’ and another for general crawling.” He drew a parallel to Google, which “is said to have had an index stratified into three buckets, one for housing real-time crawl, one for regularly crawled and one for rarely crawled.” Both engines, in other words, appear to embrace segmented crawling driven by how often content updates.

The leak also tied crawling directly to site architecture. Per Dan Taylor’s SEJ coverage, “URLs that are reachable from the homepage have a ‘higher’ level of importance.” That’s a clean bridge from “crawler mechanics” to “why internal linking matters” — the same crawl depth logic that applies to Googlebot.

(For scale: coverage noted the widely-cited “1,922 ranking factors” figure was specific to one archive file, with the fuller codebase reportedly containing far more across multiple files — keep a date and a source on any specific number if you cite one.)

Why YandexBot still matters — and how common it is in robots.txt

Yandex’s global share is tiny, but its Russia share is not: ~71% in Russia vs. Google’s ~27% as of June 2026 (StatCounter). That stability is the entire reason YandexBot deserves separate treatment. If you have any Russia/CIS-facing business, blocking YandexBot forecloses the dominant engine in that market.

How often do sites even bother configuring for it? Rarely, but rising. In the Web Almanac 2022 SEO chapter (I was a reviewer that year; I was lead author of the 2021 chapter): YandexBot appeared in “just 0.5% of robots.txt files in 2021. By 2022, there was a six-fold increase, with 3% of files specifying Yandexbot.” Small, but a clear upward trend — and a useful “how common is this in the wild” baseline.

If you’re deciding whether to block it, the honest framing is a business question, not a technical one — and it’s a real debate that plays out in webmaster forums. For the broader mechanics YandexBot lives inside — URL discovery, the crawl scheduler, rendering, and the crawl-vs-index-vs-rank distinctions — see the crawling hub. And for configuring Yandex specifically as part of a Russia/CIS strategy, the international-SEO and market-specific-SEO material ties it together.

AI summary

A condensed take on the Advanced version:

• YandexBot = Yandex Search’s main indexing crawler — the Yandex equivalent of Googlebot/Bingbot. UA string: Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268 (don’t match on the Chrome version — it changes).
• Two robots.txt tokens, different scopes: YandexBot = main indexing bot only; Yandex = the broader bot family. A User-agent: Yandex block makes Yandex ignore User-agent: *. Some Yandex bots may ignore robots.txt entirely.
• Clean-param is Yandex-only — consolidates URL parameters that don’t change content (no Google/Bing equivalent). It doesn’t require Disallow, and it’s intersectional (can go anywhere in the file).
• Crawl-delay is dead — Yandex stopped honoring it on February 22, 2018; use the Crawl rate tool instead. Some SEO guides still wrongly claim Yandex supports it. (Bing does; Yandex doesn’t.)
• Disallow ≠ noindex — disallowed pages “can participate in Yandex search.” Use noindex (meta or HTTP header) to actually remove a page.
• JavaScript rendering is beta and “at the bot’s discretion.” The 2023 leak suggested “no separate rendering system for JavaScript” — so it’s selective and simpler than Google’s two-wave pipeline. Prefer SSR/pre-rendering; avoid hash-bang URLs (use the History API).
• 2023 source-code leak (leaked, not official): revealed a dual-crawler system (real-time “Orange Crawler” + general crawler) and that homepage-reachable URLs carry higher importance (crawl-depth as a signal).
• Verify with reverse-then-forward DNS to a yandex.ru/yandex.net/ yandex.com host — the same FCrDNS technique Google and Bing use — never the spoofable user-agent.
• Why care: ~71% Russia search share (vs. Google ~27%, June 2026). Tiny globally, dominant regionally. In robots.txt: ~0.5% of files (2021) → 3% (2022) per the Web Almanac.

Official documentation

Primary-source documentation, mostly from Yandex Webmaster Help, with Google/Bing contrast links. Heads up: several Yandex Webmaster tool pages (Crawl rate, the robots.txt analyzer) are app pages that need a logged-in browser to see fully, and Google’s/Bing’s verification pages are JavaScript-rendered — confirm details live if a link bounces.

Yandex

• The User-agent directive — the YandexBot vs. Yandex token scoping and precedence rules.
• How to check that a robot belongs to Yandex — the full user-agent string and the reverse-DNS verification method.
• Using robots.txt — the directives Yandex recognizes, the 500 KB / HTTP 200 file requirements, and the Disallow ≠ index note.
• The Clean-param directive — Yandex’s unique parameter-consolidation directive (syntax + worked example).
• The Crawl-delay directive — the page stating Crawl-delay has been ignored since February 22, 2018.
• Site crawl rate — the Crawl-delay replacement in Yandex Webmaster.
• Indexing pages with JavaScript (β) — the “at the bot’s discretion” default and the SSR recommendation.
• Indexing AJAX sites — JS execution on original URLs and the History-API recommendation.

Google / Bing (for contrast)

• Verify Requests from Google Crawlers and Fetchers — Google’s reverse/forward-DNS method (against googlebot.com/google.com/googleusercontent.com) — the same FCrDNS pattern Yandex uses.
• Which crawlers does Bing use? — Bing’s crawler list; Bing verifies against *.search.msn.com and still honors crawl-delay (Yandex doesn’t).

Quotes from the source

On-the-record statements from Yandex’s own documentation, plus the leaked-internal material relayed through industry coverage. Each link is a deep link that jumps to the quoted passage. Yandex’s docs are English translations published by Yandex itself — quoted verbatim from those pages.

Yandex — robots.txt tokens and precedence

• “If the User-agent: Yandex string is detected, the User-agent: * string is ignored.” Jump to quote
• “Some Yandex robots may ignore directives in robots.txt, including those for User-agent: Yandex.” Jump to quote
• “YandexAdditionalBot … Helps process robots.txt to prevent page content from appearing in Search with Yandex AI responses. Applies to the pages that have been indexed by the primary crawler.” Jump to quote

Yandex — verifying the bot

• “Some robots can disguise themselves as Yandex robots by indicating the relevant User Agent. You can check the authenticity of a robot using a reverse DNS lookup.” Jump to quote
• “Check whether the host belongs to Yandex. All Yandex robots have names ending in yandex.ru, yandex.net or yandex.com.” Jump to quote
• “If the IP addresses do not match, it means that the host name is fake.” Jump to quote

Yandex — robots.txt and Clean-param

• “Pages restricted in robots.txt can participate in Yandex search. To remove pages from search, specify the noindex directive in the HTML code of the page or configure the HTTP header.” Jump to quote
• “The Yandex robot uses this directive to avoid reloading duplicate information. This improves the robot’s efficiently and reduces the server load.” [sic — “efficiently” is Yandex’s own typo] Jump to quote
• “The Clean-param directive does not require mandatory combination with the Disallow directive.” Jump to quote
• “Do not restrict such pages in robots.txt, or the Yandex bot can’t index them and detect your instructions.” Jump to quote

Yandex — Crawl-delay and JavaScript

• “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive.” Jump to quote
• “Prohibit rendering if SSR (Server-Side Rendering) or pre-rendering is implemented on the site.” Jump to quote
• “When indexing an AJAX site, the Yandex bot scans the original URLs and executes JavaScript code on them.” Jump to quote

The 2023 Yandex source-code leak (leaked internal documentation, relayed via industry coverage — not an official Yandex statement)

• “Yandex has no separate rendering system for JavaScript. They say this in their documentation and, although they have Webdriver-based system for visual regression testing called Gemini, they limit themselves to text-based crawl.” — Mike King, iPullRank, via Search Engine Land. Read the coverage
• “Yandex’s documentation discusses a dual-distributed crawler system. One for real-time crawling called the ‘Orange Crawler’ and another for general crawling.” — Search Engine Land. Read the coverage
• “There’s nothing new to suggest Yandex can crawl JavaScript yet outside of already publicly documented processes.” — Dan Taylor, via Search Engine Journal. Read the coverage

Web Almanac 2022, SEO chapter (I was a reviewer)

• “Yandexbot was specified in just 0.5% of robots.txt files in 2021. By 2022, there was a six-fold increase, with 3% of files specifying Yandexbot.” Jump to quote

Should I block, allow, or throttle YandexBot?

The YandexBot question is almost always a business question wearing a technical costume. Work through it.

YandexBot myths and mistakes to avoid

The traps that come up most often — several are widely-repeated and worth correcting:

• “Crawl-delay slows YandexBot down.” No. Yandex stopped honoring Crawl-delay on February 22, 2018 (“Yandex doesn’t take into account the Crawl-delay directive”). Some SEO guides still claim Yandex supports it — that’s stale. Use the Crawl rate setting in Yandex Webmaster instead. (Do-instead: verify crawl-behavior claims against Yandex’s live docs, and throttle via Crawl rate, not Crawl-delay.)
• “Disallow in robots.txt removes the page from Yandex.” No — Yandex’s own docs say disallowed pages “can participate in Yandex search.” Disallow stops crawling, not indexing. (Do-instead: use noindex in the HTML or an HTTP header to actually exclude a page; let Yandex crawl it so it can see the tag.)
• “User-agent: YandexBot and User-agent: Yandex are the same thing.” No — YandexBot targets only the main indexing bot; Yandex targets the broader family. And a User-agent: Yandex block makes Yandex ignore your User-agent: * block entirely. (Do-instead: pick the token that matches the scope you actually intend, and don’t assume * rules reach Yandex bots.)
• “YandexBot renders JavaScript just like Googlebot.” Not established. Yandex’s rendering is beta and “at the bot’s discretion,” and the 2023 leak suggested “no separate rendering system for JavaScript.” (Do-instead: serve JS-dependent content via SSR/pre-rendering rather than assuming client-side rendering will be indexed.)
• “The user-agent string proves it’s YandexBot.” No — it’s trivially spoofed. (Do-instead: verify with reverse-then-forward DNS to a yandex.ru/yandex.net/ yandex.com host, exactly as you’d verify Googlebot or Bingbot.)
• “Any YandexBot traffic on a non-Russian site is inherently suspicious/fake.” Not necessarily — legitimate YandexBot crawls globally-facing sites too, not just .ru domains. (Do-instead: verify by DNS before deciding it’s spoofed; a real Yandex host is real regardless of your audience.)
• “The 2023 leak proved Yandex has a secret superior JS crawler.” No — coverage concluded there’s “nothing new to suggest Yandex can crawl JavaScript yet outside of already publicly documented processes.” (Do-instead: don’t over-read the leak; it corroborated the limited-rendering picture, it didn’t overturn it.)

YandexBot — cheat sheet

User-agent string (main indexing bot)

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268

Match on the YandexBot token, not the Chrome/81.0.4044.268 version — Yandex says the browser version may change.

The two robots.txt tokens

A User-agent: Yandex block causes Yandex to ignore your User-agent: * block. Some Yandex bots may ignore robots.txt entirely.

Directives Yandex recognizes

Clean-param quick form

User-agent: Yandex
Clean-param: ref /some_dir/get_book.pl

Consolidates ?ref=… variants to one URL. Doesn’t need Disallow; intersectional (can go anywhere in the file).

Verify vs. Google vs. Bing (all use forward-confirmed reverse DNS)

Fast facts

• Crawl-delay: dead since Feb 22, 2018 — use the Crawl rate tool. (Bing still honors crawl-delay; Yandex doesn’t.)
• robots.txt must be named robots.txt, be ≤ 500 KB, and return HTTP 200.
• JavaScript rendering: beta, “at the bot’s discretion” — prefer SSR.
• Russia share: ~71% (vs. Google ~27%), June 2026 (StatCounter).
• In robots.txt: 0.5% (2021) → 3% (2022) of files (Web Almanac).

Verify a bot is really YandexBot

The user-agent is trivially spoofed, so confirm with a reverse-DNS lookup (must end in a Yandex domain) followed by a forward-DNS lookup (must resolve back to the same IP). This is the exact FCrDNS pattern you’d use for Googlebot or Bingbot — only the domain suffixes differ (yandex.ru / yandex.net / yandex.com).

macOS / Linux

# 1) Reverse-DNS the IP from your logs — the host must end in
#    yandex.ru, yandex.net, or yandex.com
host 5.255.253.1
# → 1.253.255.5.in-addr.arpa domain name pointer <something>.yandex.com  (illustrative)

# 2) Forward-DNS that hostname back — it must resolve to the same IP
host <the-hostname-from-step-1>.yandex.com

Windows

nslookup 5.255.253.1
nslookup <the-hostname-from-step-1>.yandex.com

If the reverse lookup doesn’t end in a Yandex domain, or the forward lookup doesn’t return the original IP, it isn’t YandexBot — drop or rate-limit it.

Extract verified-vs-spoofed YandexBot hits from a log file

A quick shell pass to pull “YandexBot” lines and check each source IP’s PTR. Adjust field positions for your log format (this assumes a common combined format with the IP first).

# Pull unique IPs that claimed to be YandexBot, then reverse-resolve each
grep -i 'YandexBot' access.log \
  | awk '{print $1}' | sort -u \
  | while read ip; do
      host="$(host "$ip" 2>/dev/null | awk '/pointer/{print $NF}' | sed 's/\.$//')"
      case "$host" in
        *.yandex.ru|*.yandex.net|*.yandex.com) echo "REAL   $ip  $host" ;;
        "" )                                    echo "NO-PTR $ip" ;;
        * )                                     echo "FAKE   $ip  $host" ;;
      esac
    done

REAL lines still deserve a forward-lookup confirmation for anything you’ll act on (host "$host" should return the original IP), but this triages the obvious impostors first.

Regex to match the YandexBot token in a user-agent

Match the product token, not the Chrome version (which changes). Case-insensitive:

YandexBot/\d+(\.\d+)?

Broader “any Yandex robot” match (catches YandexImages, YandexMobileBot, etc.):

Yandex[A-Za-z]*/\d

Remember: a matching user-agent is necessary but not sufficient — pair any regex match with the DNS check above before trusting it.

A robots.txt starting point for Yandex

# Slow/duplicate-parameter cleanup for all Yandex bots
User-agent: Yandex
Clean-param: utm_source&utm_medium&utm_campaign /

# Block only the main indexing bot from a low-value space
User-agent: YandexBot
Disallow: /internal-search/

Sitemap: https://example.com/sitemap.xml

Remember: Disallow blocks crawling, not indexing — use noindex to remove a page from Yandex search. To fully block all Yandex bots (some ignore robots.txt), block by verified IP at the server.

YandexBot readiness checklist

A quick pass to confirm you’re handling YandexBot deliberately, not by accident:

• You’ve decided whether Yandex matters for your site (any Russia/CIS audience or business?) — the whole block/allow question hinges on this.
• Bot verification uses reverse + forward DNS to a yandex.ru/yandex.net/ yandex.com host — not the spoofable user-agent, and not a hardcoded IP list.
• You’re using the right robots.txt token for your intent: YandexBot (main indexing bot only) vs. Yandex (the broader family).
• You know a User-agent: Yandex block makes Yandex ignore your User-agent: * block.
• You’re not relying on Crawl-delay (dead since Feb 22, 2018) — throttle via the Crawl rate tool in Yandex Webmaster instead.
• You’re not using Disallow to deindex — that’s noindex’s job (with crawling allowed).
• Duplicate URL parameters are consolidated with Clean-param where relevant (it doesn’t need Disallow, and it can go anywhere in the file).
• JS-dependent content is served via SSR/pre-rendering, since Yandex’s rendering is beta and “at the bot’s discretion.”
• AJAX routes use clean URLs / the History API, not hash-bang (#!) URLs.
• For full exclusion of all Yandex bots (some ignore robots.txt), you’re blocking by verified IP at the server, not just in robots.txt.

Tools for verifying and monitoring YandexBot

The two jobs that come up most with YandexBot are confirming a hit is real and seeing how much it’s actually crawling you. Start here:

• Googlebot Verifier — paste an IP from your logs and it runs the forward-confirmed reverse-DNS check for you (the same method Yandex documents for confirming a yandex.ru/yandex.net/yandex.com host), naming the real network owner when it’s a spoofer instead. Faster than running host/nslookup by hand for every suspicious hit, and it also covers Googlebot, Bingbot, and the major AI crawlers if you’re checking a mixed log.
• Log File Analyzer — drop in a server access log (nginx, Apache, IIS/W3C, or JSON) and see YandexBot’s actual crawl footprint: how much of your site it’s hitting, which sections, status-code waste, and a spoofer report flagging IPs that claim to be YandexBot but don’t check out. Useful for the “is this crawling actually straining my server” question from the decision tree above — everything runs in your browser, nothing is uploaded.

From Yandex itself

• Yandex Webmaster — the Crawl rate setting (the Crawl-delay replacement) and the robots.txt analyzer live here; you’ll need a verified, logged-in Yandex Webmaster account to use either.

Resources worth your time

My related writing

• Indexed, though blocked by robots.txt — why a robots-blocked URL still gets indexed; the same Disallow ≠ noindex trap applies to Yandex.
• Robots.txt and SEO: Everything You Need to Know — the general robots.txt reference (note: its Yandex Crawl-delay claim is outdated, per Yandex’s own dated docs — a good example of verifying against the source).
• The Story of Blocking 2 High-Ranking Pages With Robots.txt — my first-party experiment on what actually happens when you block ranking pages.
• Meet the New Web Crawlers: AI Bots Are Closing in on Search Engine Bots — how the crawler cast in your logs has changed.
• The SEO Bots That ~140 Million Websites Block the Most — my study with Xibeijia Guan on robots.txt block rates (it covers Western SEO-tool bots, not YandexBot — cited for the methodology and as a contrast to how rarely sites configure for Yandex).

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking. (Standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Yandex scrapes Google and other SEO learnings from the source code leak (Mike King / iPullRank, Search Engine Land, Jan 30, 2023) — the “Orange Crawler” dual-crawler system and “no separate rendering system for JavaScript” findings.
• Yandex Data Leak: The Ranking Factors & The Myths We Found (Dan Taylor, Search Engine Journal, Feb 1, 2023) — the crawl-depth-as-importance signal and the “nothing new on JavaScript crawling” conclusion.
• Yandex ‘leak’ reveals 1,922 search ranking factors (Search Engine Land) — context and dating for the leak’s scale.
• Does the Yandex Code Leak Tell Us Anything About Google? (seoClarity) — a measured cross-engine read of the leak.
• The Ultimate Guide to Yandex SEO (Search Engine Journal) — broader Yandex-optimization context beyond the crawler.
• Web Almanac 2022 — SEO chapter (HTTP Archive) — the source of the YandexBot-in-robots.txt adoption stat.

Stats worth citing

• ~71% Russia search share — Yandex’s share of the Russian search market versus Google’s ~27%, per StatCounter (data reported for June 2026; time-sensitive, so cite the month/year and expect drift). This is the entire reason YandexBot warrants separate treatment from “block all non-Google bots.” Source
• 0.5% → 3% of robots.txt files — YandexBot went from being specified in “just 0.5% of robots.txt files in 2021” to “3% of files specifying Yandexbot” in 2022 — a six-fold increase, though still small (Web Almanac 2022 SEO chapter, which I reviewed). Useful as a historical baseline for “how common is this in the wild.” Source
• February 22, 2018 — the date Yandex stopped honoring Crawl-delay, per its own documentation. A precise, citable deprecation date to counter stale guides claiming Yandex still supports the directive. Source

Test yourself: YandexBot

Five quick questions on YandexBot, its robots.txt behavior, and how it compares to Googlebot and Bingbot. Pick an answer for each, then check.

YandexBot

YandexBot is the main web crawler operated by Yandex, Russia's dominant search engine. It discovers and fetches pages for Yandex Search, is one of a family of Yandex robots, and supports a Yandex-only robots.txt directive (Clean-param) that Google and Bing don't have.

Related: Crawling, Crawler, User Agent

YandexBot

YandexBot is the primary web crawler for Yandex, the search engine that still holds roughly 70%+ of the search market in Russia. It’s the Yandex equivalent of Googlebot and Bingbot: it discovers URLs (through links and sitemaps), downloads pages, and hands them off to be indexed for Yandex Search. In server logs it identifies itself with a user-agent containing YandexBot — the full string Yandex documents is Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268.

“YandexBot” is really the main indexing member of a larger family of Yandex robots (YandexImages, YandexMetrika, YandexDirect, YandexMobileBot, YandexCalendar, and others), each separately controllable in robots.txt. Yandex draws a specific line: User-agent: YandexBot targets only the main indexing bot, while User-agent: Yandex targets its bots more broadly.

Two things make YandexBot worth treating separately from Googlebot: it honors a Yandex-only directive called Clean-param (which tells the crawler to ignore certain URL parameters that don’t change page content, with no Google or Bing equivalent), and it stopped honoring Crawl-delay on February 22, 2018. Like Google and Bing, Yandex says to verify a claimed YandexBot with a reverse-then-forward DNS check (hostname must end in yandex.ru, yandex.net, or yandex.com) rather than trusting the user-agent string.

Related: Crawling, Crawler, User Agent

Changelog