YandexBot

What YandexBot is, how to spot and verify it, the Yandex-only Clean-param directive, why Crawl-delay is dead, how it handles JavaScript, and how it compares to Googlebot and Bingbot.

First published: Jul 3, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #17 in Crawling#43 in How Search Works#212 in Technical SEO#290 on the site
AIAuthored byAI v2
1 evidence signal on this page

YandexBot is Yandex's main web crawler — the bot that discovers and fetches pages for Yandex Search, the engine with ~70%+ share in Russia. Its robots.txt token is YandexBot (main indexing bot only) vs. Yandex (the broader bot family). It supports a Yandex-only directive, Clean-param, that consolidates URL parameters with no Google/Bing equivalent — and it stopped honoring Crawl-delay on February 22, 2018 (some SEO guides still wrongly claim otherwise). JavaScript rendering is beta and 'at the bot's discretion,' and the 2023 source-code leak suggested there's no separate JS rendering system the way Google has one. Verify a real YandexBot by reverse-then-forward DNS to a yandex.ru/.net/.com host — the same technique Google and Bing use for their own bots — not by the user-agent string.

TL;DR — YandexBotYandexBot is the main web crawler operated by Yandex, Russia's dominant search engine. It discovers and fetches pages for Yandex Search, is one of a family of Yandex robots, and supports a Yandex-only robots.txt directive (Clean-param) that Google and Bing don't have. is Yandex Search’s main indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. Its robots.txt tokenA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory. YandexBot targets only the main indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. bot; Yandex targets the broader bot family. It supports a Yandex-only directive, Clean-param, that consolidates URL parametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO. — no Google/Bing equivalent — and it stopped honoring Crawl-delay on February 22, 2018 (use the Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. tool instead; some SEO guides still wrongly claim Yandex supports Crawl-delayCrawl-delay is a non-standard robots.txt directive that asks bots to wait between successive page fetches to ease server load. Google has ignored it since September 1, 2019 and Yandex dropped it on February 22, 2018; Bing and many other crawlers still honor it, but Bing interprets the value differently than the plain reading.). JavaScript renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is performed at the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.’s discretion, so prefer SSR/pre-renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. for critical content. Disallow ≠ noindex (same trap as Google). Verify a real YandexBot by reverse-then-forward DNS to a yandex.ru/yandex.net/ yandex.com host — the same technique Google and Bing use — never the user-agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target. string alone.

Evidence for this claim Yandex documents its search robots and their user-agent identifiers in Yandex Webmaster Help. Scope: Current official Yandex robot list. Confidence: high · Verified: Yandex Webmaster: Yandex robots Evidence for this claim Yandex provides an official method for checking whether an IP address belongs to a Yandex robot; a user-agent string alone can be spoofed. Scope: Current Yandex robot verification guidance. Confidence: high · Verified: Yandex Webmaster: Verify a robot

What YandexBot actually is

YandexBot is the primary web crawler for Yandex, the Russian search engine. It discovers URLs, fetches pages, and feeds Yandex’s index — the same role GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. play for their engines. The user-agent string Yandex documents (on its “check that a robot belongs to Yandex” page) is:

Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/81.0.4044.268

Yandex adds a useful caveat next to it: because “the browser’s version may change,” it recommends not matching on a fixed Chrome version when you’re trying to identify the bot. Match the YandexBot token, not Chrome/81.0.4044.268.

Crucially, “YandexBot” is really just the main indexing member of a family of Yandex robots — YandexImages, YandexMetrika, YandexDirect, YandexMobileBot, YandexAccessibilityBot, YandexRenderResourcesBot, YandexCalendar, and more — each independently controllable in robots.txt. Many articles conflate “YandexBot” with “all Yandex crawlers,” which is imprecise.

One more worth knowing about: Yandex’s server-logs table now documents YandexAdditionalBot (and a near-duplicate token, YandexAdditional) as a robot that “helps process robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. to prevent page content from appearing in Search with Yandex AI responses,” applied to pages the main crawler has already indexed. Per that same table, it doesn’t take the general User-agent: * rules into account — so if you want to opt a page out of Yandex’s AI features specifically, you need an explicit User-agent: YandexAdditionalBot block, the same pattern other engines’ AI-crawler opt-outs use.

YandexBot vs. “Yandex” in robots.txt — they’re not the same token

This is Yandex’s most non-obvious robots.txt quirk, and it bites people migrating from Google-centric technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own.. Start from Yandex’s own worked example — it makes the scope split explicit, comments included:

User-agent: YandexBot # will be used only by the main indexing bot
Disallow: /*id=

User-agent: Yandex # will be used by all Yandex bots
Disallow: /*sid= # except the main indexing bot

User-agent: * # will not be used by Yandex bots
Disallow: /cgi-bin

Read literally, that example’s own comments are the documentation:

  • User-agent: YandexBot — used only by the main indexing bot.
  • User-agent: Yandex — used by Yandex bots more broadly — but, per the example’s own comment on the second block, “except the main indexing bot.” The broader token isn’t universal even within the Yandex family.

Two things follow from Yandex’s rules here. First, precedence: “If the User-agent: Yandex string is detected, the User-agent: * string is ignored.” So a generic User-agent: * block won’t apply to Yandex bots if you’ve also written a Yandex block. Second — and this is the one that surprises security-minded readers — Yandex warns that “Some Yandex robots may ignore directives in robots.txt, including those for User-agent: Yandex.” Not every Yandex bot is guaranteed to obey a blanket rule, which is one more reason server-level verification and blocking matter for full exclusion.

Verifying it’s really YandexBot

Because the user-agent is spoofable, Yandex tells you to verify with DNS, exactly like Google and Bing do for their own crawlers. Yandex: “Some robots can disguise themselves as Yandex robots by indicating the relevant User AgentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target.. You can check the authenticity of a robot using a reverse DNS lookup.” The documented method:

  1. “Determine the IP address of the user agent in question using your server logs.”
  2. “Use a reverse DNS lookup of the IP address to determine the host domain name.”
  3. “Check whether the host belongs to Yandex. All Yandex robots have names ending in yandex.ru, yandex.net or yandex.com.” (If the host name has a different ending, it isn’t Yandex.)
  4. “Make sure that the name is correct. Use a forward DNS lookup to get the IP address corresponding to the host name. It should match the IP address used in the reverse DNS lookup.”

And the fail condition, in Yandex’s words: “If the IP addresses do not match, it means that the host name is fake.” Yandex also mentions an official “IP address check tool” as an alternative to running the lookups by hand.

This is the same forward-confirmed reverse-DNS (FCrDNS) pattern all three major engines land on — Google verifies against googlebot.com/google.com/ googleusercontent.com, Bing against *.search.msn.com, and Yandex against yandex.ru/yandex.net/yandex.com. None of them treat a published IP list as trustworthy enough on its own. The commands are in the Scripts tab; the domain suffixes are the only thing that changes between engines. (For the Google and Bing versions, see the Googlebot and Bingbot siblings.)

Controlling YandexBot with robots.txt

Yandex recognizes a familiar core set of directives, each defined in its own docs:

  • User-agent — “Indicates the robot to which the rules listed in robots.txt apply.”
  • Disallow — “Prohibits crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. of sections or individual pages of the site.”
  • Allow — “Allows indexing site sections or individual pages.”
  • SitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. — “Specifies the path to the Sitemap file that is posted on the site.”
  • Clean-param — “Indicates to the robot that the page URL contains parameters (like UTM tags) that should be ignored when indexing it.” (Yandex-only — see below.)

A few file requirements worth knowing: the file must be “a TXT file named “robots”, robots.txt,” its size must not exceed 500 KB, and the server must return an HTTP 200 OKHTTP 200 OK is the standard 2xx success status code, meaning the server received, understood, and fulfilled the request and is returning the resource. It's the code every page you want indexed should return — but a 200 alone doesn't guarantee Google will index the page. status for it to be read.

Disallow ≠ noindex — the same trap as Google

The single most misunderstood robots.txt fact carries straight over to Yandex, and Yandex states it plainly: “Pages restricted in robots.txt can participate in Yandex search. To remove pages from search, specify the noindex directive in the HTML code of the page or configure the HTTP header.” In other words, Disallow controls crawling, not indexing — a disallowed URL can still show up in Yandex’s results. This is the same conceptual trap Google has (I’ve written it up for Google in Indexed, though blocked by robots.txt), and the fix is identical: to actually remove a page, allow crawling and add noindex. Yandex spells out exactly why in the same section: “Do not restrict such pages in robots.txt, or the Yandex bot can’t index them and detect your instructions.” An index-control directive only works if the crawler can fetch the page to see it — Disallow and noindex on the same URL is a contradiction: the Disallow stops Yandex from ever reading the noindex tag, so the page stays exactly where it was.

Clean-param — Yandex’s unique parameter directive

Clean-param is the single most Yandex-specific directive for an audience used to Google and Bing, and it has no Google or Bing equivalent. Its purpose, per Yandex: “The Yandex robot uses this directive to avoid reloading duplicate information. This improves the robot’s efficiently and reduces the server load.” (That “efficiently” is a genuine typo on Yandex’s live page — I’m quoting it as-is rather than silently fixing it.)

The problem it solves: “The new parameter that doesn’t affect the page content may result in duplicate pages that should not be included in the search.” The syntax:

Clean-param: p0[&p1&p2&..&pn] [path]

Yandex’s own worked example — three URLs that differ only by a ref tracking parameter:

www.example.com/some_dir/get_book.pl?ref=site_1&book_id=123
www.example.com/some_dir/get_book.pl?ref=site_2&book_id=123
www.example.com/some_dir/get_book.pl?ref=site_3&book_id=123

…collapse to one canonical URL (www.example.com/some_dir/get_book.pl?book_id=123) with a single directive:

User-agent: Yandex
Clean-param: ref /some_dir/get_book.pl

Two details make Clean-param easy to get wrong. First, “The Clean-param directive does not require mandatory combination with the Disallow directive” — it stands on its own; you don’t Disallow the parameter URLs. Second, it’s intersectional: per Yandex it “is intersectional, so it can be specified anywhere in the file, regardless of the location.” Unlike Allow/Disallow, which are anchored to a path, Clean-param is a global directive you can drop anywhere in the file.

Yandex also notes it may handle some parameters automatically: “Parameters for analytics and tracking that don’t affect the page content may be automatically removed by the search engine if the algorithms determine that those parameters are insignificant.” But relying on Clean-param for the ones that matter to you is the deterministic move. This is the Yandex analog to how Google now leans on canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. signals (canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content., internal linkingLinks between pages on the same site.) since it deprecated its old URL ParametersThe `?key=value` data tacked onto the end of a URL after a question mark — used for tracking, sessions, filtering, sorting, and search — and one of the biggest sources of duplicate URLs and wasted crawling in SEO. tool — Yandex just gives you an explicit robots.txt directive where Google doesn’t.

Crawl-delay is dead (since Feb 2018)

If you’ve read that Crawl-delay works in Yandex’s robots.txt, that information is stale. Yandex’s own dedicated page is unambiguous: “From February 22, 2018, Yandex doesn’t take into account the Crawl-delay directive.”

This is worth flagging because at least one widely-read SEO resource still says the opposite. Ahrefs’ robots.txt guide (authored by Joshua Hardwick, not me) currently states “Google no longer supports this directive, but Bing and Yandex do.” On the Yandex half, that’s contradicted by Yandex’s own current documentation. I’d treat Yandex’s dedicated, dated page as authoritative here — but the broader lesson is the useful one: verify a crawl-behavior claim against the engine’s live docs before you trust a secondhand guide, because these details drift and even good sources go stale. (Note the contrast with Bing, which does still honor crawl-delay — one of the real Bingbot/YandexBot divergences.)

The replacement is the Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. setting in Yandex WebmasterYandex Webmaster (webmaster.yandex.com) is Yandex's free search console — the Yandex equivalent of Google Search Console — for verifying a site, monitoring indexing and queries, requesting reindexing, running diagnostics, and setting a site region. It's essential for any site targeting Russia and the wider CIS, where Yandex, not Google, is the dominant search engine., which lets you influence how fast YandexBot fetches your site. (Yandex’s Crawl-delay page is essentially one sentence plus a pointer to that setting.)

How YandexBot handles JavaScript

Yandex’s JavaScript rendering is explicitly labeled beta (β) by Yandex itself, and the default behavior is “at the bot’s discretion” — the bot “will independently determine whether to execute JavaScript code on the site’s pages.” When it does, it may “assess the quality and completeness of the content on the pages with and without JavaScript” and serve whichever version is likely more useful to the visitor.

There’s a meaningful tension worth surfacing. The 2023 Yandex source-code leak (exposed internal engineering docs, not an official statement — I’ll cover it more below) suggested a simpler picture. As Mike King wrote in his Search Engine Land analysis of the leak: “Yandex has no separate rendering system for JavaScript. They say this in their documentation and, although they have Webdriver-based system for visual regression testing called Gemini, they limit themselves to text-based crawl.” (That internal “Gemini” is a Yandex visual-regression testing tool — completely unrelated to Google’s Gemini AI model, despite the shared name. Worth disambiguating so nobody conflates the two.) And Dan Taylor’s Search Engine Journal write-up concluded there’s “nothing new to suggest Yandex can crawl JavaScript yet outside of already publicly documented processes.”

So the honest answer is neither “YandexBot renders JS just like Googlebot” nor “YandexBot never touches JavaScript.” It’s selective and beta, by Yandex’s own description. The leak’s “no separate rendering system” account is consistent with that framing, but it’s leaked internal material relayed through industry coverage, not a Yandex disclosure — so treat “architecturally simpler than Google” as a plausible reading of two consistent signals, not a documented fact. Don’t take either account on faith for a route that matters to you; test it directly:

  • Rendered output: fetch the page with JavaScript disabled and compare it to the JS-rendered version. If the two differ meaningfully, don’t assume Yandex saw the rendered one.
  • Resource access: confirm the JS, CSS, and API endpoints the page depends on aren’t blocked in robots.txt. Yandex’s YandexRenderResourcesBot fetches render-time resources, but (per Yandex’s own documentation of it) only for pages the main indexing bot can already reach — a blocked resource on an allowed page still won’t load.
  • Delayed and interactive content: anything that loads after the DOMContentLoaded event or behind a click isn’t guaranteed to render. Yandex’s advanced rendering settings (window.YandexRotorSettings) exist specifically for sites where “content loads with a delay” — that’s a signal worth reading, not just a config option.

The practical takeaway: Yandex’s own docs recommend you “Prohibit rendering if SSR (Server-Side Rendering) or pre-rendering is implemented on the site,” and note that “Executing JavaScript code may create additional load on your server.” If you want reliable Yandex indexing of JS-heavy content, serve it server-side rather than testing your luck on client-side rendering.

For AJAX-style sites, Yandex says “When indexing an AJAX site, the Yandex bot scans the original URLs and executes JavaScript code on them” — and it has moved away from the old HTML-snapshot hack: if you still use the deprecated meta name="fragment" approach, “the bot will ignore it and index the original page.” Its modern recommendation mirrors Google’s: “If the links on AJAX pages use the # character, change the addresses to URLs without this character. For example, you may use the History API.”

What the 2023 source-code leak revealed about the crawler

In January 2023, Yandex’s internal source code leaked — a well-corroborated event covered across Search Engine Land, Search Engine Journal, and others. Treat this as leaked internal documentation, not a Yandex statement, but it exposed real detail about how the crawler works. Per Mike King’s SEL analysis: “Yandex’s documentation discusses a dual-distributed crawler system. One for real-time crawling called the ‘Orange Crawler’ and another for general crawling.” He drew a parallel to Google, which “is said to have had an index stratified into three buckets, one for housing real-time crawl, one for regularly crawled and one for rarely crawled.” Both engines, in other words, appear to embrace segmented crawling driven by how often content updates.

The leak also tied crawling directly to site architectureSite architecture is how a website's pages are organized, categorized, and interlinked. It controls how crawlers discover pages, how link equity flows, and how clearly search engines understand each page's topical context. Silo structure, hub and spoke, and topic clusters are the three common models.. Per Dan Taylor’s SEJ coverage, “URLs that are reachable from the homepage have a ‘higher’ level of importance.” That’s a clean bridge from “crawler mechanics” to “why internal linking matters” — the same crawl depthCrawl depth usually means click depth — how many clicks it takes to reach a page from the homepage by following internal links. It can also mean a crawler setting that limits how many levels deep a crawl goes before it stops. logic that applies to Googlebot.

(For scale: coverage noted the widely-cited “1,922 ranking factors” figure was specific to one archive file, with the fuller codebase reportedly containing far more across multiple files — keep a date and a source on any specific number if you cite one.)

Why YandexBot still matters — and how common it is in robots.txt

Yandex’s global share is tiny, but its Russia share is not: ~71% in Russia vs. Google’s ~27% as of June 2026 (StatCounter). That stability is the entire reason YandexBot deserves separate treatment. If you have any Russia/CIS-facing business, blocking YandexBot forecloses the dominant engine in that market.

How often do sites even bother configuring for it? Rarely, but rising. In the Web Almanac 2022 SEO chapter (I was a reviewer that year; I was lead author of the 2021 chapter): YandexBot appeared in “just 0.5% of robots.txt files in 2021. By 2022, there was a six-fold increase, with 3% of files specifying Yandexbot.” Small, but a clear upward trend — and a useful “how common is this in the wild” baseline.

If you’re deciding whether to block it, the honest framing is a business question, not a technical one — and it’s a real debate that plays out in webmaster forums. For the broader mechanics YandexBot lives inside — URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched., the crawl scheduler, rendering, and the crawl-vs-index-vs-rank distinctions — see the crawling hub. And for configuring Yandex specifically as part of a Russia/CIS strategy, the international-SEO and market-specific-SEO material ties it together.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.