Crawl Rate

TL;DR — Crawl rate is how fast a search engine fetches pages from your site. Google sets it automatically based on how healthy your server is — there’s no longer a button to turn it up or down. If your server is getting hammered, the fix is to make it faster (or, for a day or two only, have it return a “slow down” error). Crawling faster doesn’t help you rank.

What crawl rate is

When a search engine crawls your site, it doesn’t grab every page at once — it paces itself. Crawl rate is that pace: how many pages a crawler fetches per moment, and how long it waits between fetches. Googlebot for Google, Bingbot for Bing. Evidence for this claim Google defines crawl capacity using simultaneous connections and the delay between fetches, adjusted according to site responses. Scope: Google crawler capacity, not a ranking factor. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

The whole point of the pacing is politeness. A crawler could easily overwhelm a small server if it asked for hundreds of pages a second, so it watches how your site responds and backs off when the server starts to struggle. Answer quickly and it’ll fetch a bit faster; slow down or throw errors and it crawls less.

The big thing that changed

For years the advice was “go into Search Console and turn down the crawl rate slider.” That slider is gone — Google removed it on January 8, 2024. So if you find a tutorial telling you to adjust it, that tutorial is out of date. Evidence for this claim Google deprecated the Search Console crawl-rate limiter and removed it on January 8, 2024. Scope: Google Search Console's legacy crawl-rate limiter. Confidence: high · Verified: 2026-07-14 Google: Crawl rate limiter deprecation

Today the rate is automatic. You don’t dial it; Google reads your server’s responses and decides.

How to slow Googlebot down

If your server really is being overloaded, in order of preference:

1. Make the server faster or give it more resources. This is the real fix. Googlebot will naturally crawl a healthy, fast server more comfortably.
2. For a true emergency only: have your server return a 500, 503, or 429 error instead of the normal page. Google reads that as “slow down” almost immediately. But do this for a day or two at most — leave it on longer and Google may start dropping your pages from search.

What not to do: don’t block Googlebot with 403/404 errors (it doesn’t slow crawling and risks losing pages), and don’t rely on crawl-delay in your robots.txt — Google ignores it (Bing does honor it).

Can you make Google crawl faster?

Not on demand — there’s no “crawl more” button, and you can’t request an increase. You can encourage it indirectly: a faster server, a clean sitemap, good internal links, and getting rid of duplicate or junk URLs all help. But more crawling is not a goal in itself — it won’t make you rank higher.

Want the status-code details, the deprecation history, and how Bing differs? Switch to the Advanced tab.

TL;DR — Crawl rate is the supply side of crawl budget — how fast a crawler fetches, set by what Google calls the crawl capacity limit (parallel connections

• delay between fetches). It’s automatic and reacts to server health; the manual GSC slider was removed Jan 8, 2024. To slow Googlebot now you signal via HTTP: 500/503/429 for 1–2 days max, never 401/403/404, and crawl-delay is ignored by Google (honored by Bing). You can’t request an increase — improve it indirectly. Bing still has a manual Crawl Control grid. Crawl rate is not a ranking factor, and most sites never need to touch it.

What crawl rate actually is

Crawl rate is how fast a crawler fetches pages from your server — the number of simultaneous requests it makes and the delay between them. Google’s name for it is the crawl capacity limit: “the maximum number of simultaneous parallel connections that Google can use to crawl a site, as well as the time delay between fetches.” Evidence for this claim Google defines crawl capacity using simultaneous connections and the delay between fetches, adjusted according to site responses. Scope: Google crawler capacity, not a ranking factor. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

It’s one half of crawl budget. As I put it in my Ahrefs crawl-budget guide, crawl budget breaks into “crawl demand which is how many pages a search engine wants to crawl on your site and crawl rate which is how fast they can crawl.” Crawl rate is the supply side (how fast you’ll let them); crawl demand is the demand side (how much they want to). Crawl budget is where the two meet — and rankings sit outside that loop entirely.

In my How Search Works deck I frame the crawl rate limit as simply what your site can support — it’s driven by your server’s stability and crawl health, slow responses, 5xx (server) errors, and 429 (too many requests) responses. Google backs off when your server starts to struggle because it doesn’t want to crash your site. One detail people miss: all the Googlebots share a single crawl pool — the bots for search, images, ads, and so on all draw from the same rate. So a runaway crawl of one resource type eats into the crawling of everything else.

What sets the crawl rate

The capacity limit is automatic and reacts to your server in real time. Google: “If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Google crawls less.”

There’s a second lever you don’t control: Google’s own resources. “Google has a lot of machines, but not infinite machines. We still need to make choices with the resources that we have.” So your server health sets the ceiling Google is willing to use, but Google’s own capacity and your site’s crawl demand decide how much of that ceiling actually gets used.

The myths doc confirms the server-health link works both ways: “A speedy site is a sign of healthy servers, so it can get more content over the same number of connections,” while “a significant number of 5xx HTTP response status codes (server errors) or connection timeouts signal the opposite, and crawling slows down.”

Does crawl rate affect rankings? No.

Bust this first, because it drives a lot of misguided effort. Crawling is required to appear in search, but it is not a ranking signal. Google is explicit: “Improving your crawl rate won’t necessarily lead to better positions in Google Search results.” Faster or more crawling buys you fresher discovery and indexing, not higher positions. Crawl rate is an efficiency and server-health concern — full stop.

How to reduce Googlebot’s crawl rate

In order, from the real fix to the emergency lever:

1. Fix the server (the real fix). Speed it up or add resources. Because the capacity limit tracks response time and errors, a healthier server is the durable way to keep crawling in a comfortable range — and it’s the one that doesn’t risk your indexing.

2. The emergency lever — 500/503/429. Google: “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” Its crawlers “treat the 429 status code as a signal that the server is overloaded,” and “5xx and 429 server errors prompt Google’s crawlers to temporarily slow down with crawling.” Gary Illyes put the speed of this in concrete terms: “if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling.” Prefer 429 when you can — it explicitly means “too many requests” and can carry a Retry-After header.

The catch is that this is strictly temporary. Google: “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” Leave it on and the cost is real — if those codes persist on the same URL for multiple days, “the URL may be dropped from Google’s index” and, for Google Ads, “your campaigns may be cancelled or paused, and your ads may not serve.”

3. What NOT to do. Don’t use 4xx to throttle. “The 4xx status codes, except 429, have no effect on crawl rate,” and Google says plainly “Don’t use 401 and 403 status codes for limiting the crawl rate.” In a dedicated 2023 post about people doing exactly this, Google wrote: “Over the last few months we noticed an uptick in website owners and some content delivery networks (CDNs) attempting to use 404 and other 4xx client errors (but not 429) to attempt to reduce Googlebot’s crawl rate. The short version of this blog post is: please don’t do that…” And don’t reach for robots.txt crawl-delay: “The non-standard ‘crawl-delay’ robots.txt rule is not processed by Google’s crawlers.”

4. The non-emergency request. For an ongoing problem that isn’t a fire, you can “file a special request to report a problem with unusually high crawl rate, mentioning the optimal rate for your site.” It’s slow, and it only goes one direction (see below).

Can you increase crawl rate? No — not directly.

There is no manual increase. Google: “You cannot request an increase in crawl rate, and it may take several days for the request to be evaluated and fulfilled.” What you can do is improve it indirectly. From my crawl-budget guide, the levers that actually move crawl efficiency: speed up your server / add resources; keep important pages in clean sitemaps; kill duplicate content; earn more links (external and internal); fix redirected links; use GET instead of POST where possible; and use the Indexing API where you’re eligible. Note that “a speedy site… can get more content over the same number of connections” — so server health is the lever that shows up on both the reduce and the increase side.

How to control Bingbot’s crawl rate

Here’s the sharp cross-engine contrast. Where Google retired manual control, Bing kept it. Bing Webmaster Tools has Crawl Control (under Configuration): an hour-by-hour grid where each hour shows bricks representing crawl speed — more bricks means faster crawling, fewer means slower. You can pick a preset built around your peak business hours, or choose “Custom” and draw your own pattern across the day. Bing also still honors crawl-delay in robots.txt — the exact directive Google ignores. So the two-engine rule of thumb: for Google you signal through server responses; for Bing you have an actual dial.

What happened to the Search Console crawl rate tool?

The timeline, because so much advice still references a tool that no longer exists:

• Dec 2008 — Google introduces user crawl-rate control in Webmaster Tools.
• Feb 2023 — Google publishes “don’t use 403s or 404s for rate limiting.”
• Nov 24, 2023 — Google announces the deprecation of the Crawl Rate Limiter Tool. Illyes’ reasoning: “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.” The old tool was slow and barely used — it had “a much slower effect” and “would have taken over a day for the new limits to be applied on crawling,” and was used “rarely,” with those who did “in many cases set the crawling speed to the bare minimum.”
• Jan 8, 2024 — the tool is removed. Evidence for this claim Google deprecated the Search Console crawl-rate limiter and removed it on January 8, 2024. Scope: Google Search Console's legacy crawl-rate limiter. Confidence: high · Verified: 2026-07-14 Google: Crawl rate limiter deprecation Google also lowered the floor: “With the deprecation of the crawl limiter tool, we’re also setting the minimum crawling speed to a lower rate, comparable to the old crawl rate limits.”

The practical upshot: the old manual slider had a >24-hour lag anyway; today’s server-signal approach (5xx/429) slows Googlebot almost immediately, which is strictly better for an actual emergency.

How to monitor crawl rate

The GSC Crawl Stats report is your window into what Google is actually doing: total crawl requests over time, total download size, average response time, and a host status view of your site’s availability to Google over the last ~90 days, plus a breakdown by response code, file type, crawl purpose, and Googlebot type. Watch average response time and the host-status signals — a climbing response time or a rash of 5xx is exactly what makes Googlebot throttle you, so this report is where you’ll see a self-inflicted slowdown before you go hunting for one. On Bing, Crawl Control and the crawl information in Bing Webmaster Tools are the equivalents.

Crawl rate vs crawl budget vs crawl frequency

Keep these straight:

• Crawl rate = how fast (supply / capacity).
• Crawl demand = how much they want to (popularity + staleness).
• Crawl budget = the interaction of the two — “the amount of time and resources a search engine allows for crawling a website.”
• Crawl frequency = how often a given page gets re-crawled, which is mostly a crawl-demand question (popularity and how fresh/stale the page is).

And the reassuring part, which is my standing take: most sites don’t need to worry about any of this. “Most sites don’t need to worry about crawl budget, but there are few cases where you may want to take a look” — newer sites with a lot of pages, very large or rapidly changing sites, and sites with a pile of “Discovered – currently not indexed” URLs in GSC. If that’s not you, leave crawl rate alone and let Google’s automation do its job.

AI summary

A condensed take on the Advanced version:

• Crawl rate = how fast a crawler fetches pages — the “crawl capacity limit” (parallel connections + delay between fetches). It’s the supply side of crawl budget; crawl demand is the demand side.
• It’s automatic and reacts to server health: fast/healthy server → higher limit; slow responses or 5xx/429 → Google crawls less. Google’s own capacity is a second ceiling you don’t control. All Googlebots share one crawl pool.
• The GSC manual rate slider was removed Jan 8, 2024. Advice telling you to “adjust the slider” is outdated.
• To slow Googlebot today: fix the server (best), or return 500/503/429 for 1–2 days max (longer risks de-indexing + Ads pausing). Prefer 429 (carries Retry-After). Never use 401/403/404, and crawl-delay is ignored by Google.
• You can’t request an increase. Improve it indirectly: faster server, clean sitemaps, fewer duplicate/junk URLs, more internal + external links, Indexing API where eligible.
• Bing kept manual control: Crawl Control hourly grid + crawl-delay honored.
• Crawl rate is not a ranking factor — “improving your crawl rate won’t necessarily lead to better positions.” Most sites never need to manage it.

Official documentation

Primary-source documentation from the search engines.

Google

• Reduce the Googlebot crawl rate — the headline how-to: 500/503/429, the 1–2 day limit, and the request form (you can lower, not raise).
• Crawl Budget Management — defines the crawl capacity limit and how it reacts to server health.
• How HTTP Status Codes Affect Google’s Crawlers — exactly which codes throttle crawling (5xx/429) and which don’t (4xx except 429).
• Myths and facts about crawling — crawl-delay not processed; crawl rate ≠ rankings; server health affects crawling.
• Upcoming deprecation of Crawl Rate Limiter Tool (Nov 2023) — the deprecation announcement.
• Don’t use 403s or 404s for rate limiting (Feb 2023) — why 4xx is the wrong lever.
• New and improved crawl stats for your site (Nov 2020) — how to read the Crawl Stats report.
• Optimize your crawl budget — capacity + demand, and who actually needs it.

Bing / Microsoft

• Bing Webmaster Tools — Crawl Control — the manual hour-by-hour crawl-rate grid Google retired.
• Bingbot guidance — current Bing Webmaster guidance documents crawl-delay values from 1–20 seconds.
• bingbot Series: Optimizing Crawl Frequency (Oct 2018) — Bing’s take on when to re-fetch.

Quotes from the source

On-the-record statements from Google. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what crawl rate is and what sets it

• “If the site responds quickly for a while, the limit goes up, meaning more connections can be used to crawl. If the site slows down or responds with server errors, the limit goes down and Google crawls less.” — Crawl Budget Management. Jump to quote
• “Google has a lot of machines, but not infinite machines. We still need to make choices with the resources that we have.” Jump to quote

Google — how to reduce the crawl rate

• “return 500, 503, or 429 HTTP response status code instead of 200 to the crawl requests.” — Reduce the Googlebot crawl rate. Jump to quote
• “We don’t recommend that you do this for a long period of time (meaning, longer than 1-2 days).” Jump to quote
• “You cannot request an increase in crawl rate, and it may take several days for the request to be evaluated and fulfilled.” Jump to quote

Google — which status codes throttle crawling

• “Google’s crawlers treat the 429 status code as a signal that the server is overloaded, and it’s considered a server error.” — How HTTP Status Codes Affect Google’s Crawlers. Jump to quote
• “The 4xx status codes, except 429, have no effect on crawl rate.” / “Don’t use 401 and 403 status codes for limiting the crawl rate.” Jump to quote

Google — crawl-delay and rankings

• “The non-standard ‘crawl-delay’ robots.txt rule is not processed by Google’s crawlers.” — Myths and facts about crawling. Jump to quote
• “Improving your crawl rate won’t necessarily lead to better positions in Google Search results.” Jump to quote

Gary Illyes, Google (on the crawl-rate tool deprecation)

• “if the server persistently returns HTTP 500 status codes for a range of URLs, Googlebot will automatically, and almost immediately slow down crawling.” Read the coverage
• “with the improvements we’ve made to our crawling logic and other tools available to publishers, its usefulness has dissipated.” Read the coverage

Crawl-rate checklists

Safely reduce Googlebot’s crawl rate (in order)

• First, confirm the crawler is the real problem — check GSC Crawl Stats (average response time, host status) and your server logs, not just a hunch.
• Fix the root cause: speed up the server or add resources. This is the durable fix and the only one that doesn’t risk indexing.
• If it’s a genuine emergency, return 500/503/429 to crawl requests (prefer 429 with a Retry-After). Googlebot slows almost immediately.
• Keep the emergency response on for 1–2 days at most — longer risks pages being dropped from the index and Ads being paused.
• For an ongoing (non-fire) issue, file Google’s special crawl-rate request, naming the optimal rate for your site.
• For Bing, set the speed in Crawl Control and/or add crawl-delay to robots.txt.

Do NOT do these

• Don’t use 401/403/404 to throttle — no effect on rate, and you risk losing pages.
• Don’t rely on robots.txt crawl-delay for Google — it’s ignored (Bing honors it).
• Don’t look for the old GSC crawl-rate slider — it was removed Jan 8, 2024.

When you want more crawling (you can’t force it — improve it indirectly)

• Speed up / add resources to the server.
• Keep canonical, indexable URLs in clean sitemaps with accurate lastmod.
• Remove duplicate and low-value URLs that waste crawling.
• Strengthen internal links and earn more external links.
• Use GET instead of POST where possible; use the Indexing API where eligible.

Crawl rate — cheat sheet

Status codes and what they do to Google’s crawl rate

Fast facts

• Crawl rate = how fast (supply); crawl demand = how much they want (demand); crawl budget = the two together.
• Google’s term: crawl capacity limit — parallel connections + delay between fetches. Automatic, tracks server health.
• Emergency 5xx/429 window: 1–2 days max — longer risks de-indexing + Ads pausing.
• No manual increase — you can only ask Google to lower, never raise.
• Manual GSC crawl-rate slider: removed Jan 8, 2024 (announced Nov 24, 2023).
• All Googlebots share one crawl pool (search, images, ads, etc.).
• Crawl rate is not a ranking factor.
• Bing equivalent: Crawl Control grid + crawl-delay honored.

Temporarily tell Googlebot to slow down

This is the emergency lever only — return a 503 (or 429) with a Retry-After header to the crawler instead of a 200. Google reads it as “slow down” almost immediately. Keep it on for a day or two at most: leave it longer and the affected URLs may be dropped from the index (and any Google Ads pointing at them may be paused). The right long-term fix is a faster, healthier server, not a permanent error response.

Apache (.htaccess) — return 503 with Retry-After

# Emergency only — remove within 1–2 days.
# Sends Googlebot a "slow down / try later" signal.
RewriteEngine On
RewriteCond %{HTTP_USER_AGENT} (Googlebot|bingbot) [NC]
RewriteRule ^ - [R=503,L]
Header always set Retry-After "3600"
ErrorDocument 503 "Server temporarily overloaded — please retry later."

Nginx — return 503 with Retry-After for crawlers

# Emergency only — remove within 1–2 days.
if ($http_user_agent ~* (Googlebot|bingbot)) {
    return 503;
}
# Send a Retry-After hint with the 503 response.
add_header Retry-After 3600 always;

Express / Node.js — 429 Too Many Requests with Retry-After

// Emergency only — remove within 1–2 days.
// 429 explicitly means "too many requests" and carries a Retry-After.
app.use((req, res, next) => {
  const ua = req.get("user-agent") || "";
  if (/Googlebot|bingbot/i.test(ua)) {
    res.set("Retry-After", "3600"); // seconds
    return res.status(429).send("Too many requests — please retry later.");
  }
  next();
});

Reminder: this only slows crawling temporarily. It is not a way to pause or block indexing, and it is not a substitute for fixing server capacity.

Tools for seeing and controlling crawl rate

• Google Search Console — Crawl Stats report — Google’s own view: total crawl requests over time, total download size, average response time, host status (site availability to Google over ~90 days), and breakdowns by response code, file type, crawl purpose, and Googlebot type. This is where you’ll see throttling.
• Bing Webmaster Tools — Crawl Control — the manual hour-by-hour grid (presets or custom-drawn) for setting Bingbot’s speed by time of day. The manual lever Google retired.
• robots.txt crawl-delay — honored by Bing (and some other engines), ignored by Google. Useful for Bing, useless for Google.
• Server log file analysis — the ground truth for how fast bots are actually hitting you and what status codes they’re getting. (See log file analysis.)
• Site audits / crawlers — Ahrefs Site Audit and Screaming Frog SEO Spider to find the duplicate, parameterized, and trap-like URLs that waste crawling.

What should you do about crawl rate?

Incident playbook: crawler traffic is overloading the origin

1. Verify the bot. Confirm the source IP belongs to the claimed crawler. If not, block or rate-limit the impersonator through normal security controls.
2. Measure impact. Correlate crawler requests with latency, saturation, timeouts, and 5xx responses. If they do not align, investigate the actual load source.
3. Protect availability. Shed only the necessary traffic; for a temporary Googlebot emergency, use 429 or 503, not 403 or 404.
4. Find the hot patterns. Group requests by directory, parameters, response code, and bytes. If a runaway URL space dominates, fix its links or generation rules.
5. Repair the cause. Increase capacity, cache safe responses, remove crawler traps, or use the specific crawler’s supported controls.
6. Restore and verify. Remove temporary throttling, then confirm user latency and crawler error rates return to the site’s baseline.

Crawl-rate mistakes

• Using crawl-delay for Googlebot. Google ignores it. Use temporary 429/503 responses only for an emergency and fix the underlying load.
• Returning 403 or 404 to slow Googlebot. Those statuses communicate access or absence, not temporary overload. Use the correct temporary signal.
• Trying to force a permanent increase. The retired Search Console slider is gone, and more crawling does not improve rankings. Improve server health and demand signals.
• Trusting the user-agent string. Spoofers can claim to be Googlebot. Verify the IP before changing site behavior.
• Leaving emergency throttling in place. Long-running errors can harm crawling and indexing. Set an owner and removal condition before deployment.

The verify → protect → repair framework

• Verify: prove the traffic is a real crawler and that it correlates with server harm.
• Protect: use the narrowest temporary control that preserves user availability and communicates the right HTTP meaning.
• Repair: remove the capacity bottleneck or runaway URL space, then withdraw the temporary control.

Keep crawl rate separate from crawl demand: a healthy server can raise the capacity ceiling, but it cannot force a search engine to want more URLs.

Prove a crawl-rate intervention worked

Temporary throttle response

Test to run: request a throttled test URL with curl -I. Expected result: the planned 429 or 503 appears only during the incident and normal URLs remain available. Failure interpretation: the rule is mis-scoped or the wrong status is being returned. Monitoring window: immediate. Rollback trigger: users or unrelated bots receive the throttle unexpectedly.

Recovery after removing the throttle

Test to run: repeat header checks and monitor server plus access logs. Expected result: normal 200 responses return, crawler errors decline, and user latency stays at baseline. Failure interpretation: the temporary rule remains active or the capacity problem persists. Monitoring window: immediate for HTTP behavior; continue through the next normal crawl window. Rollback trigger: renewed saturation or 5xx errors require returning to the incident plan.

URL-space repair

Test to run: crawl and log-check the parameter or path pattern that caused excess requests. Expected result: new trap URLs are no longer generated or linked, while valuable URLs remain reachable. Failure interpretation: another discovery path still exposes the pattern. Monitoring window: compare equivalent log windows. Rollback trigger: valuable pages or required resources become inaccessible.

Metrics for crawl-rate health

Verified crawler request rate

Metric: requests per time unit from verified crawler IPs. What it tells you: actual crawler pace. How to pull it: access logs after bot verification. Benchmark / realistic range: establish a baseline by crawler and traffic period; no universal safe rate exists. Cadence: daily during incidents, monthly otherwise.

Crawler-correlated error and latency rate

Metric: 5xx/timeouts and origin latency during crawler activity. What it tells you: whether rate exceeds capacity. How to pull it: align server telemetry with crawler log timestamps. Benchmark / realistic range: use the site’s normal non-incident baseline and capacity objectives. Cadence: continuous alerting for critical sites.

Useful-request share

Metric: verified crawler requests to valuable 200 pages versus redirects, errors, and known trap URLs. What it tells you: whether capacity is spent productively. How to pull it: classify log URLs and statuses. Benchmark / realistic range: trend against the site’s own inventory; avoid a universal target. Cadence: monthly.

Test yourself: Crawl rate

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — crawl rate as one half of crawl budget, plus the reduce/increase levers.
• What Is Googlebot & How Does It Work? — how Googlebot decides how fast and what to crawl (and why “change crawl rate” is now deprecated).
• The Beginner’s Guide to Technical SEO — where crawling and crawl rate fit in the bigger picture.

My speaking

• How Search Works (SlideShare) — where I frame the crawl rate limit as “what your site can support” and note all Googlebots share one crawl pool. (Standing disclaimer: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Search Engine Land — Googlebot crawl rate tool is going away and the tool is now gone — the deprecation, with Illyes’ reasoning.
• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• Search Engine Journal — Google Removing Crawl Rate Limiter Tool From Search Console — Roger Montti’s relay of the Nov-2023 deprecation with the new auto-set minimum crawl speed detail.
• Search Engine Journal — Don’t Use 403/400 Error Responses For Rate Limiting Googlebot — covers Google’s Feb-2023 blog post warning against using 4xx to throttle crawling.
• Bing Webmaster Blog — bingbot Series: Optimizing Crawl Frequency — Bing’s own explanation of when and how Bingbot decides to re-fetch; useful counterpoint to Google’s automatic approach.
• Bing Webmaster Blog — Getting most of Bingbot via Bing Webmaster Tools — how to use Crawl Control and other BWT settings to manage Bingbot’s crawl rate.

Crawl rate

Crawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.

Related: Crawl Budget, Crawling

Crawl rate

Crawl rate is how fast a search engine crawler requests pages from your server — the number of simultaneous parallel connections it opens and the time it waits between fetches. Google calls its version the crawl capacity limit, and it’s one of the two halves of crawl budget; the other half is crawl demand (how much an engine actually wants to crawl your pages).

You don’t set crawl rate by hand anymore. Googlebot raises or lowers it automatically based on how your server responds: if the site answers quickly, the limit goes up; if it slows down or returns server errors, Google crawls less. Google retired its manual crawl-rate slider in Search Console (removed January 8, 2024), so today the only reliable way to signal “slow down” is your server’s HTTP responses — returning 500, 503, or 429 for short emergencies. You can’t request a permanent increase. Bing still offers a manual Crawl Control grid in Bing Webmaster Tools.

Crawl rate is distinct from both crawl budget and rankings. A higher crawl rate doesn’t lift your positions — crawling is required to appear in search, but it isn’t a ranking signal. For most sites, crawl rate is a server-health and efficiency question, not something to manage.

Related: Crawl Budget, Crawling