Crawl Frequency

TL;DR — Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. is how often a search engine comes back to re-check a page it already knows about. Pages that are popular and change a lot get re-checked constantly; pages that never change get re-checked rarely — those are the two biggest factors, though not the only ones. You can’t set it — you nudge it by making a page more important and actually updating it.

What crawl frequency means

When Google or Bing finds a page for the first time, that’s a discovery crawl. But the web changes, so search engines come back later to see if the page is different — that’s a refresh crawl. Crawl frequency is how often that re-checking happens. Evidence for this claim Google's crawl-demand guidance says popular URLs tend to be crawled more often and systems seek to recrawl often enough to detect changes. Scope: Adaptive Google recrawling; no fixed per-page cadence is promised. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

It’s not the same for every page. A busy news homepage might get re-crawled every couple of hours. A small business “About” page that hasn’t changed in two years might go months between crawls. Those are illustrative examples, not a published schedule — Google doesn’t publish fixed hour/day/week intervals by page type. The search engine decides per page, dynamically.

What makes a page get crawled more often

Two things matter most:

1. How important the page is. Popular pages — ones with lots of links pointing at them — get re-crawled more often so Google keeps them fresh.
2. How often the page actually changes. Search engines learn your pattern. If a page updates every day, they’ll start checking it daily. If it never changes, they back off and check it less and less.

Those two are the biggest levers, but they’re not the whole list. Google’s own documentation also names things like how many URLs it thinks you have (perceived inventory) and site-wide events — a site migrationA site migration is any significant change to a website's URL structure, domain, platform, protocol, or hosting that can affect how search engines crawl, index, and rank it. The risk scales with how much you change at once., for example — as inputs that can shift crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. up or down.

What does not make Google crawl more

This is where people get tripped up:

• Publishing every day doesn’t force faster crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. by itself. Google crawls more when your pages are important and genuinely change — not just because you hit publish a lot.
• Tags in your sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. don’t control it. There are old settings called changefreq and priority — Google ignores both completely. Evidence for this claim Google ignores sitemap priority and changefreq values and may use accurate lastmod values. Scope: Google sitemap processing. Confidence: high · Verified: 2026-07-14 Google: Build and submit a sitemap
• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. more often does not help your rankings. A page has to be crawled to rank at all, but getting crawled more won’t move you up.

The honest answer to “how do I get crawled more?”

There’s no button. What actually helps: link to the page from your important pages, keep your site fast and error-free, and make real, meaningful updates (not just changing the year in your footer). If you’ve changed something important and want Google to take a fresh look at one specific page, you can request it in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version..

Want the version with the Google and Bing quotes, the lastmod details, and how this differs from crawl budgetThe number of URLs an engine will crawl in a timeframe. and crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.? Switch to the Advanced tab.

TL;DR — Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. is the cadence of recrawl for a known URL, set by the scheduler mainly from two inputs: importance (popularity / PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems. / links) and staleness (how often the page genuinely changes) — Google’s documentation names other demand inputs too, such as perceived inventoryCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. and site-wide events. Google learns your per-page update pattern and adapts — even backing off on stable pages (3 → 10 → 30 → 100 days). changefreq and priority are ignored; only a verifiable lastmod tied to a significant change is honored. You can’t set frequency directly — you improve the inputs (importance, real change, accurate lastmod, server health) and, for a single URL, request a recrawl. More crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. does not improve rankings. Distinct from crawl rate (speed) and crawl budget (demand + capacity).

What crawl frequency actually is

Crawl frequency is how often a search engine re-fetches a URL it already knows about, to check whether it changed. It’s the cadence of recrawl, decided per URL by the crawl scheduler. The two big inputs are how important the page is and how often it really changes — Google’s documentation calls these popularity and staleness. Evidence for this claim Google's crawl-demand guidance says popular URLs tend to be crawled more often and systems seek to recrawl often enough to detect changes. Scope: Adaptive Google recrawling; no fixed per-page cadence is promised. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

This is the piece of the crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. story that’s easiest to blur into its siblings, so let me draw the lines clearly.

Crawl frequency vs. crawl budget vs. crawl rate

These three get used interchangeably and they shouldn’t:

Google’s own definition ties it together: “Google defines a site’s crawl budgetThe number of URLs an engine will crawl in a timeframe. as the set of URLs that Google can and wants to crawl.” Frequency is a per-URL cadence that lives inside that budget. Rate is the throttle on the pipe. As I put it in my crawl-budget guide, “Crawl budget is the amount of time and resources a search engine allows for crawling a website. It is made up [of] crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. which is how many pages a search engine wants to crawl on your site and crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. which is how fast they can crawl.”

Discovery crawls vs. refresh crawls

Frequency is really about refresh crawling. John Mueller laid out the split plainly: “One is a discovery crawl where we try to discover new pages on your website. And the other is a refresh crawl where we update existing pages that we know about.”

Refresh cadence varies enormously by page. Mueller again: “We would refresh crawl the homepage, I don’t know, once a day, or every couple of hours, or something like that.” And on the other end: “If we recognize that individual pages change very rarely, then we realize we don’t have to crawl them all the time.”

The key insight is that Google learns your pattern per page: “If you have a news website and you update it hourly, then we should learn that we need to crawl it hourly. Whereas if it’s a news website that updates once a month, then we should learn that we don’t need to crawl every hour.”

What determines how often Google recrawls a page

In my How Search Works deck I list the crawl-demand factors that drive recrawl: PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems., how frequently the page changes, time since it was last crawled, and major site changes. That’s the same popularity + staleness model Google documents, just from my own framing — and it’s not an exhaustive list either. Google’s documentation also names perceived inventory (how many URLs it thinks your site has) and site-wide events, such as a domain or URL-structure migration, as things that can move crawl demand up or down. Popularity and staleness are the two you can influence most directly, so they’re the ones worth the most attention. Breaking those down:

Popularity, PageRank, and links

“URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.” More internal and external links to a page → more perceived importance → more frequent recrawls. In my crawl-budget post I say it the same way: “Popular pages, or those with more links and PageRank, will generally receive priority over other pages.”

Staleness — and the backoff

Google’s systems “want to recrawl documents frequently enough to pick up any changes.” The flip side is that pages which don’t change get crawled less and less. From my crawl-budget guide: “If Google sees that a page isn’t changing, they will crawl the page less frequently.” There’s no fixed interval — it’s a backoff: “if they crawl a page and see no changes after a day, they may wait three days before crawling again, ten days the next time, 30 days, 100 days, etc.”

Quality and search demand

Gary Illyes frames the scheduler as something you have to convince: “If you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.” It’s dynamic: “Scheduling is very dynamic. As soon as we get the signals back from search indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. that the quality of the content has increased across this many URLs, we would just start turning up demand.” It cuts both ways — “If search demand goes down, then that also correlates to the crawl limit going down.”

Worth knowing: Google is actively trying to crawl stable pages less for efficiency reasons. Illyes has publicly described wanting to “crawl even less” and reduce bytes on the wire. So don’t expect the scheduler to err toward over-crawling your unchanging pages.

Server health (rate enables frequency)

Frequency can only go as high as your rate allows. The crawl capacity limit is roughly the maximum number of simultaneous parallel connections Google will use; a fast, error-free server raises that ceiling, while a slow site or one returning 5xx/429 gets crawled less. Server health doesn’t increase frequency — it just stops being a bottleneck.

The role of sitemaps and lastmod

Here’s the biggest myth to bust. People think sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. settings control cadence. They mostly don’t.

changefreq and priority are ignored

Straight from Google’s sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. docs: “Google ignores <priority> and <changefreq> values.” Setting <changefreq>hourly</changefreq> does nothing. Evidence for this claim Google ignores sitemap priority and changefreq values and may use accurate lastmod values. Scope: Google sitemap processing. Confidence: high · Verified: 2026-07-14 Google: Build and submit a sitemap Stop optimizing those fields.

lastmod is the one signal — if it’s honest

The single freshness signal Google honors from a sitemap is lastmod, and only conditionally: “Google uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.” If you lie about it, Google notices and stops trusting it.

And it has to reflect a real change: “The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., or links on the page is generally considered significant, however an update to the copyright date is not.” Bumping lastmod because you changed the footer year won’t help — and erodes the trust that makes lastmod work at all.

Can you force or speed up crawling?

The honest answer: there’s no dial for frequency. What you actually have:

• Improve the inputs — importance (links, internal linkingAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them.), genuine content changes, an accurate lastmod, and a fast, healthy server.
• Request a single URL — Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s URL Inspection has a “Request indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.” action for one page at a time. It’s a request, not a guarantee, and it doesn’t change ongoing cadence. Google is explicit that mashing the button doesn’t help either: requesting the same URL repeatedly will not get it crawled faster.
• Push a change (on Bing and others) — see below.

What you can’t do: set a frequency, force daily crawls by publishing daily, or use the old GSC crawl-rate slider (it was retired). Bing has a Crawl Control grid, but that controls rate, not frequency.

Bing: adaptive crawling and IndexNow

Bing thinks of crawl frequency as a cost problem. From their crawl-frequency post, the cadence “depends on the frequency of which the content is edited and updated,” and “Defining when to fetch the web page next is the hard problem we are looking to optimize.” Their adaptive answer: “What we learned was that we could optimize our system to avoid fetching the same content over and over, and instead check periodically for major changes” — which in one case yielded “about 40% crawl saving on this site!”

Bing’s “you can nudge it” mechanism is IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it.: instead of waiting for the scheduler, you signal a change. “Whether you’re adding, updating, or deleting content, IndexNow notifies multiple search engines of your content changes as soon as they happen,” which is about “limiting the need for costly exploratory crawls.” Note the asymmetry: Google does not use IndexNow for general recrawling.

How to check your crawl frequency

• GSC → Crawl Stats — total requests over time, broken down by response code, file type, and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type. This is how you observe cadence at the site level; it’s not a per-URL last-crawl report.
• GSC → URL Inspection — the “last crawl” date for a single URL. This, not Crawl Stats, is where you check when a specific page was last fetched.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Crawl Control (rate) and IndexNow Insights.
• Server log analysis — the ground truth: exactly which URLs bots hit and how often.

Common myths about crawl frequency

• “Publishing daily forces faster crawling.” No — Google learns your pattern and crawls more for importance and genuine change, not publishing volume.
• “changefreq/priority control cadence.” No — both are ignored.
• “Just bump lastmod to get recrawled.” Only works if it’s verifiable and tied to a significant change; gaming it makes Google distrust it.
• “More crawling means better rankings.” No. As I’ve written in my crawl budget guide, “The rate of crawling isn’t going to impact your rankings.” Crawling is a prerequisite, not a boost — “More crawling doesn’t mean you’ll rank better, but if your pages aren’t crawled and indexed they aren’t going to rank at all.”
• “There’s a fixed schedule.” No — it’s dynamic, per-URL, and adaptive.
• “I can set crawl frequency in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance..” No — the rate slider is gone, and there’s no frequency control.

AI summary

A condensed take on the Advanced version:

• Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. = cadence of recrawl for a URL Google already knows. Driven mainly by popularity (links / PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems.) and staleness (how often the page really changes) — Google’s documentation also names perceived inventoryCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. and site-wide events as demand inputs.
• Google learns your per-page pattern and adapts — hourly for a busy homepage, rarely for a static page, backing off (3 → 10 → 30 → 100 days) on pages that don’t change.
• Distinct from siblings: rate = how fast (server-throttled), budget = demand + capacity (“URLs Google can and wants to crawl”). Frequency lives inside budget.
• SitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. reality: changefreq and priority are ignored; only a verifiable lastmod tied to a significant change is honored.
• You can’t set frequency. Improve the inputs (importance, genuine change, honest lastmod, server health); request a single URL via GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version.; push changes via IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. on Bing (not Google).
• More crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ better rankings. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is required to rank, not a ranking signal.
• Observe it in GSC Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). (site-level requests over time — not a per-URL log), URL Inspection’s last-crawl date (the per-URL check), Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility., or server logs.

Official documentation

Primary-source documentation on recrawl cadence.

Google

• Optimize your crawl budget — crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. (perceived inventoryCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side., popularity, staleness) and crawl capacityThe number of URLs an engine will crawl in a timeframe.; the model behind recrawl frequency.
• Build and submit a sitemap — why changefreq/priority are ignored and how lastmod is actually used.
• In-Depth Guide to How Google Search Works — the algorithmic crawl scheduler (“which sites to crawl, how often, and how many pages”).

Bing / Microsoft

• bingbot Series: Optimizing Crawl Frequency — Bing’s framing of frequency as a content-change-driven scheduling problem.
• Bing Webmaster Tools — Crawl Control — set BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s hourly rate (rate, not frequency).
• IndexNow — the push protocol for signaling changed URLs instead of waiting for a recrawl.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what drives recrawl cadence

• “URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.” — Google Search Central docs. Jump to quote
• “Our systems want to recrawl documents frequently enough to pick up any changes.” Jump to quote
• “Google defines a site’s crawl budgetThe number of URLs an engine will crawl in a timeframe. as the set of URLs that Google can and wants to crawl.” Jump to quote

Google — sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., changefreq, lastmod

• “Google ignores <priority> and <changefreq> values.” Jump to quote
• “Google uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.” Jump to quote
• “The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., or links on the page is generally considered significant, however an update to the copyright date is not.” Jump to quote

John Mueller, Google (SEO office-hours, Jan 2022)

• “One is a discovery crawl where we try to discover new pages on your website. And the other is a refresh crawl where we update existing pages that we know about.” Jump to quote
• “We would refresh crawl the homepage, I don’t know, once a day, or every couple of hours, or something like that.” Jump to quote
• “If you have a news website and you update it hourly, then we should learn that we need to crawl it hourly. Whereas if it’s a news website that updates once a month, then we should learn that we don’t need to crawl every hour.” Jump to quote

Gary Illyes, Google (crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. priorities)

• “If you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.” Jump to quote
• “Scheduling is very dynamic. As soon as we get the signals back from search indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. that the quality of the content has increased across this many URLs, we would just start turning up demand.” Jump to quote

Microsoft Bing — Optimizing Crawl FrequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.

• “The answer depends on the frequency of which the content is edited and updated.” Jump to quote
• “Defining when to fetch the web page next is the hard problem we are looking to optimize with your help.” Jump to quote
• “Whether you’re adding, updating, or deleting content, IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. notifies multiple search engines of your content changes as soon as they happen.” — IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it.. Jump to quote

The mental models

1. Frequency = popularity × staleness. Two inputs drive how often a known URL gets refreshed: how important it is (popularity / PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems. / links) and how often it genuinely changes (staleness). A page that’s both important and changes often gets crawled most; one that’s neither gets crawled least. Everything else is a downstream lever on these two.

2. Discovery crawl vs. refresh crawl. A discovery crawl finds a new URL once. A refresh crawl re-checks a known URL — and frequency is entirely about refresh. If a page isn’t getting recrawled, ask whether it’s an importance problem or a “Google thinks nothing changes here” problem; the fix is different for each.

3. Frequency ≠ rate ≠ budget. Frequency is cadence (how often). Rate is speed (how fast, server-throttled). Budget is demand + capacity (the URL set Google can and wants to crawl). Frequency lives inside budget; rate caps how much frequency is even possible.

4. The scheduler learns your pattern. Google adapts per page to your real update cadence and backs off on stable pages (3 → 10 → 30 → 100 days). You don’t beat the scheduler by publishing more — you change what it learns by actually being important and actually changing.

5. The one honest sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. signal. changefreq and priority do nothing. A verifiable lastmod tied to a significant change is the only freshness signal that counts — and only as long as you don’t lie about it.

What actually increases recrawl frequency

A pass over the levers that move cadence — and the traps that don’t:

• Make the page more important. Add internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. from strong pages; earn external links. Popularity is the biggest frequency lever.
• Make real changes. Update main content, structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding., or links — not cosmetic edits. Google learns whether a page genuinely changes.
• Keep lastmod accurate and verifiable. It must reflect the last significant update; a copyright-year bump doesn’t count and erodes trust.
• Don’t bother with changefreq / priority — Google ignores both.
• Keep the server fast and error-free. 5xx/429/timeouts lower your crawl capacityThe number of URLs an engine will crawl in a timeframe., which caps frequency.
• Request a single URL when it matters — GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. → Request indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. for one page after a meaningful update.
• On Bing/Yandex, use IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. to push change notifications instead of waiting for the scheduler (Google doesn’t use it).
• Don’t expect publishing volume alone to help — frequency follows importance and genuine change, not how often you hit publish.
• Check Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). for last-crawl dates and discovery-vs-refresh split before assuming a frequency problem exists.

Tools for seeing crawl frequency

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). — requests over time, by response code, file type, and GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. type. The cleanest site-level view of how often Google comes back overall — not a per-URL log.
• URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. (GSC) — the “last crawl” date for a single URL, plus the Request indexing action when you’ve made a meaningful change. Repeated requests for the same URL don’t speed it up.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — Crawl Control (set BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share.’s hourly rate) and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. Insights for the push side.
• Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — the ground truth on exactly which URLs bots hit and how often. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysis.)
• Ahrefs Site Audit / Webmaster Tools — simulate a crawl and surface the importance signals (internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., depth) that feed frequency.

Crawl-frequency mistakes

• Changing sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lastmod without changing the page. False dates make the signal less useful. Update lastmod only for meaningful content changes.
• Relying on changefreq or priority. Google ignores those sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. hints. Use strong discovery, importance, and honest modification signals instead.
• Requesting indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. repeatedly for an unchanged page. Another fetch does not create value or force indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Improve the page or its signals first.
• Treating frequent crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. as a ranking win. Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. describes fetching, not quality or position. Measure whether important changes are picked up in time.
• Forcing cosmetic edits on a schedule. Search engines learn real change patterns. Make useful updates rather than changing dates or whitespace.

Crawl-frequency signal cheat sheet

Metrics for crawl frequency

Median recrawl interval by template

Metric: median time between verified crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. fetches for the same URL. What it tells you: the learned revisit cadence. How to pull it: sort access-log requests by normalized URL and timestamp, then segment by template. Benchmark / realistic range: compare with each template’s actual change cadence; no sitewide target fits both news and evergreen pages. Cadence: monthly.

Change-to-recrawl lag

Metric: time between a meaningful publish/update event and the next fetch. What it tells you: whether important changes are discovered promptly. How to pull it: join CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms. or deployment timestamps with access logsLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened.. Benchmark / realistic range: establish a per-template baseline and investigate regressions. Cadence: monthly and after sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing./internal-link changes.

Honest-lastmod rate

Metric: sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lastmod changes that correspond to meaningful page changes. What it tells you: whether the freshness signal remains trustworthy. How to pull it: compare sitemap history with content hashes or release records. Benchmark / realistic range: every changed date should be explainable by a meaningful update. Cadence: each sitemap release or weekly sampling.

Test yourself: Crawl frequency

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — the closest companion piece: demand, rate, staleness backoff, and why frequency isn’t a ranking factor.
• What Is Googlebot & How Does It Work? — the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. mechanics behind the scheduler.
• Crawl Me Maybe? How Website Crawlers Work — a general crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. primer.
• The Beginner’s Guide to Technical SEO — where crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. fits in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of the crawl-demand factors (PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems., change frequency, time since last crawl, major site changes). (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• Google Has Two Types Of Crawling – Discovery & Refresh (Search Engine Journal) — John Mueller’s verbatim quotes on discovery vs. refresh crawls and how Google learns per-page update patterns.
• Google’s Crawling Priorities: Insights From Analyst Gary Illyes (Search Engine Journal) — Gary Illyes on the dynamic scheduler, quality signals driving crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side., and convincing Google your content is worth fetching.
• Google Considers Reducing Webpage Crawl Rate (Search Engine Journal) — coverage of Google’s stated goal to crawl stable pages less, corroborating the backoff pattern.
• bingbot Series: Maximizing Crawl Efficiency (Bing WebmasterMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. Blog) — Bing’s companion piece on crawl efficiency; pairs with the Optimizing Crawl FrequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. post in the Official Docs tab.
• r/TechSEO — the community for crawl/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. debugging.

Crawl frequency

Crawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.

Related: Crawl Budget, Crawling

Crawl frequency

Crawl frequency is the cadence of recrawl — how often a search engine refreshes a URL it has already discovered to check whether the page has changed. A news homepage might be refresh-crawled every couple of hours; an “About” page that never changes might be crawled once every few months. The scheduler decides this per URL, dynamically, based largely on a page’s perceived importance (PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems., links, popularity) and how often it actually changes (staleness).

It’s easy to confuse with two siblings:

• Crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. (crawl capacityThe number of URLs an engine will crawl in a timeframe.) is how fast a crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. hits your server — parallel connections and the delay between fetches — throttled by your server’s health. That’s speed, not cadence.
• Crawl budgetThe number of URLs an engine will crawl in a timeframe. is the combination of crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. and crawl capacity: “the set of URLs that Google can and wants to crawl.” Frequency is a per-URL cadence that lives inside that budget.

You can’t set crawl frequency directly. There’s no dial in Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.. What you can do is improve the inputs: make a page more important (internal and external links), genuinely change its content, keep an accurate lastmod, and keep your server fast and error-free. Google ignores <changefreq> and <priority> in sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. entirely, and only trusts lastmod when it’s verifiably accurate and tied to a significant change. CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. more often does not improve rankings — crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is a prerequisite to ranking, not a boost.

Related: Crawl Budget, Crawling

Changelog