Crawl Frequency

TL;DR — Crawl frequency is how often a search engine comes back to re-check a page it already knows about. Pages that are popular and change a lot get re-checked constantly; pages that never change get re-checked rarely — those are the two biggest factors, though not the only ones. You can’t set it — you nudge it by making a page more important and actually updating it.

What crawl frequency means

When Google or Bing finds a page for the first time, that’s a discovery crawl. But the web changes, so search engines come back later to see if the page is different — that’s a refresh crawl. Crawl frequency is how often that re-checking happens. Evidence for this claim Google's crawl-demand guidance says popular URLs tend to be crawled more often and systems seek to recrawl often enough to detect changes. Scope: Adaptive Google recrawling; no fixed per-page cadence is promised. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

It’s not the same for every page. A busy news homepage might get re-crawled every couple of hours. A small business “About” page that hasn’t changed in two years might go months between crawls. Those are illustrative examples, not a published schedule — Google doesn’t publish fixed hour/day/week intervals by page type. The search engine decides per page, dynamically.

What makes a page get crawled more often

Two things matter most:

1. How important the page is. Popular pages — ones with lots of links pointing at them — get re-crawled more often so Google keeps them fresh.
2. How often the page actually changes. Search engines learn your pattern. If a page updates every day, they’ll start checking it daily. If it never changes, they back off and check it less and less.

Those two are the biggest levers, but they’re not the whole list. Google’s own documentation also names things like how many URLs it thinks you have (perceived inventory) and site-wide events — a site migration, for example — as inputs that can shift crawl demand up or down.

What does not make Google crawl more

This is where people get tripped up:

• Publishing every day doesn’t force faster crawling by itself. Google crawls more when your pages are important and genuinely change — not just because you hit publish a lot.
• Tags in your sitemap don’t control it. There are old settings called changefreq and priority — Google ignores both completely. Evidence for this claim Google ignores sitemap priority and changefreq values and may use accurate lastmod values. Scope: Google sitemap processing. Confidence: high · Verified: 2026-07-14 Google: Build and submit a sitemap
• Crawling more often does not help your rankings. A page has to be crawled to rank at all, but getting crawled more won’t move you up.

The honest answer to “how do I get crawled more?”

There’s no button. What actually helps: link to the page from your important pages, keep your site fast and error-free, and make real, meaningful updates (not just changing the year in your footer). If you’ve changed something important and want Google to take a fresh look at one specific page, you can request it in Google Search Console’s URL Inspection tool.

Want the version with the Google and Bing quotes, the lastmod details, and how this differs from crawl budget and crawl rate? Switch to the Advanced tab.

TL;DR — Crawl frequency is the cadence of recrawl for a known URL, set by the scheduler mainly from two inputs: importance (popularity / PageRank / links) and staleness (how often the page genuinely changes) — Google’s documentation names other demand inputs too, such as perceived inventory and site-wide events. Google learns your per-page update pattern and adapts — even backing off on stable pages (3 → 10 → 30 → 100 days). changefreq and priority are ignored; only a verifiable lastmod tied to a significant change is honored. You can’t set frequency directly — you improve the inputs (importance, real change, accurate lastmod, server health) and, for a single URL, request a recrawl. More crawling does not improve rankings. Distinct from crawl rate (speed) and crawl budget (demand + capacity).

What crawl frequency actually is

Crawl frequency is how often a search engine re-fetches a URL it already knows about, to check whether it changed. It’s the cadence of recrawl, decided per URL by the crawl scheduler. The two big inputs are how important the page is and how often it really changes — Google’s documentation calls these popularity and staleness. Evidence for this claim Google's crawl-demand guidance says popular URLs tend to be crawled more often and systems seek to recrawl often enough to detect changes. Scope: Adaptive Google recrawling; no fixed per-page cadence is promised. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide

This is the piece of the crawling story that’s easiest to blur into its siblings, so let me draw the lines clearly.

Crawl frequency vs. crawl budget vs. crawl rate

These three get used interchangeably and they shouldn’t:

Google’s own definition ties it together: “Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl.” Frequency is a per-URL cadence that lives inside that budget. Rate is the throttle on the pipe. As I put it in my crawl-budget guide, “Crawl budget is the amount of time and resources a search engine allows for crawling a website. It is made up [of] crawl demand which is how many pages a search engine wants to crawl on your site and crawl rate which is how fast they can crawl.”

Discovery crawls vs. refresh crawls

Frequency is really about refresh crawling. John Mueller laid out the split plainly: “One is a discovery crawl where we try to discover new pages on your website. And the other is a refresh crawl where we update existing pages that we know about.”

Refresh cadence varies enormously by page. Mueller again: “We would refresh crawl the homepage, I don’t know, once a day, or every couple of hours, or something like that.” And on the other end: “If we recognize that individual pages change very rarely, then we realize we don’t have to crawl them all the time.”

The key insight is that Google learns your pattern per page: “If you have a news website and you update it hourly, then we should learn that we need to crawl it hourly. Whereas if it’s a news website that updates once a month, then we should learn that we don’t need to crawl every hour.”

What determines how often Google recrawls a page

In my How Search Works deck I list the crawl-demand factors that drive recrawl: PageRank, how frequently the page changes, time since it was last crawled, and major site changes. That’s the same popularity + staleness model Google documents, just from my own framing — and it’s not an exhaustive list either. Google’s documentation also names perceived inventory (how many URLs it thinks your site has) and site-wide events, such as a domain or URL-structure migration, as things that can move crawl demand up or down. Popularity and staleness are the two you can influence most directly, so they’re the ones worth the most attention. Breaking those down:

Popularity, PageRank, and links

“URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.” More internal and external links to a page → more perceived importance → more frequent recrawls. In my crawl-budget post I say it the same way: “Popular pages, or those with more links and PageRank, will generally receive priority over other pages.”

Staleness — and the backoff

Google’s systems “want to recrawl documents frequently enough to pick up any changes.” The flip side is that pages which don’t change get crawled less and less. From my crawl-budget guide: “If Google sees that a page isn’t changing, they will crawl the page less frequently.” There’s no fixed interval — it’s a backoff: “if they crawl a page and see no changes after a day, they may wait three days before crawling again, ten days the next time, 30 days, 100 days, etc.”

Quality and search demand

Gary Illyes frames the scheduler as something you have to convince: “If you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.” It’s dynamic: “Scheduling is very dynamic. As soon as we get the signals back from search indexing that the quality of the content has increased across this many URLs, we would just start turning up demand.” It cuts both ways — “If search demand goes down, then that also correlates to the crawl limit going down.”

Worth knowing: Google is actively trying to crawl stable pages less for efficiency reasons. Illyes has publicly described wanting to “crawl even less” and reduce bytes on the wire. So don’t expect the scheduler to err toward over-crawling your unchanging pages.

Server health (rate enables frequency)

Frequency can only go as high as your rate allows. The crawl capacity limit is roughly the maximum number of simultaneous parallel connections Google will use; a fast, error-free server raises that ceiling, while a slow site or one returning 5xx/429 gets crawled less. Server health doesn’t increase frequency — it just stops being a bottleneck.

The role of sitemaps and lastmod

Here’s the biggest myth to bust. People think sitemap settings control cadence. They mostly don’t.

changefreq and priority are ignored

Straight from Google’s sitemap docs: “Google ignores <priority> and <changefreq> values.” Setting <changefreq>hourly</changefreq> does nothing. Evidence for this claim Google ignores sitemap priority and changefreq values and may use accurate lastmod values. Scope: Google sitemap processing. Confidence: high · Verified: 2026-07-14 Google: Build and submit a sitemap Stop optimizing those fields.

lastmod is the one signal — if it’s honest

The single freshness signal Google honors from a sitemap is lastmod, and only conditionally: “Google uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.” If you lie about it, Google notices and stops trusting it.

And it has to reflect a real change: “The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not.” Bumping lastmod because you changed the footer year won’t help — and erodes the trust that makes lastmod work at all.

Can you force or speed up crawling?

The honest answer: there’s no dial for frequency. What you actually have:

• Improve the inputs — importance (links, internal linking), genuine content changes, an accurate lastmod, and a fast, healthy server.
• Request a single URL — Google Search Console’s URL Inspection has a “Request indexing” action for one page at a time. It’s a request, not a guarantee, and it doesn’t change ongoing cadence. Google is explicit that mashing the button doesn’t help either: requesting the same URL repeatedly will not get it crawled faster.
• Push a change (on Bing and others) — see below.

What you can’t do: set a frequency, force daily crawls by publishing daily, or use the old GSC crawl-rate slider (it was retired). Bing has a Crawl Control grid, but that controls rate, not frequency.

Bing: adaptive crawling and IndexNow

Bing thinks of crawl frequency as a cost problem. From their crawl-frequency post, the cadence “depends on the frequency of which the content is edited and updated,” and “Defining when to fetch the web page next is the hard problem we are looking to optimize.” Their adaptive answer: “What we learned was that we could optimize our system to avoid fetching the same content over and over, and instead check periodically for major changes” — which in one case yielded “about 40% crawl saving on this site!”

Bing’s “you can nudge it” mechanism is IndexNow: instead of waiting for the scheduler, you signal a change. “Whether you’re adding, updating, or deleting content, IndexNow notifies multiple search engines of your content changes as soon as they happen,” which is about “limiting the need for costly exploratory crawls.” Note the asymmetry: Google does not use IndexNow for general recrawling.

How to check your crawl frequency

• GSC → Crawl Stats — total requests over time, broken down by response code, file type, and Googlebot type. This is how you observe cadence at the site level; it’s not a per-URL last-crawl report.
• GSC → URL Inspection — the “last crawl” date for a single URL. This, not Crawl Stats, is where you check when a specific page was last fetched.
• Bing Webmaster Tools — Crawl Control (rate) and IndexNow Insights.
• Server log analysis — the ground truth: exactly which URLs bots hit and how often.

Common myths about crawl frequency

• “Publishing daily forces faster crawling.” No — Google learns your pattern and crawls more for importance and genuine change, not publishing volume.
• “changefreq/priority control cadence.” No — both are ignored.
• “Just bump lastmod to get recrawled.” Only works if it’s verifiable and tied to a significant change; gaming it makes Google distrust it.
• “More crawling means better rankings.” No. As I’ve written in my crawl budget guide, “The rate of crawling isn’t going to impact your rankings.” Crawling is a prerequisite, not a boost — “More crawling doesn’t mean you’ll rank better, but if your pages aren’t crawled and indexed they aren’t going to rank at all.”
• “There’s a fixed schedule.” No — it’s dynamic, per-URL, and adaptive.
• “I can set crawl frequency in Search Console.” No — the rate slider is gone, and there’s no frequency control.

AI summary

A condensed take on the Advanced version:

• Crawl frequency = cadence of recrawl for a URL Google already knows. Driven mainly by popularity (links / PageRank) and staleness (how often the page really changes) — Google’s documentation also names perceived inventory and site-wide events as demand inputs.
• Google learns your per-page pattern and adapts — hourly for a busy homepage, rarely for a static page, backing off (3 → 10 → 30 → 100 days) on pages that don’t change.
• Distinct from siblings: rate = how fast (server-throttled), budget = demand + capacity (“URLs Google can and wants to crawl”). Frequency lives inside budget.
• Sitemap reality: changefreq and priority are ignored; only a verifiable lastmod tied to a significant change is honored.
• You can’t set frequency. Improve the inputs (importance, genuine change, honest lastmod, server health); request a single URL via GSC URL Inspection; push changes via IndexNow on Bing (not Google).
• More crawling ≠ better rankings. Crawling is required to rank, not a ranking signal.
• Observe it in GSC Crawl Stats (site-level requests over time — not a per-URL log), URL Inspection’s last-crawl date (the per-URL check), Bing Webmaster Tools, or server logs.

Official documentation

Primary-source documentation on recrawl cadence.

Google

• Optimize your crawl budget — crawl demand (perceived inventory, popularity, staleness) and crawl capacity; the model behind recrawl frequency.
• Build and submit a sitemap — why changefreq/priority are ignored and how lastmod is actually used.
• In-Depth Guide to How Google Search Works — the algorithmic crawl scheduler (“which sites to crawl, how often, and how many pages”).

Bing / Microsoft

• bingbot Series: Optimizing Crawl Frequency — Bing’s framing of frequency as a content-change-driven scheduling problem.
• Bing Webmaster Tools — Crawl Control — set Bingbot’s hourly rate (rate, not frequency).
• IndexNow — the push protocol for signaling changed URLs instead of waiting for a recrawl.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what drives recrawl cadence

• “URLs that are more popular on the Internet tend to be crawled more often to keep them fresher in our systems.” — Google Search Central docs. Jump to quote
• “Our systems want to recrawl documents frequently enough to pick up any changes.” Jump to quote
• “Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl.” Jump to quote

Google — sitemaps, changefreq, lastmod

• “Google ignores <priority> and <changefreq> values.” Jump to quote
• “Google uses the <lastmod> value if it’s consistently and verifiably (for example by comparing to the last modification of the page) accurate.” Jump to quote
• “The <lastmod> value should reflect the date and time of the last significant update to the page. For example, an update to the main content, the structured data, or links on the page is generally considered significant, however an update to the copyright date is not.” Jump to quote

John Mueller, Google (SEO office-hours, Jan 2022)

• “One is a discovery crawl where we try to discover new pages on your website. And the other is a refresh crawl where we update existing pages that we know about.” Jump to quote
• “We would refresh crawl the homepage, I don’t know, once a day, or every couple of hours, or something like that.” Jump to quote
• “If you have a news website and you update it hourly, then we should learn that we need to crawl it hourly. Whereas if it’s a news website that updates once a month, then we should learn that we don’t need to crawl every hour.” Jump to quote

Gary Illyes, Google (crawling priorities)

• “If you want to increase how much we crawl, then you somehow have to convince search that your stuff is worth fetching, which is basically what the scheduler is listening to.” Jump to quote
• “Scheduling is very dynamic. As soon as we get the signals back from search indexing that the quality of the content has increased across this many URLs, we would just start turning up demand.” Jump to quote

Microsoft Bing — Optimizing Crawl Frequency

• “The answer depends on the frequency of which the content is edited and updated.” Jump to quote
• “Defining when to fetch the web page next is the hard problem we are looking to optimize with your help.” Jump to quote
• “Whether you’re adding, updating, or deleting content, IndexNow notifies multiple search engines of your content changes as soon as they happen.” — IndexNow. Jump to quote

The mental models

1. Frequency = popularity × staleness. Two inputs drive how often a known URL gets refreshed: how important it is (popularity / PageRank / links) and how often it genuinely changes (staleness). A page that’s both important and changes often gets crawled most; one that’s neither gets crawled least. Everything else is a downstream lever on these two.

2. Discovery crawl vs. refresh crawl. A discovery crawl finds a new URL once. A refresh crawl re-checks a known URL — and frequency is entirely about refresh. If a page isn’t getting recrawled, ask whether it’s an importance problem or a “Google thinks nothing changes here” problem; the fix is different for each.

3. Frequency ≠ rate ≠ budget. Frequency is cadence (how often). Rate is speed (how fast, server-throttled). Budget is demand + capacity (the URL set Google can and wants to crawl). Frequency lives inside budget; rate caps how much frequency is even possible.

4. The scheduler learns your pattern. Google adapts per page to your real update cadence and backs off on stable pages (3 → 10 → 30 → 100 days). You don’t beat the scheduler by publishing more — you change what it learns by actually being important and actually changing.

5. The one honest sitemap signal. changefreq and priority do nothing. A verifiable lastmod tied to a significant change is the only freshness signal that counts — and only as long as you don’t lie about it.

What actually increases recrawl frequency

A pass over the levers that move cadence — and the traps that don’t:

• Make the page more important. Add internal links from strong pages; earn external links. Popularity is the biggest frequency lever.
• Make real changes. Update main content, structured data, or links — not cosmetic edits. Google learns whether a page genuinely changes.
• Keep lastmod accurate and verifiable. It must reflect the last significant update; a copyright-year bump doesn’t count and erodes trust.
• Don’t bother with changefreq / priority — Google ignores both.
• Keep the server fast and error-free. 5xx/429/timeouts lower your crawl capacity, which caps frequency.
• Request a single URL when it matters — GSC URL Inspection → Request indexing for one page after a meaningful update.
• On Bing/Yandex, use IndexNow to push change notifications instead of waiting for the scheduler (Google doesn’t use it).
• Don’t expect publishing volume alone to help — frequency follows importance and genuine change, not how often you hit publish.
• Check Crawl Stats for last-crawl dates and discovery-vs-refresh split before assuming a frequency problem exists.

Tools for seeing crawl frequency

• Google Search Console — Crawl Stats report — requests over time, by response code, file type, and Googlebot type. The cleanest site-level view of how often Google comes back overall — not a per-URL log.
• URL Inspection (GSC) — the “last crawl” date for a single URL, plus the Request indexing action when you’ve made a meaningful change. Repeated requests for the same URL don’t speed it up.
• Bing Webmaster Tools — Crawl Control (set Bingbot’s hourly rate) and IndexNow Insights for the push side.
• Server log file analysis — the ground truth on exactly which URLs bots hit and how often. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform. (See log file analysis.)
• Ahrefs Site Audit / Webmaster Tools — simulate a crawl and surface the importance signals (internal links, depth) that feed frequency.

Crawl-frequency mistakes

• Changing sitemap lastmod without changing the page. False dates make the signal less useful. Update lastmod only for meaningful content changes.
• Relying on changefreq or priority. Google ignores those sitemap hints. Use strong discovery, importance, and honest modification signals instead.
• Requesting indexing repeatedly for an unchanged page. Another fetch does not create value or force indexing. Improve the page or its signals first.
• Treating frequent crawling as a ranking win. Crawl frequency describes fetching, not quality or position. Measure whether important changes are picked up in time.
• Forcing cosmetic edits on a schedule. Search engines learn real change patterns. Make useful updates rather than changing dates or whitespace.

Crawl-frequency signal cheat sheet

Metrics for crawl frequency

Median recrawl interval by template

Metric: median time between verified crawler fetches for the same URL. What it tells you: the learned revisit cadence. How to pull it: sort access-log requests by normalized URL and timestamp, then segment by template. Benchmark / realistic range: compare with each template’s actual change cadence; no sitewide target fits both news and evergreen pages. Cadence: monthly.

Change-to-recrawl lag

Metric: time between a meaningful publish/update event and the next fetch. What it tells you: whether important changes are discovered promptly. How to pull it: join CMS or deployment timestamps with access logs. Benchmark / realistic range: establish a per-template baseline and investigate regressions. Cadence: monthly and after sitemap/internal-link changes.

Honest-lastmod rate

Metric: sitemap lastmod changes that correspond to meaningful page changes. What it tells you: whether the freshness signal remains trustworthy. How to pull it: compare sitemap history with content hashes or release records. Benchmark / realistic range: every changed date should be explainable by a meaningful update. Cadence: each sitemap release or weekly sampling.

Test yourself: Crawl frequency

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — the closest companion piece: demand, rate, staleness backoff, and why frequency isn’t a ranking factor.
• What Is Googlebot & How Does It Work? — the crawler mechanics behind the scheduler.
• Crawl Me Maybe? How Website Crawlers Work — a general crawler primer.
• The Beginner’s Guide to Technical SEO — where crawling fits in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of the crawl-demand factors (PageRank, change frequency, time since last crawl, major site changes). (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From others

• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• Google Has Two Types Of Crawling – Discovery & Refresh (Search Engine Journal) — John Mueller’s verbatim quotes on discovery vs. refresh crawls and how Google learns per-page update patterns.
• Google’s Crawling Priorities: Insights From Analyst Gary Illyes (Search Engine Journal) — Gary Illyes on the dynamic scheduler, quality signals driving crawl demand, and convincing Google your content is worth fetching.
• Google Considers Reducing Webpage Crawl Rate (Search Engine Journal) — coverage of Google’s stated goal to crawl stable pages less, corroborating the backoff pattern.
• bingbot Series: Maximizing Crawl Efficiency (Bing Webmaster Blog) — Bing’s companion piece on crawl efficiency; pairs with the Optimizing Crawl Frequency post in the Official Docs tab.
• r/TechSEO — the community for crawl/index debugging.

Crawl frequency

Crawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial.

Related: Crawl Budget, Crawling

Crawl frequency

Crawl frequency is the cadence of recrawl — how often a search engine refreshes a URL it has already discovered to check whether the page has changed. A news homepage might be refresh-crawled every couple of hours; an “About” page that never changes might be crawled once every few months. The scheduler decides this per URL, dynamically, based largely on a page’s perceived importance (PageRank, links, popularity) and how often it actually changes (staleness).

It’s easy to confuse with two siblings:

• Crawl rate (crawl capacity) is how fast a crawler hits your server — parallel connections and the delay between fetches — throttled by your server’s health. That’s speed, not cadence.
• Crawl budget is the combination of crawl demand and crawl capacity: “the set of URLs that Google can and wants to crawl.” Frequency is a per-URL cadence that lives inside that budget.

You can’t set crawl frequency directly. There’s no dial in Search Console. What you can do is improve the inputs: make a page more important (internal and external links), genuinely change its content, keep an accurate lastmod, and keep your server fast and error-free. Google ignores <changefreq> and <priority> in sitemaps entirely, and only trusts lastmod when it’s verifiably accurate and tied to a significant change. Crawling more often does not improve rankings — crawling is a prerequisite to ranking, not a boost.

Related: Crawl Budget, Crawling

Changelog