Caching for SEO

How browser and server caching with Cache-Control, ETags, and CDNs improves performance and Core Web Vitals, and the caching pitfalls that affect crawling.

First published: Jul 2, 2026 · Last updated: Jul 17, 2026 · Advanced
demand #6 in Web Performance#38 in Technical SEO#52 on the site
AIAuthored byAI v2

Caching stores a copy of a page or resource — in the browser, on a CDN edge, or in a crawler's own cache — so it doesn't have to be regenerated or re-downloaded. It's not a direct ranking factor, but it feeds two things that matter: page speed / Core Web Vitals (via TTFB and LCP) and crawl efficiency. Google's crawler only honors ETag and Last-Modified (plus max-age as a recrawl hint) — it prefers ETag, and 'other HTTP caching directives aren't supported.' The riskiest caching mistakes aren't slow cache durations; they're CDN misconfigurations and stale caches that block or mislead bots.

TL;DR — Caching operates on three layers that matter for SEO: browser, CDN edge, and the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.’s own conditional-request cache. It’s not a ranking factor, but it drives page speed (TTFBTime to First Byte — the time from the start of a request to when the first byte of the response arrives. It's a diagnostic metric (not a Core Web Vital) and a major input to FCP and LCP; ≤0.8 s is good./LCPLargest Contentful Paint — render time of the largest visible image or text block, relative to when the page started loading. ≤2.5 s (at the 75th percentile) is good., and — via bfcacheThe back/forward cache (bfcache) is a browser feature that freezes a whole page in memory when you navigate away, so pressing Back or Forward can restore it instantly — no reload, no re-render, no network requests — as long as the browser hasn't evicted that frozen page first. It's a browser optimization, not a search ranking factor. — repeat-navigation Core Web Vitals) and crawl efficiency. Google’s crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. prefers ETagA conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page. over Last-Modified, reads max-age only as a recrawlCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. hint, and — per its own docs — “other HTTP caching directives aren’t supported.” CDNs earn a higher crawl-rate allowance, but only once their cache is warm; the real risks are cold-cache launches and CDN/WAF misconfigurations that block bots outright.

Evidence for this claim HTTP caching uses Cache-Control and validators to control reuse and revalidation. Scope: Current official or standards documentation. Confidence: high · Verified: MDN: HTTP caching Evidence for this claim Browser caches can reuse stored responses according to HTTP caching semantics. Scope: Current official or standards documentation. Confidence: high · Verified: MDN: HTTP caching

The three layers of caching

Caching for SEOCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency. isn’t one thing — it’s three, each controlled a little differently:

  1. Browser cache — the visitor’s device stores files so repeat views skip the network. This is what PageSpeed InsightsPageSpeed Insights (PSI) is a free Google tool at pagespeed.web.dev that reports two kinds of data for a URL: real-user field data from the Chrome UX Report and a single Lighthouse lab run with the 0–100 Performance score. Only the field Core Web Vitals are what Google uses for ranking. nags about with “Serve static assets with an efficient cache policy.”
  2. CDN / edge cache — a content delivery network stores copies at edge nodes worldwide (see the CDN and SEOA CDN (content delivery network) is a geographically distributed network of edge servers that caches and serves your content from a location close to each visitor and crawler. It isn't a ranking factor itself, but it affects things Google and Bing do use — page speed and Core Web Vitals, crawl efficiency, uptime, and HTTPS delivery — and, if misconfigured, can block crawlers or muddle canonicalization. deep dive for the full treatment). Google’s own description: CDNs are an intermediary between your origin and the user, and historically their biggest focus is caching — storing a URL’s contents so your server doesn’t have to serve that file again for a while.
  3. Crawler-side cache — GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. and BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. keep their own record of whether content changed, using conditional requestsA conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page.. This is the crawl-budget lever, and the mechanics belong to the conditional requests deep dive; here I’ll keep it at a summary level.

“Browser cache” above is shorthand for more than one mechanism. Per MDN’s HTTP caching guide, worth knowing apart: the private HTTP cache (per-browser, keyed by request, and in modern browsers partitioned by top-level site to limit cross-site tracking), the in-memory cache used for the current session, the bfcache covered below, and — separately — a service worker’s Cache Storage, which a site’s own JavaScript controls and which Cache-Control headers don’t govern directly. “Check the browser cache” can mean four different debugging steps depending on which one is actually misbehaving.

Why caching feeds Core Web Vitals

Fetching resources over the network is slow and expensive. Caching removes the network latency and the transfer cost for anything that hasn’t changed. That flows straight into two vitals-adjacent measures: TTFBTime to First Byte — the time from the start of a request to when the first byte of the response arrives. It's a diagnostic metric (not a Core Web Vital) and a major input to FCP and LCP; ≤0.8 s is good. (a cached response skips regeneration at the origin) and LCPLargest Contentful Paint — render time of the largest visible image or text block, relative to when the page started loading. ≤2.5 s (at the 75th percentile) is good. (cached images/CSS/fonts render sooner).

Cache-Control, in the directives that matter

Cache-Control is the main header. The ones worth knowing:

  • max-age=<seconds> — how long a fresh copy is good for. For immutable, versioned assets, Chrome’s LighthouseLighthouse is Google's free, open-source tool that audits a page under simulated lab conditions and scores it 0–100 across Performance, Accessibility, Best Practices, and SEO. It's lab data — useful for debugging, not a ranking signal. docs recommend caching for a year or longer — e.g. Cache-Control: max-age=31536000.
  • no-cache — not “don’t cache.” It means “store it, but revalidate with the server before reusing it.” It still enables the lightweight 304 flow.
  • no-store — the one that actually means don’t store any copy anywhere in an HTTP cache. It’s a caching directive, not a general privacy switch — per RFC 9111 it isn’t a reliable way to erase browser history, and it says nothing about a service worker’s own Cache Storage.
  • public / private — whether shared caches (like a CDN) may store the response, or only the end user’s browser.
  • immutable — skip revalidation entirely while the response is still fresh. It doesn’t mean “never goes stale” — once max-age runs out, normal freshness rules apply again.
  • must-revalidate — the opposite end of the timeline: it only matters after a response goes stale, and tells the cache it must revalidate with the origin rather than serving the stale copy anyway.
  • s-maxage, stale-while-revalidate, stale-if-error — finer controls mostly for CDNs and other shared caches (a separate freshness lifetime for shared caches, bounded stale reuse while a background fetch runs, and bounded stale reuse on an origin error, respectively). Support varies by browser/CDN, so check current support before relying on them — and none of this extra directive set is honored by Google’s crawler, as we’ll see.

Cache-busting with versioned filenames

The trick that lets you cache aggressively and update instantly: put a content hash in the filename — style.x234dff.css. Because the URL is the cache key, changing the file changes the URL, so caches fetch the new version immediately while old versions stay cached for as long as you like. Both Google’s web.dev HTTP cache guide and Bing’s own front-end engineering write-up describe the same pattern — Bing hashes file contents into the URL so “the URL acts as the cache key,” which keeps caches consistent and allows long expiration times.

The bfcache trap — where no-store quietly hurts CWV

Here’s an under-covered one. The back/forward cache (bfcache) is what makes hitting “back” restore a page instantly. A bfcache restore skips LCP/CLSCumulative Layout Shift — a unitless score for unexpected visual movement, taken from the largest burst (session window) of layout shifts, not the lifetime sum. ≤0.1 is good./INPInteraction to Next Paint — the input-to-paint latency at the 75th percentile of a page's interactions. ≤200 ms is good. measurement entirely, so it’s pure upside for your CrUXChrome User Experience Report — Google's public dataset of real-world (field) performance data from eligible Chrome users. It's the official field-data source behind the Core Web Vitals program. field dataPerformance metrics captured from real users, not lab tests.. But per Google’s bfcache guide, setting Cache-Control: no-store on the page document itself has historically made browsers refuse to store that page in bfcache. If you need freshness on an HTML document but don’t want to sacrifice back/forward-cache eligibility, reach for no-cache or max-age=0 instead of no-store.

How a cache decides “fresh enough”

Before any validator gets involved, a cache checks freshness: has the stored response’s age passed the freshness lifetime Cache-Control gave it (or, absent an explicit lifetime, a heuristic one the cache is allowed to guess at)? The Age response header reports how long a shared cache has already held a response, which is how you can tell — in DevTools or from a CDN log — how much freshness lifetime is left. Fresh means the cache can reuse it immediately, no request at all. Stale means it should validate before reusing it, which is exactly where ETag/If-None-Match and Last-Modified/If-Modified-Since earn their keep — described next, for Googlebot’s narrower version of this general HTTP mechanism.

How Googlebot uses caching (the crawl-efficiency angle)

Google made an unusually direct ask in its December 2024 Crawling December: HTTP caching post: enable caching so its crawlers can skip re-downloading unchanged pages. The striking data point in that post is that cacheable fetches have been declining — about 0.026% of total fetches were cacheable 10 years ago, and today that number is 0.017%. Small numbers, but Google clearly wants them going the other way.

ETag vs. Last-Modified — which Google prefers

Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. infrastructure supports the two standard validators: ETag (with If-None-Match) and Last-Modified (with If-Modified-Since). Google strongly recommends ETag because its value is unstructured and therefore less prone to the parsing mistakes a date string invites — and if both are present, its crawlers use the ETag value as the HTTP standard requires. Google still suggests setting both anyway, since other applications such as CMSes use them. If you do use Last-Modified, the date has to follow the HTTP format (for example, Fri, 4 Sep 1998 19:15:56 GMT) or it won’t parse.

TIP Verify the validators on the response crawlers actually receive

An ETag is optional, but it gives Googlebot and other clients a strong validator for conditional requests. Inspect the complete response before assuming the application or CDN preserved it.

Check the live header map with my free HTTP Header Checker Free

  1. Inspect the final response and every redirect hop for Cache-Control, ETag, and Last-Modified.
  2. Compare production with staging or another URL when a deployment may have changed the headers.
  3. Send a conditional request separately to confirm a matching validator produces the expected 304 response.
No ETag is not automatically an error, but it is a useful prompt to verify the site’s conditional-request strategy.

The HTTP Header Checker reports: No ETag. ETag is optional but can support efficient conditional crawling.

When the crawler’s stored validator still matches, your server returns a 304 Not Modified with no body — which is the whole point. As Google puts it, no body means your server doesn’t spend compute generating content and doesn’t spend bandwidth transferring it. (That 304 machinery is the crawl-budget mechanism covered in depth in the conditional requests article; here it’s enough to know it exists and saves you money on both sides.)

The nuance almost everyone misses

Google’s crawler does not act on the full Cache-Control directive set the way a browser or CDN does. Per the official crawler overview, beyond ETag/Last-Modified, “other HTTP caching directives aren’t supported.” The one partial exception: Google says you can optionally set max-age to help crawlers determine when to recrawl a URL — a recrawl hint, not a hard lock. So no-cache, s-maxage, stale-while-revalidate, and friends still shape browser and CDN behavior, but they don’t change how Googlebot caches. And Google’s advice on when to invalidate is sensible: require a cache refresh on significant changes — updating just the copyright date in the footer isn’t significant.

CDNs and crawling

A CDN buys you more than speed. Google’s crawling infrastructure is designed to allow higher crawl rates on sites backed by a CDN, inferred from the IP address serving the URLs — it assumes a CDN-backed origin can handle more simultaneous requests.

But there’s a catch worth planning around: the cold cache. On the first access of a URL, the CDN’s cache is “cold” — nobody has requested it yet, so your origin still has to serve it at least once to warm the cache. Google warns that launching many URLs at once is therefore a real burden on crawl budgetThe number of URLs an engine will crawl in a timeframe., with a high crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. for a few days. If you’re doing a big launch or a site migrationA site migration is any significant change to a website's URL structure, domain, platform, protocol, or hosting that can affect how search engines crawl, index, and rank it. The risk scales with how much you change at once., budget for the origin taking full load per URL before the CDN starts helping.

TIP Find the requests that really missed cache during a page load

Headers on one URL explain its policy; a HAR shows which requests in an actual browser journey recorded cache misses, revalidation, transfer weight, and third-party cost. Review that evidence locally with my free HAR Analyzer Free

  1. Capture a repeatable page load in the browser Network panel and export the HAR with content.
  2. Import it locally, then review explicit cache misses alongside response status, transfer bytes, duration, and host ownership.
  3. Fix the relevant browser, CDN, or origin policy and compare a fresh capture under the same cache state.

CDN misconfiguration is a crawling risk

The scariest caching-adjacent problems aren’t slow cache durations — they’re CDN and WAF setups that block bots. Google’s CDN post is explicit that for temporary blockages, sending 503/429 is the preferred way to signal it, whereas network timeouts are treated as terminal, “hard” errors that can get URLs removed from the index. The subtle one is a soft block: a bot-verification interstitial. The crawler only sees the challenge page, not your site — so Google strongly recommends returning a 503 to automated clients instead. The easiest way to check that a CDN isn’t quietly blocking Google is the URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. — look at the rendered image; if it shows a bot challenge or an empty page, talk to your CDN.

Offloading redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. to the CDN is a technique I’m a fan of. On the Marketing Speak podcast I described it as “One of my personal favorites that I don’t think it’s used enough, it’s actually just off loading your redirects to the CDN level.” (Jump to quote)

Caching pitfalls that hurt crawling and indexing

This is the angle most “caching for SEO” articles skip. A cache doesn’t just make things fast — a wrong cache can serve the wrong bytes to a bot and break crawling or indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..

A real one: a shared cache serving a blocking robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.. I dug into a case of intermittent Googlebot blocking that traced back to a shared CDN cache between a test environment and the live site. As I wrote in Indexed, though blocked by robots.txt: “One possible cause would be a shared cache between a test environment and a live environment. When the cache from the test environment is active, the robots.txt file may include a blocking directive.” The fix was to split the cache — or exclude .txt files from the cache in the test environment. A caching misconfiguration directly caused a crawling failure; that’s the category of risk that actually bites.

Other pitfalls in the same family:

  • Stale CDN cache serving outdated content to bots. If your edge cache holds an old version long after you’ve published a change, bots keep seeing the old one. Purge on publish, or tie cache lifetime to how often the page really changes.
  • Vary / User-Agent cache fragmentation. A shared cache’s key is normally just the URL; Vary adds request headers (like User-Agent or Accept-Language) to that key so different variants get stored separately. Miss a header that actually changes the response and one requester can get another’s variant — the mobile/desktop or bot/human mixup. Add too many headers to Vary and you fragment the cache into so many near-duplicate keys it barely improves hit rate. Separately, modern browsers also partition their own caches by top-level site for privacy, so a resource cached while embedded on one site generally isn’t reused when embedded on another — a different mechanism from Vary, worth not conflating with it when you’re debugging a “why isn’t this cached” report.

My general rule of thumb on duration comes from the LCP work: as I put it in my Ahrefs guide to Largest Contentful Paint, “Your cache time should be as long as you are comfortable with” — and “An ideal setup is to cache for a really long period of time but purge the cache when you make a change to a page.” Long cache, instant purge. That combination is what keeps you both fast and fresh.

Is caching a ranking factor?

No — not directly. There’s no ranking signal for having ETags set or for a good Cache-Control policy. What caching does is feed two things that matter to visibility: page speed / Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. (an explicit page-experience input) and crawl efficiency (which governs how fast new and updated content gets discovered and refreshed, indirectly touching freshness-sensitive results). Set it up because it makes your site fast and easy to crawl — not because you expect a direct rankings bump.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.