Conditional Requests: ETag, If-Modified-Since & 304 Not Modified

How conditional requests — ETag, Last-Modified, If-Modified-Since/If-None-Match, and the 304 Not Modified response — let Googlebot skip re-downloading unchanged pages and preserve crawl budget on large sites.

First published: Jun 28, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #19 in Crawling#53 in How Search Works#285 in Technical SEO#381 on the site
AIAuthored byAI v2
1 evidence signal on this page

Conditional requests are how Googlebot asks 'has this page changed since I last crawled it?' before re-downloading. It sends If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag); if nothing changed, your server should return 304 Not Modified with no body and the crawler reuses its existing copy — roughly a kilobyte instead of the whole page. Google prefers ETag when both are present, doesn't send the headers on every crawl, and a 304 does not freeze indexing signals. The payoff is crawl efficiency on large sites with many rarely-changing URLs — it's not a ranking factor. Sites quietly break it three ways: always returning 200 with a full body, ETags that change on every request (timestamps, per-request tokens, per-node CDN variance), and a Last-Modified that doesn't reflect real content changes.

TL;DR — Conditional requestsA conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page. let GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. validate a cached copy instead of re-downloading it. It sends If-Modified-Since (validated against your Last-Modified header) and/or If-None-Match (validated against your ETag); if nothing changed, your server returns 304 Not Modified with no body and the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. reuses its copy — roughly ~1 KB vs. 100 KB+ for a full page. ETag wins when both are present; Google recommends ETag (no date-format pitfalls) but says set both. Google doesn’t send the headers on every crawl (use-case dependent — AdsBot is likelier to), you can proactively serve a 304 even without a conditional header, and a 304 does not freeze indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. signals. Three misconfigurations defeat it: always-200, volatile ETags (timestamps, per-request tokensA token is the smallest unit of text (or image/audio/video) an LLM processes — roughly 4 characters, or about ¾ of an English word. A context window is the maximum number of tokens (input plus output) a model can hold at once, like its short-term memory., per-node CDN variance), and a Last-Modified that doesn’t track real changes. It’s a crawl-efficiency lever for large sites — not a ranking factor, and distinct from crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor., frequency, and budget.

Evidence for this claim Conditional requests use validators such as ETag and Last-Modified with If-None-Match or If-Modified-Since. Scope: Current official or standards documentation. Confidence: high · Verified: RFC 9110: Conditional requests Evidence for this claim A 304 response indicates a conditional request can reuse a stored representation and does not include a message body. Scope: Current official or standards documentation. Confidence: high · Verified: RFC 9110: 304 Not Modified

The mechanism, precisely

Where crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is how fast GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. fetches and crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. is how often it comes back, conditional requests are about how cheaply each individual recrawl can be answered. They’re a validation handshake between the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. and your server, built on two pairs of HTTP headers:

You send (response header)Crawler sends back (request header)Validates by
Last-Modified: <date>If-Modified-Since: <date>Comparing dates
ETag: "<fingerprint>"If-None-Match: "<fingerprint>"Comparing content IDs

The flow, step by step:

  1. On the first crawl, your server returns the page with 200 OK plus a Last-Modified date and/or an ETag fingerprint.
  2. On a later crawl, Googlebot may send If-Modified-Since (echoing back the date it last saw) and/or If-None-Match (echoing back the ETag).
  3. Your server checks them. If nothing relevant changed, it returns 304 Not Modified with no response body — just status and headers.
  4. Googlebot reuses the version it crawled last time. If the content did change, your server returns 200 OK with the full body.

Google documents the handshake almost word for word: “Google’s crawlers that support cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency. will send the ETag value returned for a previous crawl of that URL in the If-None-Match header. If the ETag value sent by the crawler matches the current value the server generated, your server should return an HTTP 304HTTP 304 Not Modified is the response to a conditional GET/HEAD whose condition evaluates false, in the 3xx class but not a redirect: it has no Location header and no body. It tells a client its cached copy is still valid. For SEO it has no direct ranking effect and no indexing effect beyond a possible signal recalculation, though it can help crawl efficiency indirectly on large sites. (Not modified) status code with no HTTP body.”

The validator makes the recrawl conditional: unchanged content returns a lightweight 304, while a real change returns the full 200 response. Source: Google Search Central

A crawler revalidates its saved copy by sending If-None-Match with an ETag or If-Modified-Since with a date. The server compares that validator. If the representation is unchanged, it returns 304 Not Modified without a response body and the crawler reuses its saved copy. If the representation changed, the server returns 200 OK with the full updated response body.

© Patrick Stox LLC · CC BY 4.0 ·

ETag vs. Last-Modified — and why Google prefers ETag

Both validators are supported. Google’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. docs state it plainly: “Google’s crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header.”

When both are present, ETag wins: “If both ETag and Last-Modified response header fields are present in the HTTP response, Google’s crawlers use the ETag value as required by the HTTP standard.” That’s not just a Google quirk — it’s the HTTP spec, and it matches the precedence rule I’ve described in my Ahrefs glossary entry on 304 Not Modified: when both If-None-Match and If-Modified-Since are used, If-None-Match takes precedence.

Why prefer ETag? Because it’s an opaque string — a fingerprint of the content — so it sidesteps the date-parsing traps that plague Last-Modified. Google’s own recommendation: “For Google’s crawlers specifically, we recommend using ETag instead of the Last-Modified header to indicate caching preference as ETag doesn’t have date formatting issues.” If you do use Last-Modified, the date has to be formatted to the HTTP standard (Weekday, DD Mon YYYY HH:MM:SS Timezone) or Google may not parse it. And Google’s practical advice is to set both anyway if you can — belt and suspenders.

There’s a bonus signal too: Cache-Control: max-age. Google says it’s not required, but you can set it to the number of seconds you expect the content to stay unchanged to help crawlers decide when to recrawl. Note the asymmetry — Google’s crawling infrastructure honors max-age as a hint but doesn’t treat other Cache-Control directives the way a browser would.

What a 304 does — and what it doesn’t

The payoff is dramatic on the wire. A 304 carries no body, so per Gary Illyes’ rough comparison it’s on the order of ~1 KB instead of 100 KB+ for a full page — and it’s cheaper to generate (no full render or database query) and cheaper for Google to process (no re-parsing, re-renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., or re-running the whole indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. pipeline on the body).

But be precise about what a 304 does not do: it does not freeze your ranking signals. Google’s own status-code guidance notes that on a 304 the indexing pipeline “may recalculate signals for the URL,” even though it didn’t re-fetch the body. A 304 skips the download and re-processing of content — not every downstream evaluation. So there’s no “serve 304s to lock in my rankings” trick here.

Google doesn’t always ask — and that’s normal

Here’s a detail most write-ups miss: Googlebot doesn’t send the conditional headers on every request. From Google’s crawling-errors doc: “Google generally supports the If-Modified-Since and If-None-Match HTTP request headers for crawling. Google’s crawlers don’t send the headers with all crawl attempts; it depends on the use case of the request (for example, AdsBot is more likely to set the If-Modified-Since and If-None-Match HTTP request headers).”

So if you look in your logs and see Googlebot rarely sending If-None-Match, that does not mean your setup is broken — Google may simply not be asking on that crawl. Google also notes that individual crawlers and fetchers may or may not use caching depending on the product they serve.

And the flip side — a genuinely underused implementation detail — you can serve a 304 without receiving a conditional header at all. Google: “Independently of the request headers, you can send a 304 (Not Modified) HTTP status codeAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index. and no response body for any Googlebot request if the content hasn’t changed since Googlebot last visited the URL. This will save your server processing time and resources, which may indirectly improve crawl efficiency.” That’s an advanced CDN/edge technique — and a dangerous one if you get it wrong, because serving 304 for content that did change is exactly the “your validators lie” failure mode. Treat it as an edge optimization, not a shortcut to fake freshness.

Why this matters for crawl budget

The savings compound hardest on the exact profile Google’s crawl-budget guidance already flags: large sites with a big share of rarely-changing URLs — e-commerce catalogs with long-tail SKUs, news and publisher archives, big documentation sites. On those sites, a meaningful slice of crawl budgetThe number of URLs an engine will crawl in a timeframe. can be spent re-downloading pages that are byte-for-byte identical to the last crawl. Answer those with 304s and you free that budget for new and changed pages instead.

The honest, slightly deflating part: Google is asking for more adoption here, not reporting that everyone already does it. Illyes has noted that the share of Googlebot’s fetches that are cacheable has actually fallen over the last decade — from around 0.026% to about 0.017% — even as the web has grown. In other words, this is an underused lever, and the “Crawling December” post’s own section heading is a near-literal plea: “Allow us to cache, pretty please.” Most sites simply aren’t configured for it.

Common misconfigurations that defeat conditional requests

Setting a validator is not the same as benefiting from one. Three (well, four) ways sites quietly break this:

1. Always returning 200 with a full body

The most common failure is doing nothing — the server never sets validators or never checks them, so every crawl is a full re-download regardless of whether anything changed. Usually this is a CMSA content management system (CMS) is software that lets users create, manage, and publish digital content — like blog posts and pages — without writing raw code. WordPress, Drupal, and Joomla are the most common open-source CMS platforms./server stack with no caching layer configured at all. This is the default state Illyes is nudging people out of.

2. ETags that change on every single request

This one is nastier because it looks correct. If your ETag is computed from something volatile — an embedded timestamp, a per-request or per-session token, a request ID, or a build artifact that bakes in a build time — then every response gets a “new” ETag even when the visible content is identical. If-None-Match never matches, so your server never has grounds to return 304. Fix: derive the ETag from a hash of the actual response body or a stable content-version identifier, not from anything that varies per request.

3. Per-node CDN/cluster variance

A close cousin: different origin nodes generating different (weak) ETags for identical content. A CDN or crawler alternating between nodes never sees a stable value and keeps re-validating without ever landing a clean 304. Fix: generate ETags deterministically from content, not per-instance state, or centralize ETag generation across the fleet.

4. A Last-Modified that doesn’t reflect real changes

If your Last-Modified gets stamped with “now” on every render, or bumped by a trivial change like an auto-updating copyright year in the footer, it either triggers pointless full re-crawls (the date always looks new) or, if it’s stale/gamed, erodes Google’s trust in the signal. This is the same honesty principle Google applies to the sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. lastmod value — only use it if it verifiably reflects a significant change. The crawl-frequency article covers that lastmod logic; the identical discipline applies to the Last-Modified HTTP header.

The Illyes edge case: when a 304 locks in a broken page

Worth its own callout, because it’s genuinely counterintuitive and almost nobody covers it. Illyes has described how a 304 can “backfire spectacularly”: a server bug serves a broken, empty page with a 200; the crawler treats it as a transient error and schedules a recrawl to verify; the still-broken page then correctly reports 304 (“unchanged”); and the crawler concludes the error state is the durable, “real” content and stops rechecking as often. His numbered walkthrough ends with the crawler “learn[ing] the error is persistable.” His own caveat: does this happen? Yes. Often? Absolutely not — “but it’s worth keeping this somewhere deep in your mind because debugging it is an absolute nightmare.” The lesson: conditional requests can entrench an error if the underlying content generation is broken, so don’t bolt them on over a flaky origin.

How Bing handles it

This isn’t a Google-only feature, and Bing’s support is arguably older and more explicit. Bing (back when it was Live Search) announced RFC-2616-compliant conditional GET in 2008: it “generally will not download the page unless it has changed since the last time it crawled it,” sending If-Modified-Since with the time of last download and, when available, If-None-Match with the ETag. Today Bing folds this into a named, trackable metric — crawl efficiency — which Fabrice Canel defines as “how often we crawl and discover new and fresh content per page crawled.” Unnecessary re-crawls of unchanged content directly lower that score. So the same validators serve both engines; Bing just gives the concept a scoreboard.

Does this affect rankings?

No. Keep the same discipline the crawl-rate and crawl-budget articles hold to: conditional requests are a crawl-efficiency and resource lever, not a ranking factor. No official source ties ETag/If-Modified-Since/304 to rankings. The indirect benefit is that large or frequently-updated sites can get new and changed content crawled and indexed sooner — but crawling more efficiently is not itself a ranking signal.

How to verify it’s working

The ground truth is your server logs. Look for two things: Googlebot sending If-None-Match/If-Modified-Since request headers, and your server returning 304 responses to it. Real-world data is thin here, which is what makes Dave Smart’s server-log study at Tame the Bots valuable: monitoring verified Googlebot traffic, he found only about 1.3% of requests got a 304 — mostly 200s — and that the If-None-Match requests tended to cluster when a URL was requested again shortly after a prior fetch. His conclusion matches the guidance: infrequent on small sites, but potentially “significant savings” on heavily-crawled ones. Don’t expect a high 304 rate on a small site; do expect it to matter at scale.

Conditional requests vs. rate vs. frequency vs. budget

To keep the crawl-budget family straight:

  • Crawl rate — how fast Googlebot fetches (supply side, throttled by server health).
  • Crawl frequencyCrawl frequency is how often a search engine comes back to re-fetch a page it already knows about. Popular pages that change often get refreshed many times a day; stable pages can go weeks or months between crawls — and you influence it indirectly, not by setting a dial. — how often a known URL is re-fetched (popularity + staleness).
  • Crawl budget — the demand + capacity envelope: the set of URLs Google can and wants to crawl.
  • Conditional requests — how cheaply each individual recrawl can be answered. They don’t change how fast or how often Google crawls; they make each unchanged-page visit almost free, which is how you stop wasting budget on identical content.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.