Conditional Requests: ETag, If-Modified-Since & 304 Not Modified

TL;DR — A conditional request is Googlebot asking your server “has this page changed since I last grabbed it?” before downloading it again. If the answer is no, your server sends back a tiny 304 Not Modified reply with no page in it, and the crawler reuses the copy it already has. That saves work on both sides — but it only helps in a meaningful way on big sites, and it doesn’t make you rank higher.

What a conditional request is

Normally, when Googlebot wants a page, it just downloads the whole thing. A conditional request is smarter: the crawler says “only send me the page if it’s actually different from the last time I visited.”

It does this with a little extra information it remembers from the previous crawl:

• A date — “last time I got this page, you told me it was last modified on such-and-such date. Anything newer?”
• A fingerprint — “last time, you gave this page an ID (an ETag). Is that ID still the same?”

If nothing changed, your server replies 304 Not Modified. That reply has no page attached — it’s just a short message that means “same as before.” Googlebot then reuses the copy it saved last time instead of re-downloading everything.

If something did change, your server sends the full page back as normal (a 200 OK), and the crawler takes the fresh copy.

Why anyone cares

Think of a giant online store with a million product pages that mostly never change. Without conditional requests, Googlebot re-downloads all of them every time it comes back — a huge amount of wasted bandwidth and server work for pages that are identical to last week. With conditional requests, most of those visits get a tiny “nothing changed” reply, and the crawler can spend its time on the pages that did change or are brand new.

That’s the whole benefit: efficiency. It’s part of what’s called your crawl budget — the amount of crawling a search engine is willing to do on your site.

The honest caveat

Two things worth knowing before you get excited:

1. This mostly matters for large sites. On a small site, the savings are tiny and usually not worth the setup. As I’ve written before, for small websites the caching opportunity a 304 provides is “not that crucial.”
2. It won’t improve your rankings. Crawling more efficiently doesn’t move you up the results — it just helps big sites get their new and changed pages crawled a bit faster.

Want the exact headers, what Google has actually said it supports, the three ways sites quietly break this, and how it differs from crawl rate and crawl frequency? Switch to the Advanced tab.

TL;DR — Conditional requests let Googlebot validate a cached copy instead of re-downloading it. It sends If-Modified-Since (validated against your Last-Modified header) and/or If-None-Match (validated against your ETag); if nothing changed, your server returns 304 Not Modified with no body and the crawler reuses its copy — roughly ~1 KB vs. 100 KB+ for a full page. ETag wins when both are present; Google recommends ETag (no date-format pitfalls) but says set both. Google doesn’t send the headers on every crawl (use-case dependent — AdsBot is likelier to), you can proactively serve a 304 even without a conditional header, and a 304 does not freeze indexing signals. Three misconfigurations defeat it: always-200, volatile ETags (timestamps, per-request tokens, per-node CDN variance), and a Last-Modified that doesn’t track real changes. It’s a crawl-efficiency lever for large sites — not a ranking factor, and distinct from crawl rate, frequency, and budget.

The mechanism, precisely

Where crawl rate is how fast Googlebot fetches and crawl frequency is how often it comes back, conditional requests are about how cheaply each individual recrawl can be answered. They’re a validation handshake between the crawler and your server, built on two pairs of HTTP headers:

The flow, step by step:

1. On the first crawl, your server returns the page with 200 OK plus a Last-Modified date and/or an ETag fingerprint.
2. On a later crawl, Googlebot may send If-Modified-Since (echoing back the date it last saw) and/or If-None-Match (echoing back the ETag).
3. Your server checks them. If nothing relevant changed, it returns 304 Not Modified with no response body — just status and headers.
4. Googlebot reuses the version it crawled last time. If the content did change, your server returns 200 OK with the full body.

Google documents the handshake almost word for word: “Google’s crawlers that support caching will send the ETag value returned for a previous crawl of that URL in the If-None-Match header. If the ETag value sent by the crawler matches the current value the server generated, your server should return an HTTP 304 (Not modified) status code with no HTTP body.”

ETag vs. Last-Modified — and why Google prefers ETag

Both validators are supported. Google’s crawling docs state it plainly: “Google’s crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header.”

When both are present, ETag wins: “If both ETag and Last-Modified response header fields are present in the HTTP response, Google’s crawlers use the ETag value as required by the HTTP standard.” That’s not just a Google quirk — it’s the HTTP spec, and it matches the precedence rule I’ve described in my Ahrefs glossary entry on 304 Not Modified: when both If-None-Match and If-Modified-Since are used, If-None-Match takes precedence.

Why prefer ETag? Because it’s an opaque string — a fingerprint of the content — so it sidesteps the date-parsing traps that plague Last-Modified. Google’s own recommendation: “For Google’s crawlers specifically, we recommend using ETag instead of the Last-Modified header to indicate caching preference as ETag doesn’t have date formatting issues.” If you do use Last-Modified, the date has to be formatted to the HTTP standard (Weekday, DD Mon YYYY HH:MM:SS Timezone) or Google may not parse it. And Google’s practical advice is to set both anyway if you can — belt and suspenders.

There’s a bonus signal too: Cache-Control: max-age. Google says it’s not required, but you can set it to the number of seconds you expect the content to stay unchanged to help crawlers decide when to recrawl. Note the asymmetry — Google’s crawling infrastructure honors max-age as a hint but doesn’t treat other Cache-Control directives the way a browser would.

What a 304 does — and what it doesn’t

The payoff is dramatic on the wire. A 304 carries no body, so per Gary Illyes’ rough comparison it’s on the order of ~1 KB instead of 100 KB+ for a full page — and it’s cheaper to generate (no full render or database query) and cheaper for Google to process (no re-parsing, re-rendering, or re-running the whole indexing pipeline on the body).

But be precise about what a 304 does not do: it does not freeze your ranking signals. Google’s own status-code guidance notes that on a 304 the indexing pipeline “may recalculate signals for the URL,” even though it didn’t re-fetch the body. A 304 skips the download and re-processing of content — not every downstream evaluation. So there’s no “serve 304s to lock in my rankings” trick here.

Google doesn’t always ask — and that’s normal

Here’s a detail most write-ups miss: Googlebot doesn’t send the conditional headers on every request. From Google’s crawling-errors doc: “Google generally supports the If-Modified-Since and If-None-Match HTTP request headers for crawling. Google’s crawlers don’t send the headers with all crawl attempts; it depends on the use case of the request (for example, AdsBot is more likely to set the If-Modified-Since and If-None-Match HTTP request headers).”

So if you look in your logs and see Googlebot rarely sending If-None-Match, that does not mean your setup is broken — Google may simply not be asking on that crawl. Google also notes that individual crawlers and fetchers may or may not use caching depending on the product they serve.

And the flip side — a genuinely underused implementation detail — you can serve a 304 without receiving a conditional header at all. Google: “Independently of the request headers, you can send a 304 (Not Modified) HTTP status code and no response body for any Googlebot request if the content hasn’t changed since Googlebot last visited the URL. This will save your server processing time and resources, which may indirectly improve crawl efficiency.” That’s an advanced CDN/edge technique — and a dangerous one if you get it wrong, because serving 304 for content that did change is exactly the “your validators lie” failure mode. Treat it as an edge optimization, not a shortcut to fake freshness.

Why this matters for crawl budget

The savings compound hardest on the exact profile Google’s crawl-budget guidance already flags: large sites with a big share of rarely-changing URLs — e-commerce catalogs with long-tail SKUs, news and publisher archives, big documentation sites. On those sites, a meaningful slice of crawl budget can be spent re-downloading pages that are byte-for-byte identical to the last crawl. Answer those with 304s and you free that budget for new and changed pages instead.

The honest, slightly deflating part: Google is asking for more adoption here, not reporting that everyone already does it. Illyes has noted that the share of Googlebot’s fetches that are cacheable has actually fallen over the last decade — from around 0.026% to about 0.017% — even as the web has grown. In other words, this is an underused lever, and the “Crawling December” post’s own section heading is a near-literal plea: “Allow us to cache, pretty please.” Most sites simply aren’t configured for it.

Common misconfigurations that defeat conditional requests

Setting a validator is not the same as benefiting from one. Three (well, four) ways sites quietly break this:

1. Always returning 200 with a full body

The most common failure is doing nothing — the server never sets validators or never checks them, so every crawl is a full re-download regardless of whether anything changed. Usually this is a CMS/server stack with no caching layer configured at all. This is the default state Illyes is nudging people out of.

2. ETags that change on every single request

This one is nastier because it looks correct. If your ETag is computed from something volatile — an embedded timestamp, a per-request or per-session token, a request ID, or a build artifact that bakes in a build time — then every response gets a “new” ETag even when the visible content is identical. If-None-Match never matches, so your server never has grounds to return 304. Fix: derive the ETag from a hash of the actual response body or a stable content-version identifier, not from anything that varies per request.

3. Per-node CDN/cluster variance

A close cousin: different origin nodes generating different (weak) ETags for identical content. A CDN or crawler alternating between nodes never sees a stable value and keeps re-validating without ever landing a clean 304. Fix: generate ETags deterministically from content, not per-instance state, or centralize ETag generation across the fleet.

4. A Last-Modified that doesn’t reflect real changes

If your Last-Modified gets stamped with “now” on every render, or bumped by a trivial change like an auto-updating copyright year in the footer, it either triggers pointless full re-crawls (the date always looks new) or, if it’s stale/gamed, erodes Google’s trust in the signal. This is the same honesty principle Google applies to the sitemap lastmod value — only use it if it verifiably reflects a significant change. The crawl-frequency article covers that lastmod logic; the identical discipline applies to the Last-Modified HTTP header.

The Illyes edge case: when a 304 locks in a broken page

Worth its own callout, because it’s genuinely counterintuitive and almost nobody covers it. Illyes has described how a 304 can “backfire spectacularly”: a server bug serves a broken, empty page with a 200; the crawler treats it as a transient error and schedules a recrawl to verify; the still-broken page then correctly reports 304 (“unchanged”); and the crawler concludes the error state is the durable, “real” content and stops rechecking as often. His numbered walkthrough ends with the crawler “learn[ing] the error is persistable.” His own caveat: does this happen? Yes. Often? Absolutely not — “but it’s worth keeping this somewhere deep in your mind because debugging it is an absolute nightmare.” The lesson: conditional requests can entrench an error if the underlying content generation is broken, so don’t bolt them on over a flaky origin.

How Bing handles it

This isn’t a Google-only feature, and Bing’s support is arguably older and more explicit. Bing (back when it was Live Search) announced RFC-2616-compliant conditional GET in 2008: it “generally will not download the page unless it has changed since the last time it crawled it,” sending If-Modified-Since with the time of last download and, when available, If-None-Match with the ETag. Today Bing folds this into a named, trackable metric — crawl efficiency — which Fabrice Canel defines as “how often we crawl and discover new and fresh content per page crawled.” Unnecessary re-crawls of unchanged content directly lower that score. So the same validators serve both engines; Bing just gives the concept a scoreboard.

Does this affect rankings?

No. Keep the same discipline the crawl-rate and crawl-budget articles hold to: conditional requests are a crawl-efficiency and resource lever, not a ranking factor. No official source ties ETag/If-Modified-Since/304 to rankings. The indirect benefit is that large or frequently-updated sites can get new and changed content crawled and indexed sooner — but crawling more efficiently is not itself a ranking signal.

How to verify it’s working

The ground truth is your server logs. Look for two things: Googlebot sending If-None-Match/If-Modified-Since request headers, and your server returning 304 responses to it. Real-world data is thin here, which is what makes Dave Smart’s server-log study at Tame the Bots valuable: monitoring verified Googlebot traffic, he found only about 1.3% of requests got a 304 — mostly 200s — and that the If-None-Match requests tended to cluster when a URL was requested again shortly after a prior fetch. His conclusion matches the guidance: infrequent on small sites, but potentially “significant savings” on heavily-crawled ones. Don’t expect a high 304 rate on a small site; do expect it to matter at scale.

Conditional requests vs. rate vs. frequency vs. budget

To keep the crawl-budget family straight:

• Crawl rate — how fast Googlebot fetches (supply side, throttled by server health).
• Crawl frequency — how often a known URL is re-fetched (popularity + staleness).
• Crawl budget — the demand + capacity envelope: the set of URLs Google can and wants to crawl.
• Conditional requests — how cheaply each individual recrawl can be answered. They don’t change how fast or how often Google crawls; they make each unchanged-page visit almost free, which is how you stop wasting budget on identical content.

AI summary

A condensed take on the Advanced version:

• Conditional requests = “has this changed?” Googlebot sends If-Modified-Since (validated against your Last-Modified) and/or If-None-Match (validated against your ETag). Unchanged → your server returns 304 Not Modified with no body and the crawler reuses its copy.
• ETag wins when both are present; Google recommends ETag (no date-format pitfalls) but says set both. Cache-Control: max-age is an optional supplementary recrawl-timing hint.
• A 304 is ~1 KB vs. 100 KB+ and skips re-download/re-processing — but it does not freeze indexing signals (Google may still recalculate them).
• Google doesn’t always send the headers (use-case dependent; AdsBot is likelier to), and you can proactively serve 304 even without a conditional header.
• Three misconfigurations defeat it: always-200 with a full body; volatile ETags (timestamps, per-request tokens, per-node CDN variance); and a Last-Modified that doesn’t track real changes (same honesty rule as sitemap lastmod).
• Edge case: a 304 served over a broken/empty page can make Googlebot treat the error as permanent (Illyes).
• Bing has supported RFC-compliant conditional GET since 2008 and tracks it as “crawl efficiency.”
• Adoption is low and falling (~0.026% → ~0.017% cacheable fetches over a decade); it’s an underused lever for large sites — and not a ranking factor.
• Verify in server logs (304 rate, If-None-Match presence). Real data (Tame the Bots) shows it’s rare on small sites, meaningful at scale.

Official documentation

Primary-source documentation on conditional requests and crawler caching.

Google

• Things to Know about Google’s Web Crawling — the current HTTP-caching contract: ETag/If-None-Match, Last-Modified/If-Modified-Since, ETag precedence, and the Cache-Control: max-age hint. (Note: this content moved here from the older /search/docs/... path; the guidance is the same.)
• Crawling December: HTTP caching — Gary Illyes’ Dec 2024 post behind that doc update (“Allow us to cache, pretty please”).
• Troubleshoot Google Search crawling errors — the If-Modified-Since/304 mechanics, the AdsBot example, and the “serve a 304 without a conditional header” allowance.
• HTTP status codes, network, and DNS errors — how Google treats a 304 (signals may still be recalculated; no negative effect on indexing).
• Optimize your crawl budget — the parent concept and who actually needs to care.
• Build and submit a sitemap — the lastmod honesty principle that mirrors the Last-Modified header discipline.

Bing / Microsoft

• Announcing crawler improvements for Live Search — Bing’s original 2008 RFC-2616 conditional-GET announcement (If-Modified-Since + If-None-Match/ETag).
• bingbot Series: Maximizing Crawl Efficiency — Bing’s “crawl efficiency” metric and why unchanged re-crawls lower it.
• Bing Webmaster Tools — Crawl Control — the rate-side lever, contrasted with conditional requests as the per-crawl efficiency lever.

Quotes from the source

On-the-record statements from Google and Bing. Each link is a deep link that jumps to the quoted passage on the source page.

Google — what’s supported, exactly

• “Google’s crawling infrastructure supports heuristic HTTP caching as defined by the HTTP caching standard, specifically through the ETag response- and If-None-Match request header, and the Last-Modified response- and If-Modified-Since request header.” — Google, Things to Know about Google’s Web Crawling. Jump to quote
• “If both ETag and Last-Modified response header fields are present in the HTTP response, Google’s crawlers use the ETag value as required by the HTTP standard.” Jump to quote
• “For Google’s crawlers specifically, we recommend using ETag instead of the Last-Modified header to indicate caching preference as ETag doesn’t have date formatting issues.” Jump to quote
• “Google’s crawlers that support caching will send the ETag value returned for a previous crawl of that URL in the If-None-Match header. If the ETag value sent by the crawler matches the current value the server generated, your server should return an HTTP 304 (Not modified) status code with no HTTP body.” Jump to quote

Google — when it sends the headers, and the proactive-304 allowance

• “Google generally supports the If-Modified-Since and If-None-Match HTTP request headers for crawling. Google’s crawlers don’t send the headers with all crawl attempts; it depends on the use case of the request (for example, AdsBot is more likely to set the If-Modified-Since and If-None-Match HTTP request headers).” — Google, Troubleshoot Google Search crawling errors. Jump to quote
• “If our crawlers send the If-Modified-Since header, the header’s value is the date and time the content was last crawled. Based on that value, the server may choose to return a 304 (Not Modified) HTTP status code with no response body, in which case Google will reuse the content version it crawled the last time.” Jump to quote
• “Independently of the request headers, you can send a 304 (Not Modified) HTTP status code and no response body for any Googlebot request if the content hasn’t changed since Googlebot last visited the URL. This will save your server processing time and resources, which may indirectly improve crawl efficiency.” Jump to quote

Gary Illyes, Google (LinkedIn — the 304-backfire failure mode)

• “HTTP 304 (not modified) is super useful to signal crawlers that the content they’re accessing hasn’t changed since it was last crawled, but it can also backfire spectacularly.” Read the post
• On how often the empty-page-then-304 trap actually happens: “Does this ever happen? Yes. Often? Absolutely not. But it’s worth keeping this somewhere deep in your mind because debugging it is an absolute nightmare.” Read the post

Bing (Live Search), Fabrice Canel (2008 conditional-GET announcement)

• “Live Search supports conditional get as defined by RFC 2616 (Section 14.25), and generally will not download the page unless it has changed since the last time it crawled it.” Jump to quote

Should I bother implementing conditional requests?

Not every site needs this. Work through whether it’s worth your time before you touch a server config.

SOP: implement conditional requests correctly

A repeatable procedure for adding (or fixing) conditional-request support on a large site. Do it in this order — measure, implement, verify.

1. Confirm you actually have the problem. Pull verified-Googlebot traffic from your server logs. If most requests to unchanged URLs return 200 with a full body and few or no If-None-Match / If-Modified-Since headers appear, you have room to improve. If you’re already seeing 304s on unchanged pages, stop — there may be little to gain.

2. Generate a stable, content-based ETag. Compute the ETag from a hash of the actual response body or a stable content-version identifier. Do not derive it from a timestamp, a per-request or per-session token, a request ID, or anything that varies per server node. Verify by requesting the same unchanged URL twice and confirming the ETag value is identical both times (and identical across your CDN/origin fleet).

3. Set an honest Last-Modified. Set Last-Modified to the date/time of the last significant content change — not the current render time, and not a value that moves when only the footer year or a timestamp changes. Format it to the HTTP standard (Weekday, DD Mon YYYY HH:MM:SS Timezone). If you can, set both ETag and Last-Modified; Google will prefer ETag.

4. Return 304 correctly. When an incoming If-None-Match matches your current ETag (or If-Modified-Since is not older than your Last-Modified) and nothing changed, return 304 Not Modified with no response body — status and headers only. Most web servers and frameworks can do this automatically; confirm yours isn’t stripped by a proxy or CDN in front of the origin.

5. (Optional) Add a Cache-Control max-age hint. Set Cache-Control: max-age=<seconds> to the number of seconds you expect the content to stay unchanged, as a supplementary recrawl-timing signal. It’s a hint, not a guarantee.

6. Verify in logs — then leave it alone. Re-check the logs after a few weeks. You want to see Googlebot’s If-None-Match requests met with 304s on unchanged pages, and 200s only when content genuinely changed. Don’t expect a high 304 rate on a small site; expect it to matter at scale.

Guardrail: never serve 304 for a page that actually changed, and never bolt conditional requests onto a flaky origin — a 304 served over a broken/empty page can make Googlebot treat the error as permanent.

Myths and mistakes to avoid

The recurring misconceptions that cause wasted effort or quiet breakage:

• “Googlebot always sends If-Modified-Since/If-None-Match, so if I don’t see it in my logs my server is broken.” False. Google says explicitly that it doesn’t send the headers on every crawl — it’s use-case dependent, and the main Search crawler sends them inconsistently (AdsBot is called out as more likely to). Real log data (Tame the Bots) showed well under 2% of requests got a 304 on the tested site.

• “A 304 means Google won’t re-evaluate my rankings/signals.” False. Google’s own docs note the indexing pipeline may still recalculate signals for the URL on a 304; only the content download and re-processing is skipped.

• “Setting an ETag automatically means I’ll get 304s.” False if the ETag is volatile. An ETag built from a timestamp, a per-request token, or per-node state changes on every request, so If-None-Match never matches — and it’s harder to diagnose because it looks correctly configured at a glance.

• “Last-Modified just needs to be present — any date works.” False. If it doesn’t track real content changes (stamped with “now” every render, or bumped by a footer tweak), it either triggers wasted re-crawls or erodes Google’s trust in the signal — the same principle Google states for sitemap lastmod.

• “304 is only ever a reactive answer to a conditional GET.” False. Google explicitly allows servers to proactively return 304 with no body for any Googlebot request when the server independently knows the content is unchanged.

• “Implementing conditional requests will boost my rankings.” Not established by any official source. The documented benefit is crawl/server efficiency, which can help large sites get new content crawled faster — it is not a ranking factor.

• “This only matters for giant enterprise sites.” Mostly true in urgency — Google targets this at large, rarely-changing-content sites — but the mechanism and the misconfigurations apply to any site. On a small site it’s just rarely worth the setup.

Check a URL’s caching validators with curl

See what validators your server sends, and whether it returns a 304 when you echo them back.

1) See the response headers (look for ETag and Last-Modified)

curl -sI https://example.com/some-page | grep -iE 'etag|last-modified|cache-control'
# ETag: "a1b2c3d4e5"
# Last-Modified: Thu, 22 Jan 2026 01:28:49 GMT
# Cache-Control: max-age=94043

2) Ask conditionally with the ETag — you want a 304 back

curl -sI https://example.com/some-page \
  -H 'If-None-Match: "a1b2c3d4e5"'
# HTTP/2 304        ← correct: server confirms nothing changed, sends no body
# HTTP/2 200        ← if you get this on an unchanged page, your validators aren't working

3) Ask conditionally with the date instead

curl -sI https://example.com/some-page \
  -H 'If-Modified-Since: Thu, 22 Jan 2026 01:28:49 GMT'
# HTTP/2 304

Detect a volatile ETag

If the same unchanged URL returns a different ETag on back-to-back requests, your ETag is volatile (timestamp/token-based) and If-None-Match will never match:

# Request twice; the two ETag values should be IDENTICAL for an unchanged page
for i in 1 2; do curl -sI https://example.com/some-page | grep -i '^etag:'; done
# ETag: "a1b2c3d4e5"
# ETag: "a1b2c3d4e5"   ← good (stable)
# ETag: "9f8e7d6c5b"   ← BAD if different: your ETag changes every request

Spot check 304s in your access logs

Verify real conditional-request activity from Googlebot. Adjust field positions to your log format (this assumes a combined-log-style status field):

# Count status codes returned to Googlebot
grep -i 'googlebot' access.log | awk '{print $9}' | sort | uniq -c | sort -rn
#   4211 200
#     53 304        ← these are the conditional-request wins
#     12 301

# See which URLs are getting 304s
grep -i 'googlebot' access.log | awk '$9==304 {print $7}' | sort | uniq -c | sort -rn | head

(Verify the request is really Googlebot with a reverse + forward DNS check before trusting the user-agent string — fake Googlebot traffic is common.)

Quick DevTools Console check in the browser

Paste into the Chrome DevTools Console to read the validators the browser received for the current page:

// Reads the ETag / Last-Modified the server sent for THIS page
fetch(location.href, { method: 'HEAD' }).then(r => {
  console.log('ETag:', r.headers.get('etag'));
  console.log('Last-Modified:', r.headers.get('last-modified'));
  console.log('Cache-Control:', r.headers.get('cache-control'));
});

The validator → request → response framework

1. Validator: the first 200 response supplies an ETag, Last-Modified, or both.
2. Request: a later fetch sends that value back in If-None-Match or If-Modified-Since.
3. Response: unchanged content returns 304 with no body; changed content returns 200 with the new body and validator.
4. Stability check: validators should change when representation content changes, not because a request hit another server or contained a timestamp.

Conditional-request header cheat sheet

Tools for checking validators

• HTTP Header Checker shows ETag, Last-Modified, cache headers, and header changes across redirects.
• Browser DevTools’ Network panel lets you inspect the initial validators and the conditional headers on a repeat request.
• curl is the clearest reproducible test: capture a validator, send it back, and confirm the unchanged response is 304.
• Access-log analysis reveals whether crawler conditional requests actually receive 304 responses at scale.

Metrics for conditional-request health

Conditional hit rate

Metric: eligible conditional requests returning 304. What it tells you: whether validators avoid unchanged-body transfers. How to pull it: group access-log requests carrying If-None-Match or If-Modified-Since by response status. Benchmark / realistic range: establish a baseline by template and change cadence; frequently updated pages should naturally differ from stable archives. Cadence: monthly, and after cache/CDN changes.

Bytes avoided by 304 responses

Metric: estimated response-body bytes not transferred. What it tells you: the bandwidth benefit rather than just response counts. How to pull it: compare each 304 URL with the latest 200 body size in logs or a crawl export. Benchmark / realistic range: compare against the site’s own prior period; no universal target is honest. Cadence: monthly.

Validator instability

Metric: unchanged URLs whose ETag or Last-Modified value changes between checks. What it tells you: whether per-request or per-node variance defeats revalidation. How to pull it: repeat identical header requests against a fixed sample. Benchmark / realistic range: validators should remain stable while the representation is unchanged. Cadence: after deployments, CDN changes, or load-balancer changes.

Resources worth your time

My related writing

• What Is 304 Not Modified? (Ahrefs SEO Glossary) — my walkthrough of the full conditional-request flow, the If-None-Match-takes-precedence rule, and why the 304 opportunity is “not that crucial” for small sites but a “great opportunity” for large ones.
• When Should You Worry About Crawl Budget? — the parent concept: what crawl budget actually is, demand vs. capacity, and who needs to care before you narrow into conditional requests.
• What Is Googlebot & How Does It Work? — how Googlebot decides what and how fast to crawl, the context around the scheduler.
• The Beginner’s Guide to Technical SEO — where crawl efficiency fits in the bigger picture.

My speaking

• How Search Works (SlideShare) — my walkthrough of crawling, rendering, indexing, and ranking, including the crawl-demand factors that decide what gets re-fetched. (My standing disclaimer applies: “This is my understanding of systems… not going to be 100% complete or accurate.”)

From around the industry

• Crawling December: HTTP caching (Google Search Central) — Gary Illyes’ post asking site owners to enable caching, and the source of the declining-adoption stat.
• Does Googlebot Use Etag Headers? (Dave Smart, Tame the Bots) — a rare real server-log study of Googlebot’s conditional-request behavior; most competing write-ups are pure spec paraphrase with no observed data.
• Google clarifies how Google’s crawlers handle cache control headers (Barry Schwartz, Search Engine Land) — coverage of the Dec 2024 doc update, with the core ETag/Last-Modified quotes reproduced verbatim.
• Google’s Updated Crawler Guidance Recommends ETags (Roger Montti, Search Engine Journal) — why Google prefers ETag, and the “individual crawlers may or may not use caching” nuance.
• Announcing crawler improvements for Live Search (Fabrice Canel, Bing Webmaster Blog) — Bing’s original 2008 RFC-2616 conditional-GET announcement, with a sample request/response.
• bingbot Series: Maximizing Crawl Efficiency (Bing Webmaster Blog) — Bing’s “crawl efficiency” framing, where unchanged re-crawls directly lower the metric.
• MDN: HTTP conditional requests (Mozilla) — the neutral spec-level reference for the validators and the 304 mechanics.

Test yourself: Conditional Requests

Five quick questions on ETag, If-Modified-Since, and 304 Not Modified. Pick an answer for each, then check.

Conditional Requests

A conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page.

Related: Crawl Budget, Caching for SEO

Conditional Requests

A conditional request is an HTTP request that only asks a server to send the full response if the content has changed since a known reference point. Instead of blindly re-downloading a page, the client (a browser or a search crawler like Googlebot) attaches one or both validators from the previous fetch:

• If-Modified-Since — a date, validated against your Last-Modified response header.
• If-None-Match — an opaque content fingerprint, validated against your ETag response header.

If the server determines nothing relevant has changed, it replies 304 Not Modified with no response body — just status and headers. The crawler then reuses the version it already has. If the content has changed, the server returns a normal 200 OK with the full body. When both validators are present, ETag/If-None-Match takes precedence, per the HTTP standard and Google’s own stated preference.

For SEO this is a crawl-efficiency lever, not a ranking one. A 304 is dramatically cheaper to generate and transmit than a full page (headers only — roughly a kilobyte instead of the whole document), so on large sites with many rarely-changing URLs, correctly answering conditional requests frees crawl budget for new and updated pages. Google doesn’t send the conditional headers on every crawl, and a 304 doesn’t freeze indexing signals — it only skips re-downloading and re-processing the body.

Related: Crawl Budget, Caching for SEO

Changelog