Cloudflare Workers SEO

How to make technical SEO changes on Cloudflare Workers — the fetch handler, HTMLRewriter for canonical/hreflang/JSON-LD injection, KV-backed redirects, the Cache API vs edge cache vs Cache-Control, the cloaking boundary, and how Bot Fight Mode can block Googlebot.

First published: Jul 3, 2026 · Last updated: Jul 18, 2026 · Advanced
AIAuthored byAI v2
1 evidence signal on this page

Cloudflare Workers SEO is doing technical SEO on Cloudflare's serverless runtime: a Worker only sees requests its route matches, and every one of those enters one fetch handler where you rewrite the request, the response headers, and the response body. Body rewriting runs through HTMLRewriter — the actual mechanism for injecting a canonical tag, fixing hreflang, or adding JSON-LD without a CMS deploy — and it needs to be idempotent against missing, duplicate, and non-HTML cases. Redirects at scale belong in KV or D1; Bulk Redirects/Rules are simpler for small sets, but pick one owner per URL so systems don't conflict. Three separate things share the word cache — the Workers Cache API, the Cloudflare edge cache, and origin Cache-Control — and conflating them is why an injected tag seems to not show up; diagnose by cache key, layer, TTL, and invalidation. The hard rule is cloaking: apply identical logic for Googlebot and users. The most Workers-specific self-inflicted wound is Bot Fight Mode blocking Googlebot on a pipeline your WAF allow-rules don't even reach. Ship every SEO-affecting change with recorded version metadata and a tested rollback, then verify with GSC URL Inspection and the CF-Cache-Status header. See the Edge SEO hub for the general concept.

TL;DR — A Cloudflare WorkerA serverless function that runs at Cloudflare's global edge, close to every user. only sees requests matched by its configured route, and it intercepts each one in a single fetch handler, where you do three things in sequence: rewrite the request, rewrite the response headers, and rewrite the response body via HTMLRewriter. That’s the real mechanism behind “inject a canonical” or “fix a title” — and it needs to be idempotent, tested against missing, duplicate, and non-HTML responses, not just the happy path. RedirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't. at scale live in KV (fast key lookup) or D1 (relational); Bulk Redirects/Rules cover small sets more simply, and I generally prefer edge-level redirects over server-level — but pick one owner per URL, since a Worker redirect, a Bulk Redirect, and an origin redirect can all fire on the same path. Three different things share the word “cache” — the Workers Cache API (caches.default), the Cloudflare edge cache, and origin Cache-Control — and confusing them is the usual cause of “my tag didn’t show up”; diagnose staleness by cache key, layer, TTL, and invalidation rather than guessing. Google’s own ETagA conditional request lets a crawler (or browser) ask a server 'has this changed since I last fetched it?' — using If-Modified-Since (checked against your Last-Modified header) and/or If-None-Match (checked against your ETag). If nothing changed, the server replies 304 Not Modified with no body, so the crawler reuses its existing copy instead of re-downloading the page. / If-None-Match / 304 guidance is directly actionable for a Worker that owns the response. The cloaking line: identical logic for every requester. The most Workers-specific self-inflicted wound is BotA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. Fight Mode, which runs outside the WAF Ruleset Engine, so ordinary “allow” rules don’t reach it. Ship every SEO-affecting change with recorded version metadata, a tested rollback, and a stop condition — then verify with GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. and CF-Cache-Status.

What this article is (and isn’t)

This is the practitioner, code-level companion to the Edge SEOEdge SEO (serverless SEO) is the practice of making technical SEO changes — redirects, meta tags, canonicals, hreflang, robots.txt, structured data — at the CDN/edge worker layer before the response reaches the user or crawler, instead of editing the origin server or CMS. hub. The hub owns the general definition, the platform comparison table (Workers, Akamai, Fastly, Lambda@Edge, Vercel, Netlify), the Snippets-vs-Workers decision, and the full treatment of the cloaking rule. I’m not re-deriving any of that here. This page goes one level deeper into Cloudflare Workers specifically — the runtime this very site’s own worker runs on, wired in through run_worker_first in wrangler.toml — with real APIs rather than “edge compute can inject tags” hand-waving.

A note before the code: Google has no Cloudflare-Workers-specific documentation. The official guidance that governs this (cloaking policy, HTTP cachingCaching stores a copy of a page or resource — in a browser, a CDN edge node, or a search crawler's own cache — so it can be served again without regenerating or re-downloading it. It isn't a direct ranking factor, but it feeds page speed and crawl efficiency., CDN crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.) is general and applies to any edge implementation. I’d rather say that plainly than imply a Google doc exists that doesn’t.

How a Worker sits in the request/response path

A Worker is a serverless script running on V8 isolates. Every request it’s routed to enters through a fetch handler. Evidence for this claim A Cloudflare Worker receives HTTP requests through a fetch handler. Scope: Cloudflare Workers handlers. Confidence: high · Verified: Cloudflare Workers: Fetch handler “Routed to” is doing real work in that sentence: a Worker only sees requests that match its configured route or custom domain — everything else never reaches the fetch handler at all. Where two routes could both match the same URL, the more specific pattern takes precedence, so before you trust a Worker’s behavior for a given URL, confirm the route actually matches it and check which deployed version is live on that route (Wrangler environments and gradual rollouts mean the version serving traffic isn’t always the one in your editor). Inside the handler you can do three distinct things, in order:

  1. Rewrite the request before it goes to your origin.
  2. Rewrite the response headers on the way back out.
  3. Rewrite the response body — via HTMLRewriter.

Here’s the minimal shape:

export default {
  async fetch(request, env, ctx) {
    // 1. (optionally) inspect/modify the request
    const response = await fetch(request);   // hit the origin

    // 2. rewrite headers
    const headers = new Headers(response.headers);
    headers.set("X-Robots-Tag", "index, follow");

    // 3. rewrite the body with HTMLRewriter (see next section)
    return new Response(response.body, { ...response, headers });
  },
};

The SALT.agency team, who coined “edge SEOEdge SEO (serverless SEO) is the practice of making technical SEO changes — redirects, meta tags, canonicals, hreflang, robots.txt, structured data — at the CDN/edge worker layer before the response reaches the user or crawler, instead of editing the origin server or CMS.” off Cloudflare Workers research, built their tooling as a filter chain — a request filter, a response filter, and a body filter. That’s the same three-phase pattern; they just named it. Keeping those three phases separate in your head keeps a Worker legible.

Rewriting HTML with HTMLRewriter

HTMLRewriter is Cloudflare’s streaming HTML parser, and it’s the actual API behind every “inject a tag” trick. Evidence for this claim Cloudflare HTMLRewriter provides selector-based handlers that can transform streamed HTML elements. Scope: Cloudflare Workers HTMLRewriter API. Confidence: high · Verified: Cloudflare Workers: HTMLRewriter You register .on(selector, handler) element handlers, and the handler gets getAttribute / setAttribute, prepend / append, setInnerContent, and replace. Because it streams, you’re not buffering the whole document in memory.

Injecting or fixing a canonical tag

class CanonicalHandler {
  constructor(url) { this.url = url; }
  element(el) { el.setAttribute("href", this.url); }
}

const rewriter = new HTMLRewriter()
  .on('link[rel="canonical"]', new CanonicalHandler("https://example.com/preferred/"));

return rewriter.transform(response);

If the page has no canonical at all, you attach a handler to head and append one instead of editing an existing tag. Either way, remember the lesson from the canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. side of this: rel=canonical is a hint, not a command — a Worker lets you set it consistently across a whole platform, but Google still decides.

The CanonicalHandler above assumes a tag already exists and the response is HTML. Neither is guaranteed in production, and getting it wrong is how you end up with two canonical tagsA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. on one page instead of one. Before you ship a rewrite like this, make it idempotent and test it against:

  • No existing canonical — your handler needs to detect the missing case and append one to head, not silently no-op on link[rel="canonical"] matching nothing.
  • A duplicate or malformed canonical already present — decide whether you remove the stray tag or leave your rewrite to add a second one (the latter is a real bug, not an edge case — duplicate canonicals are a common self-inflicted issue).
  • A non-HTML response — an API route, an image, or a redirect response passed through the same Worker shouldn’t be run through HTMLRewriter at all; scope the transform to routes and content types you’ve actually checked.
  • Running the transform twice on the same response (a retry, a nested fetch) — confirm it doesn’t re-append a second tag.

Adding or correcting hreflang alternates

Same mechanism, driven off config. You append one link[rel="alternate"] per locale to head. If your alternates are per-locale and relational, that config belongs in D1; if it’s a flat lookup, KV is fine. The point is that HTMLRewriter injects them identically for every requester — you’re not branching on user agentA user agent is the HTTP request header a client (browser, crawler, or bot) sends to identify itself. For crawlers, a short user-agent token — a substring of that string — is what robots.txt rules actually target..

Injecting JSON-LD structured data

new HTMLRewriter().on("head", {
  element(head) {
    head.append(
      `<script type="application/ld+json">${JSON.stringify(schema)}</script>`,
      { html: true }
    );
  },
});

CPU limits that bite at scale

One competitor claim I’d push back on is “sub-millisecond, no constraints.” The real ceiling is CPU time: 10 ms on the free plan, 30 ms on paid (wall-clock time waiting on fetch doesn’t count — CPU time does). For typical rewrites you’ll never notice. For heavy HTMLRewriter passes over very large pages, it’s a real constraint to design around, not scaremongering.

Redirects at the edge: KV vs D1 vs Rules

I typically prefer to have redirects on the edge (CDN level) over having them on the server — it offloads the work from your origin and applies before the page is ever generated. On Cloudflare specifically, in my Ahrefs guide to redirects for SEO I laid out that you’ve got several options: single or bulk redirects, redirect rules, page rules, or Workers with key-value pairs — or a Worker that modifies headers to add a redirect.

For a Worker-driven table, KV is the natural home: a fast, eventually-consistent key lookup keyed by URL.

export default {
  async fetch(request, env) {
    const url = new URL(request.url);
    const target = await env.REDIRECTS.get(url.pathname);   // KV namespace
    if (target) return Response.redirect(target, 301);
    return fetch(request);
  },
};

Reach for D1 when redirects are relational (per-locale, per-segment SQL you want to query). And know when a Worker is overkill: for a small, static set of redirects, Cloudflare’s Bulk Redirects or Redirect Rules are simpler and need no code at all. Don’t hand-roll a KV Worker for fifty redirects.

Pick one owner for a given URL and don’t let a Worker redirect, a Bulk Redirect, a Redirect Rule, and an origin redirect all apply to the same path — they’re separate systems that can each fire on the same request, and when more than one matches, you’re debugging precedence instead of a clean redirect. Before you add a redirect anywhere, check whether one already exists for that path in the other systems, and pick the layer based on match complexity (simple 1:1 vs. pattern-based), scale, and who needs to observe or roll it back — a Worker redirect lives in your code and logs; a Bulk Redirect or Rule lives in the dashboard and is easier for a non-developer to audit or revert.

Caching: three different things, one confusing name

This is the section competitor pages skip, and it’s the one that generates the most “why didn’t my change show up” confusion. Three separate layers share the word cache:

  • The Workers Cache API — caches.default and caches.open(). This is a Worker-scoped programmable cache you read and write in code.
  • The Cloudflare edge cache — the CDN cache that serves your assets. Distinct from the Cache API.
  • Origin Cache-Control — the headers your origin (or your Worker) sets, which influence both of the above and what GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. does.

Conflate them and you’ll swear a change didn’t deploy when it’s just being served from a layer you didn’t purge.

When a change genuinely isn’t showing up, don’t guess — diagnose it layer by layer:

  1. Cache key. What request attributes determine whether two requests hit the same cached entry (URL, and sometimes headers or cookies if your cache key includes them)? A rewrite that varies by something not in the cache key can serve the wrong variant.
  2. Which layer served the response. Check CF-Cache-Status (HIT/MISS/EXPIRED/DYNAMIC) to see whether the edge cache answered at all, or the request reached your Worker.
  3. Location/state. Cloudflare’s cache is distributed across data centers — a purge or a fresh deploy doesn’t necessarily invalidate every edge location instantly.
  4. TTL and the rule that set it. Confirm whether a cache rule, a Cache-Control header from your origin, or a header your Worker itself set is controlling the TTL.
  5. Invalidation. Did you purge the specific URL, purge everything, or rely on TTL expiry? A Worker-owned Cache API entry (caches.default) needs its own explicit delete() — purging the CDN cache doesn’t touch it.

What Googlebot does with ETag / If-None-Match / 304

If your Worker generates or rewrites the response, it owns the caching headers — which means Google’s December 2024 HTTP caching guidance is directly actionable for you. Google supports heuristic HTTP caching through ETag/If-None-Match and Last-Modified/If-Modified-Since, strongly recommends ETag because it’s less error-prone, and says that when the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.’s ETag matches, your server should return a 304 Not ModifiedHTTP 304 Not Modified is the response to a conditional GET/HEAD whose condition evaluates false, in the 3xx class but not a redirect: it has no Location header and no body. It tells a client its cached copy is still valid. For SEO it has no direct ranking effect and no indexing effect beyond a possible signal recalculation, though it can help crawl efficiency indirectly on large sites. with no body. A response-generating Worker can implement exactly that: compute an ETag, compare it against If-None-Match, and short-circuit to a 304 itself — saving compute and giving Googlebot a fast, cacheable signal.

The max-age recrawl tradeoff

Google also says to consider setting Cache-Control: max-age to help crawlers decide when to recrawl. The catch for a Worker that rewrites HTML: an aggressive max-age on a page whose Worker-injected tags just changed can delay Googlebot from seeing the update you just shipped. Don’t slap a long cache lifetime on rewritten HTML and forget about it.

The cloaking boundary, applied to Workers

The hard rule, in Workers terms: run the same logic for every requester. Google’s spam policy defines cloaking as presenting different content to users and search engines to manipulate rankings, and specifically calls out inserting text or keywords only when the requester is a search engine.

A couple of clarifications, because people over-correct here:

  • Inspecting the User-Agent isn’t automatically cloaking. Logging bot traffic, or serving a cached response faster to any client, is fine. The line is a content difference by requester identity, done to manipulate rankings.
  • Page-split A/B testing on Workers is fine. Splitting users by URL and treating every requester the same is legitimate. Splitting by who’s asking — bot vs. human — is not.

A worked example of the safe pattern: this site’s own preview gate is a Worker that 404s any /preview/ path unless a cookie matches a secret. It returns that 404 to everyone without the cookie — Googlebot included. That’s precisely the safe shape: it isn’t hiding one thing from bots and showing another to users; it applies one rule uniformly.

And don’t lean on a bot-only pre-render step even if you build it cleanly on Workers. Google has called dynamic renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. a workaround and not a long-term solution; a Worker that pre-renders only for bots inherits that deprecation.

How a Worker can accidentally block or slow Googlebot

This is the most Workers-specific way to shoot yourself in the foot, and it’s usually not in your Worker code.

Bot Fight Mode runs outside the Ruleset Engine

Bot Fight Mode (and Super Bot Fight Mode) can produce false positives against legitimate crawlers, Googlebot included. The trap: Bot Fight Mode is evaluated on a separate pipeline from the WAF Ruleset Engine, so your ordinary WAF “allow” or “skip” custom rules don’t override it. If Bot Fight Mode is challenging Googlebot, you don’t fix it with an allow-rule — you have to change or disable the mode itself. (Confirm the current mechanics against Cloudflare’s Bot Fight Mode and Super Bot Fight Mode docs before you rely on it — bot products change.)

The verified-bots custom rule pattern

Cloudflare exposes a cf.client.bot field and a verified-bots allow pattern so you can permit known-good crawlers in your custom rules — useful for the WAF side, though (per above) it does not reach Bot Fight Mode.

TIP Verify the crawler before changing a Worker or WAF rule

A crawler user-agent string is only a claim. Match the source to the operator's published ranges before granting or denying special treatment.

Spot-check the source IP with my free Googlebot Verifier Free

  1. Copy the claimed user-agent and source IP from the edge log.
  2. Compare the IP with the crawler operator’s published ranges instead of trusting the header.
  3. Apply the verified-bot rule consistently, then compare the final response seen by users and the crawler.
A mismatch is evidence not to trust this request as verified Googlebot; it is not evidence about all Googlebot traffic.

The Googlebot Verifier row lists IP 203.0.113.9, a Googlebot user-agent claim, and a published-range mismatch marked for spot-checking.

The CDN itself is neutral-to-positive

To be clear about the myth: Cloudflare-the-CDN doesn’t hurt SEO. Google’s own 2024 Crawling December work notes that Google increases crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. when it detects a CDN — but that a CDN can also accidentally block Googlebot via WAF/bot rules, and that a 503 is better than a bot-verification interstitial. The risk is a misconfigured Worker or bot setting, not the infrastructure.

Verifying what Googlebot actually received

After any Worker deploy, confirm what a crawler actually got — don’t assume:

  • GSC URL Inspection → Test Live URL. Fetches the page as Google and shows the rendered HTML, so you can confirm your injected canonical/hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others./JSON-LDJSON-LD (JavaScript Object Notation for Linked Data) is a script-based structured data format, typically paired with the schema.org vocabulary to describe page content for search engines and AI systems. Google recommends it over Microdata and RDFa because it's the easiest format to implement and maintain at scale — but all three work, and structured data isn't a ranking signal. is actually present.
  • Check CF-Cache-Status alongside the HTML. HIT / MISS / EXPIRED tells you whether you’re looking at a fresh Worker response or a cached one — the fastest way to catch a “change didn’t show up” that’s really a cache-layer issue.
  • Fetch as Googlebot directly. Request with Googlebot’s user agent and compare — but remember matching the string proves nothing about identity; verify real Googlebot with reverse + forward DNS against Google’s published ranges (see the Scripts tab).

Deployment hygiene specific to Workers

A successful wrangler deploy tells you the script shipped — it doesn’t tell you Googlebot is getting the right rendered output. Treat every SEO-affecting Worker change as a release with a record, not just a push:

  • Scope your routes. Don’t run a Worker on /* by default. Match it to the paths it needs in your wrangler.toml route patterns so a bug can’t take down your whole site.
  • Check current limits before you promise scale. CPU time, subrequest counts, and script size limits vary by plan and change over time — confirm against Cloudflare’s current limits page before you design a rewrite around a specific ceiling, rather than relying on a remembered number.
  • Record version metadata for the release. Cloudflare’s versions and deployments model tracks the source version, compatibility date, bindings, and routes for each deploy — note which version is live on which route so a “the Worker is doing X” claim is checkable against what’s actually deployed, not what’s in your editor.
  • Version and roll back with Wrangler environments. Ship to a staging environment, gradually roll out by percentage, and keep the ability to revert to the prior version instantly.
  • Use scoped logs to monitor the rollout — with their limits in mind. Cloudflare’s Workers Logs and log tailing can support debugging a gradual rollout, but logs are sampled and retained for a limited window — treat them as scoped evidence for the requests they captured, not a complete record of every crawler visit.
  • Set a stop condition and test the rollback before you need it. Decide up front what observed behavior (error rate, a wrong response on a spot-check, a crawl-rate drop) halts the rollout, and confirm the rollback path actually works rather than assuming it will.
  • Purge cache as part of the deploy. Since three cache layers are in play, make cache purge/invalidation an explicit step of shipping a rewrite, not an afterthought.

A Bing note, and one forward-looking thing

Bing has no Cloudflare/edge-specific guidance either. But because a Worker deploy is instant while crawls aren’t, IndexNow is the natural pairing — fire it the moment a Worker-driven redirect table or tag change ships so Bing (and other participating engines) re-crawl promptly. And worth a glance: Cloudflare shipped edge-enforced canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. as a product feature (“Redirects for AI Training”) — verified AI-training crawlers get a 301 to your canonical URL with one toggle. It’s a useful contrast to hand-rolling canonical logic in your own Worker, and a reminder that “serving crawlers something different from users” is a pattern Microsoft has publicly been skeptical of on Cloudflare’s other AI-crawler features — a good gut check on any bot-conditional Worker.

For the broader picture — the platform comparison, Snippets vs. Workers, the dev-queue and governance angles — head back to the Edge SEOEdge SEO (serverless SEO) is the practice of making technical SEO changes — redirects, meta tags, canonicals, hreflang, robots.txt, structured data — at the CDN/edge worker layer before the response reaches the user or crawler, instead of editing the origin server or CMS. hub.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.