Technical SEO

TL;DR — Technical SEO is the behind-the-scenes work that lets search engines find, read, and understand your pages. It isn’t a trick that pushes you up the rankings — it’s the foundation that lets your content and links do that. If the technical side is broken, even great pages can’t show up. The whole thing collapses to one sentence: you can’t rank a page Google won’t index. This guide walks you through what to care about, how to check your own site, and what to safely ignore.

What technical SEO actually is

When people first learn SEO, they think about two things: content (writing good pages) and links (getting other sites to point at yours). Technical SEO is the third leg, and it comes before the other two can matter. It’s everything that decides whether a search engine can reach your page, read it correctly, and file it away in its index.

In my beginner’s guide to technical SEO I’ve described it for years as the practice of helping search engines find, crawl, understand, and index your pages. That’s the whole job in four verbs. Notice what’s not in there: writing great copy, choosing keywords, earning links. Those are real and important — they’re just not technical SEO.

The analogy I keep coming back to is a house. Content is the furniture and the paint. Links are the neighbors recommending the place. Technical SEO is the plumbing and the wiring — invisible, unglamorous, and the thing that ruins everything when it breaks. Nobody compliments your plumbing. They sure notice when it stops working.

The pipeline everything runs on

Google Search describes the process in three stages. Its documentation is blunt about it: “Google Search works in three stages, and not all pages make it through each stage.” Evidence for this claim Google Search describes crawling, indexing, and serving as three stages; discovery is part of the crawling stage, and not every page advances through each stage. Scope: Google Search documentation; conceptual explanation, not a promise of ranking outcomes. Confidence: high · Verified: 2026-07-14 Google: How Search Works

1. Crawl — a bot (Googlebot for Google, Bingbot for Bing) discovers your URL and downloads the page.
2. Index — the engine works out what the page is about and stores it in a giant database of everything it might show.
3. Serve (rank) — when someone searches, the engine pulls the best matches from that database and puts them in order.

That clause — not all pages make it through each stage — is the entire reason technical SEO exists. A page can be crawled but not indexed, or indexed but never shown for a query. Most technical SEO is just removing whatever is stopping your pages from passing through those gates.

Why it’s a foundation, not a trick

Here’s the part that surprises people: technical SEO usually doesn’t make a page rank higher. There’s no “technical SEO algorithm” to game. What it does is clear the roadblocks so your good content and links can actually count. Fixing a broken canonical tag or a slow server doesn’t add points — it stops you from losing them.

I sometimes put it this way: technical SEO is the most important part of SEO until it isn’t. The moment your pages can be crawled and indexed, the leverage shifts to content and links, which move rankings far more than most technical projects ever will. So the goal isn’t technical perfection — it’s clearing the gates and then getting out of your own way.

The foundations every site needs

You can ignore a surprising amount of technical SEO. But there’s a short list every site should get right:

• Crawl access. Make sure you’re not accidentally blocking pages you want found. Your robots.txt file controls which URLs bots may request. The classic beginner mistake is using it to try to hide a page — that’s not what it’s for (more on that in the myths section). Evidence for this claim A robots.txt rule controls crawling rather than guaranteeing removal from Google Search; a URL can still appear when Google cannot crawl it. Scope: Google Search crawler behavior. Other crawlers can interpret robots.txt differently. Confidence: high · Verified: 2026-07-14 Google: Introduction to robots.txt
• A sitemap. An XML sitemap is a list of your important URLs handed straight to search engines. Submit it in Google Search Console and Bing Webmaster Tools. It matters most for big or brand-new sites.
• One clear version of each page. If the same content lives at several URLs (with and without www, http vs https, tracking parameters), tell search engines which one is the canonical — the real one — with a rel="canonical" tag. This is canonicalization, and it’s most of what beginners need to know about duplicate content.
• HTTPS. Serve your site over a secure connection. It’s a small ranking signal and basic table stakes for trust.
• Reasonable speed and a mobile-friendly layout. Google indexes the mobile version of your site, so it has to work on a phone. Don’t obsess over speed scores (see the myths) — just don’t be painfully slow.

That’s genuinely most of it for a normal site. Crawlable, has a sitemap, one canonical version per page, HTTPS, works on mobile, not unbearably slow.

How to check your own technical SEO

You don’t need expensive tools to start. The free ones the search engines give you are the ground truth:

• Google Search Console → Page Indexing report. This tells you which pages are indexed and why the others aren’t. It’s the single most useful screen in technical SEO. If an important page isn’t indexed, this is where you find out.
• The URL Inspection tool (also in Search Console). Paste any URL and Google tells you exactly how it crawled, rendered, and indexed that page — and lets you request indexing.
• Bing Webmaster Tools. Bing’s equivalent, and worth setting up — its index now feeds a lot of AI answers, so it matters more than its search market share suggests.

Start there. If your important pages are indexed and your Page Indexing report isn’t full of surprises, your technical SEO is probably fine.

What technical SEO is not

A lot of confusion comes from people filing everything under “technical SEO.” It’s not:

• Content quality or keyword research — that’s on-page SEO and content strategy.
• Link building — that’s off-page SEO.
• Writing title tags and meta descriptions — that’s on-page, though it lives right on the boundary.

The boundaries genuinely blur in places — schema markup, internal links, and page speed all straddle technical and on-page. Don’t worry about which bucket they’re in. The label matters less than getting the work done.

Common myths to ignore

Half of being good at technical SEO is not wasting time on things that don’t matter. The big ones:

• “Blocking a page in robots.txt removes it from Google.” It doesn’t. Google says plainly that robots.txt “is not a mechanism for keeping a web page out of Google.” A blocked page can still show up if other sites link to it — Google just can’t see what’s on it. To actually remove a page, let it be crawled and add a noindex tag.
• “I need to worry about crawl budget.” Almost certainly not. Most sites never need to think about it; it only matters for very large or rapidly changing sites. Evidence for this claim Google says crawl-budget guidance is mainly relevant to very large sites, including sites with over one million unique pages or over ten thousand rapidly changing pages. Scope: Google Search guidance; the page-count examples are diagnostic starting points, not hard eligibility thresholds. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide
• “Core Web Vitals are a huge ranking factor.” They’re a real but minor signal. I generally don’t prioritize them for rankings unless a site is extremely slow. Improve them for your users, not for an imagined ranking boost. Evidence for this claim Google says Core Web Vitals are used by ranking systems, while page experience does not override more relevant content. Scope: Google Search ranking guidance; no fixed weight or ranking-position effect is promised. Confidence: high · Verified: 2026-07-14 Google: Core Web Vitals and Search Google: Page experience
• “You need a perfect 100 PageSpeed score.” No. Those lab scores aren’t what Google ranks on — it uses real-world field data. Chasing 100 is a waste.
• “Duplicate content is a penalty.” It isn’t. Google calls some duplication “normal.” It’s a canonicalization problem, not a danger.
• “IndexNow tells Google about my pages.” It doesn’t — Google doesn’t use IndexNow. It’s a Bing-and-others thing.

Technical SEO in the age of AI search

AI search — Google’s AI Overviews, ChatGPT, Perplexity — runs on the same plumbing. These systems still have to crawl, read, and understand your pages before they can cite you, and there’s a new wave of AI crawlers (GPTBot, ClaudeBot, PerplexityBot) fetching the web. So the foundations in this guide aren’t going away in the AI era — if anything, being technically clean is now what makes you eligible to be quoted in an AI answer at all.

Where to go next

When you’re ready for the practitioner version — the pipeline in real detail, crawl budget thresholds, the rendering decision, canonicalization’s many signals, and how to run technical SEO as an ongoing system — switch to the Advanced Guide tab.

And this page is the hub for the whole Technical SEO pillar. The deep dives are organized into How Search Works (crawling, discovery, indexing, rendering), Site Migrations, On-Page meta tags, Search Engine Tools (Search Console and Bing Webmaster Tools), and JavaScript SEO — all in the sidebar.

TL;DR — Technical SEO is the same crawl → render → index → serve pipeline on every site — there’s no separate “technical SEO algorithm” — and it’s a foundation, not a ranking factor of its own. Treat the pipeline as a series of gates and diagnose which one a page is stuck at before changing anything. The leverage is mostly negative (not losing what you’ve earned), so the boring structural work — canonicalization, redirects, internal links — pays best, and it compounds at scale. Most sites don’t need to manage crawl budget; rendering is a separate step that can lag; Core Web Vitals are three distinct problems and a minor ranking lever; canonicalization is a weighted decision across ~40 signals; and from 2025 on, AI search gates eligibility on clean technical signals before it ranks or cites you. The highest skill here is prioritization — knowing what to ignore.

Technical SEO decides eligibility, not position

Technical SEO is the one part of SEO whose payoff is almost entirely negative: its job is to keep you from losing rankings, not to win them. Google doesn’t hand out positions for clean plumbing. The same crawling, indexing, and ranking systems run whether your site is pristine or a disaster — there’s no separate “technical SEO algorithm” sitting behind them. What technical SEO actually decides is whether your pages can enter those systems at all, and whether the engine understands them correctly once they’re in.

So the right mental model isn’t “do technical SEO to rank.” It’s “do technical SEO so your content and links are allowed to rank.” That inversion is the whole reason the unglamorous work — canonicalization, redirects, internal links — is the highest-value work, and why the single most useful skill in this discipline is prioritization: knowing what to fix and, just as often, what to leave alone.

The pipeline, as gates

Everything hangs off one pipeline, and the operative clause from Google is “not all pages make it through each stage.” Evidence for this claim Google Search describes crawling, indexing, and serving as three stages; discovery is part of the crawling stage, and not every page advances through each stage. Scope: Google Search documentation; conceptual explanation, not a promise of ranking outcomes. Confidence: high · Verified: 2026-07-14 Google: How Search Works Don’t picture a conveyor belt that carries every page to the finish. Picture a series of gates, each with its own pass/fail:

• Crawl — discovery (links + sitemaps + push protocols) plus the fetch. A page nothing links to, or one disallowed in robots.txt, may never arrive.
• Render — Google runs your JavaScript in a recent headless Chrome (the Web Rendering Service) before it can fully understand the page. This is a separate step from the fetch, it’s stateless, and it can lag.
• Index — the engine processes the page, picks a canonical among duplicates, and decides whether to store it. “Indexing isn’t guaranteed” even when crawling and rendering succeed.
• Serve — query understanding, then ranking across many automated systems, then the search features layered on top.

Keep crawl ≠ render ≠ index ≠ rank separate in your head and most of technical SEO stops being mysterious. When a page underperforms, you don’t guess and you don’t change ten things — you find which gate it failed and fix that stage.

One honest caveat before you treat any pipeline description as gospel, including mine: it’s a model, not the source code. How Search Works is a talk I give at conferences that walks through this whole pipeline (slides on SlideShare), and I open it with a warning I’ll repeat here: “this is my understanding of systems… not going to be 100% complete or accurate.” Hold it loosely and use it to reason about problems.

Who actually does the crawling

“Googlebot” sounds like one program. It’s a family — desktop, mobile (the one that matters, since indexing is mobile-first), image, news, video, and ads crawlers — all drawing from the same crawl budget pool, which is why a runaway image or parameter crawl can starve the crawling of your real content.

And it’s not just search engines anymore. When I analyzed Cloudflare Radar crawl data (an Ahrefs piece I wrote on the new wave of bots), search-engine crawlers still crawled the most, but AI bots were firmly in second place and on track to overtake them. If you read your logs, the cast of characters has changed — and managing it (which AI crawlers you allow, and confirming the ones hitting you are who they claim) is now part of the job.

Crawl budget: when it matters, and when it doesn’t

Google defines crawl budget as “the set of URLs that Google can and wants to crawl,” set by crawl capacity (your server’s health) and crawl demand (popularity and staleness). You raise effective budget two ways: give bots more capacity, or — far more often — stop wasting it. Consolidate duplicates, block low-value spaces, return 404/410 for permanently gone pages, fix soft 404s, keep sitemaps current with accurate lastmod, and avoid long redirect chains.

The reassuring part, and I’ll keep saying it: most sites don’t need to worry about crawl budget. Google itself tells you that if your pages are generally crawled the same day they’re published, “you don’t need to read this guide.” It starts to bite around 1M+ pages, or 10k+ pages that change rapidly. Below that, spend your energy elsewhere. Evidence for this claim Google says crawl-budget guidance is mainly relevant to very large sites, including sites with over one million unique pages or over ten thousand rapidly changing pages. Scope: Google Search guidance; the page-count examples are diagnostic starting points, not hard eligibility thresholds. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide Bing’s Fabrice Canel frames the same idea more bluntly: less is more — fewer URLs to crawl is better for SEO.

robots.txt: crawl control, not index control

The most important distinction in this whole file: robots.txt controls crawling, not indexing. Disallowing a URL stops bots from fetching it — it does not keep it out of the index. A disallowed page can still be indexed (URL only, no content) if other pages link to it, and worse, if you disallow a page you also stop Google from ever seeing a noindex tag on it.

So the rules are:

• Want a page gone from search? Allow crawling and add noindex. Never use robots.txt to deindex.
• Want bots to skip a low-value URL space (internal search, infinite faceted combinations) and don’t care about indexing? robots.txt disallow is correct.
• Managing AI crawlers? This is also where you allow or block GPTBot, ClaudeBot, PerplexityBot, CCBot and friends — a strategic decision, not a default.

Canonicalization: a weighted decision, not a command

Canonicalization is where a lot of advanced technical SEO lives, and it’s widely misunderstood. rel="canonical" is a hint, not a directive. Google weighs it against many other signals — redirects, internal links, sitemap inclusion, HTTPS, URL structure — when it picks the representative URL. My deep dive on canonicalization puts the count around 40 signals that feed canonical selection, which is why you sometimes see “Duplicate, Google chose different canonical than user” in Search Console: your tag was outvoted.

The practical implications:

• Don’t send conflicting signals. I spent years on enterprise sites (I ran technical SEO in-house at IBM), and in a talk I give called Enterprise SEO Chaos I show real pages that “redirected to one version, canonicaled to a second, and internally linked to a third.” Pick one URL and make every signal agree.
• Signal strength roughly ranks redirect > rel="canonical" > internal links > sitemap. A 301 is a much stronger statement than a canonical tag.
• Duplicate content isn’t a penalty. Google’s Gary Illyes has said roughly 60% of the web is duplicate content, and Google treats some of it as normal — not a spam violation. The cost is split signals and wasted crawling, not a punishment. The fix is consolidation, not panic.

And a note on JavaScript: I once ran a test — injecting a rel="canonical" via JavaScript on a page that had none in the HTML — and Google honored it, even though it had publicly said it wouldn’t. After that surfaced, Google updated its JavaScript SEO documentation. The lesson isn’t “use JS canonicals”; it’s that this stuff is testable, and the docs aren’t always the last word.

The rendering decision

Rendering is the step most overviews skip, and it’s where JavaScript sites get into trouble. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Evidence for this claim Google processes JavaScript pages in crawling, rendering, and indexing phases and uses a recent version of Chrome for rendering. Scope: Google Search JavaScript processing; rendering and indexing remain subject to technical and quality constraints. Confidence: high · Verified: 2026-07-14 Google: JavaScript SEO basics It’s a separate, stateless service, it can cache resources for weeks, and it can lag the initial fetch — so a JS-dependent change can take a while to be reflected.

JavaScript isn’t the enemy here. As I put it in my guide to JavaScript SEO, JavaScript is not bad for SEO, and it’s not evil — it’s just different from what many SEOs are used to. The real decision is how you render:

• Server-side rendering (SSR) — safest for SEO; the HTML arrives complete.
• Static generation (SSG/pre-rendering) — best of both worlds for content that doesn’t change per request.
• Client-side rendering (CSR) — highest risk; the content only exists after JS runs, so you’re betting on the render step.
• Dynamic rendering — Google calls it a workaround, not a recommendation; Bing is more favorable. Treat it as a bridge, not a destination.

Two traps to know cold. First, lazy-loading: Googlebot doesn’t scroll or click, so content that only loads on interaction can stay invisible — make sure it loads when it’s in the viewport. Second, links: Google can only follow a link that’s a real <a href> element. A routerLink or a click handler with no href is not a crawlable link. Verify the rendered output against the raw HTML with the URL Inspection tool whenever you suspect a gap.

Site architecture and internal linking

Internal links do three jobs at once: they help bots discover pages, they distribute PageRank, and they pass topical context through anchor text. John Mueller has called internal linking “super critical for SEO” and one of the biggest levers you have on your own site — and I agree. It’s one of the highest-ROI things you control directly.

A few systems-level points:

• Orphan pages — pages nothing links to — are the first thing to hunt for. If it’s not linked, it’s barely discoverable and gets almost no equity.
• Architecture is crawl-funnel management. Important pages belong close to the home page; deep, click-distant pages get crawled less and rank worse.
• PageRank sculpting with nofollow is dead (since 2009). Nofollowing internal links makes that equity evaporate rather than redistribute. Manage flow with real architecture, not nofollow tricks.

Core Web Vitals: three problems, not one

The biggest practitioner error with page experience is treating it as a single “make the site faster” problem. Core Web Vitals are three distinct problems with different root causes and different fixes:

• LCP (Largest Contentful Paint) — loading. Driven by server response time, render-blocking resources, and how fast the main content asset loads. Target under 2.5 seconds.
• INP (Interaction to Next Paint) — interactivity. Driven by JavaScript execution blocking the main thread. Target under 200 milliseconds. (INP replaced FID in 2024 — if you still see FID anywhere, the advice is stale.)
• CLS (Cumulative Layout Shift) — visual stability. Driven by images without dimensions, late-loading fonts, and injected content. Target under 0.1.

Two things matter beyond the definitions. Field data, not lab data: Google ranks on real-user CrUX data, not your Lighthouse score, so a Lighthouse 65 with good field data beats a Lighthouse 100 with bad field data. And proportion: I’ll be honest — I don’t think Core Web Vitals have much impact on SEO, and unless a site is extremely slow I generally won’t prioritize fixing them for rankings. Do the work for users and conversions; just don’t oversell it as a ranking lever.

Structured data: signals for search and AI

Structured data (use JSON-LD) doesn’t rank you, but it makes pages eligible for rich results and increasingly helps AI systems parse your content for citation. It’s genuinely useful — and genuinely oversold as a ranking signal. My honest framing: most of SEO is doing the basics well, and content and links move the needle more than schema does. Implement it where it unlocks a rich result or clarifies an entity; don’t expect it to lift rankings on its own. (And note: schema markup URLs are not crawlable internal links — Mueller has confirmed this.)

International, briefly

If you serve multiple languages or regions, use distinct URLs per version and hreflang annotations to map them, and prefer ccTLDs or subdirectories over URL parameters. Don’t auto-redirect by IP — Google explicitly warns against it and it breaks crawling. International SEO is deep enough to be its own pillar; this is just the technical handshake.

Technical SEO is an ongoing system, not a one-time audit

The framing every competitor guide gets wrong: technical SEO is not a checklist you complete once. Sites change constantly — deploys break canonical tags, a release slips a noindex into a template, a new ad script tanks INP, redirect chains accumulate. The mature practice is monitoring and regression detection:

• Watch GSC Page Indexing for sudden changes in indexed counts and excluded statuses.
• Watch Crawl Stats and your logs for response-code spikes and crawl-pattern shifts.
• Re-validate crawl, render, and redirects after every significant deployment.

On log files specifically: I used to treat them as a once-every-few-years troubleshooting tool. That’s changed. Logs are now the clearest place to see which AI crawlers are actually hitting you and how often — something no other tool shows you as directly — so for anyone who cares about AI search, they’ve become a lot more useful than they were.

Site migrations: the highest-stakes event

A migration — new domain, HTTP to HTTPS, a replatform, a URL restructure — is the single highest-stakes technical event, because it touches every URL at once. Map old to new 1:1, use 301/308 permanent redirects, keep them in place indefinitely (I would not rush to remove them — a couple of redirect hops is nothing to worry about), and use the GSC Change of Address tool where it applies. Migrations can be complex and involve a lot of people, but don’t panic — you can fix almost anything that goes wrong. There’s a full Site Migrations cluster under this pillar.

Technical SEO for AI search

The modern shift, and it cuts against the lazy “technical SEO is dead” take: from 2025 on, AI search systems decide eligibility before they ever rank or cite. To be quoted in an AI answer your page generally has to be cleanly canonicalized, fast enough, renderable without heroics, and structured enough to be parsed with confidence. Messy signals don’t just lower a ranking now — they can remove you from the answer entirely. Because Bing’s index feeds many LLM answers, Bing Webmaster Tools and IndexNow matter more than Bing’s search share suggests. Technical hygiene matters more in the AI era, not less.

Where the leverage actually is

If you take one thing from this guide, make it prioritization. Spend your time on indexing, canonicalization, internal links, and clean migrations — the work that decides whether pages exist in search and consolidate their equity. Don’t lose sleep over crawl budget, Core Web Vitals, duplicate content, or short redirect chains unless you have a specific, diagnosed problem. And don’t chase perfection — I doubt there’s a major website that’s technically perfect, and if there were, I’d worry they were wasting resources on things that don’t matter instead of things that do.

This hub maps the rest of the pillar: How Search Works, Site Migrations, On-Page, Search Engine Tools, and JavaScript SEO. Start wherever your site is breaking — the pipeline tells you which gate to look at first.

AI summary

A condensed take on the Advanced Guide:

• No separate algorithm. Same crawl → render → index → serve pipeline on every site. Technical SEO decides whether pages can enter the system and be understood — it’s a foundation, not a ranking trick. The leverage is mostly negative: not losing what content and links earned.
• Treat the pipeline as gates. “Not all pages make it through each stage.” Diagnose which gate (crawl, render, index, serve) a page failed before changing anything.
• The one-liner: you can’t rank a page Google won’t index — so the boring structural work (canonicalization, redirects, internal links) pays best and compounds at scale.
• Crawl budget: capacity + demand. Most sites never need to manage it (matters ~1M+ pages, or 10k+ rapidly changing). Bing: less is more.
• robots.txt controls crawling, not indexing. To remove a page: allow crawling + noindex. It’s also where you allow/block AI crawlers.
• Canonicalization is a weighted decision across ~40 signals; rel=canonical is a hint, not a command. Don’t send conflicting signals; duplicate content isn’t a penalty.
• Rendering is separate and can lag. JS isn’t evil, just different. Choose SSR / static / CSR / dynamic deliberately; watch the lazy-loading and <a href> link traps.
• Internal linking is “super critical” (Mueller). Hunt orphans; architecture is crawl-funnel management; nofollow sculpting is dead.
• Core Web Vitals = three problems (LCP/INP/CLS), judged on field data, not Lighthouse — and a minor ranking lever (Patrick doesn’t prioritize them for rankings).
• Monitoring, not a one-time audit. Watch Page Indexing, Crawl Stats, and logs; re-validate after deploys. Logs are newly useful for spotting AI crawlers.
• Migrations are the highest-stakes event — map 1:1, 301, keep redirects.
• AI search gates eligibility on clean technical signals before ranking/citing; Bing’s index feeds LLMs, so Bing Webmaster Tools + IndexNow matter more than its share.

Official documentation

The primary-source docs that anchor the whole technical-SEO pillar.

Google

• In-Depth Guide to How Google Search Works — the crawl → index → serve pipeline, URL discovery, rendering, indexing, and serving. The single most important page in technical SEO.
• SEO Starter Guide — Google’s own beginner orientation and the Search Essentials baseline.
• Crawling and Indexing — the hub for robots.txt, sitemaps, canonicalization, and crawl controls.
• Optimize your crawl budget — crawl capacity + demand, and who actually needs to care.
• JavaScript SEO basics — rendering as part of indexing and how to keep JS content indexable.
• Core Web Vitals & Page Experience — what the page-experience signals are and how they’re used.
• Site moves with URL changes — Google’s migration playbook: redirects, Change of Address, and what to monitor.
• Inside Googlebot (March 2026) — current crawl economics and byte limits.

Bing / Microsoft

• How Bing delivers search results — Bing’s crawl → index → rank pipeline and its named ranking factors.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawling and its “crawl efficiency north star.”
• IndexNow / indexnow.org — the push protocol for instantly signaling changed URLs; pairs with sitemaps.

Quotes from the source

On-the-record statements from Google and Bing that anchor the technical-SEO pillar. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the pipeline

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “Googlebot uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote
• “Indexing isn’t guaranteed; not every page that Google processes will be indexed.” Jump to quote

Google — crawl budget and robots.txt

• “Taking crawl capacity and crawl demand together, Google defines a site’s crawl budget as the set of URLs that Google can and wants to crawl.” Jump to quote
• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Jump to quote
• “A robots.txt file tells search engine crawlers which URLs the crawler can access on your site.” Jump to quote

Bing — Fabrice Canel, Microsoft

• “Crawling is the process by which bingbot discovers new and updated documents or content to be added to Bing’s searchable index.” Jump to quote
• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” Read the interview

Google reps, on the record (reproduced through industry coverage)

• John Mueller: “Technical SEO is not going away, it continues to be the foundation of everything built on the open web.” Coverage (Search Engine Journal)
• Gary Illyes, on crawl budget: “the vast majority of the people don’t have to care about it.” Coverage (Search Engine Journal)
• John Mueller, on internal links: “internal linking is super critical for SEO… one of the biggest things that you can do on a website.” Coverage (Search Engine Journal)
• John Mueller, on Core Web Vitals: “it’s more than a tie-breaker, but it also doesn’t replace relevance.” Coverage (Search Engine Journal)
• Gary Illyes, on Core Web Vitals priority: “If you don’t have anything better to do on your site, go do Core Web Vitals.” (Pubcon AMA) Coverage (Search Engine Land)
• Martin Splitt, on rendering: “two-wave indexing… plays less and less of a role.” Coverage (Onely)

Patrick Stox (my own work)

• “This is my understanding of systems and is based on a lot of public statements from Google and my own knowledge. Warning: It’s not going to be 100% complete or accurate.” — slide 3 of my How Search Works deck. See the slide

The mental models

1. There is no technical SEO algorithm. Same crawl → index → serve pipeline, same ranking systems, on every site. Technical SEO decides whether your pages can enter the system and be understood — it doesn’t hand you ranking points. Before you hunt for a “technical trick,” ask which ordinary stage is failing.

2. Foundation, not factor. The leverage of technical SEO is mostly negative: it stops you from losing what content and links earned. You can’t rank a page Google won’t index — so most of the job is clearing obstacles, not adding signals.

3. Four gates, not a conveyor belt. Crawl → render → index → serve. Each is a filter, and “not all pages make it through each stage.” When a page underperforms, locate which gate it failed before you change anything.

4. The four “not equals.”

• Crawling ≠ indexing (a blocked page can still be indexed; indexing isn’t guaranteed even when crawling succeeds).
• Crawling ≠ ranking (crawl rate is not a ranking signal).
• Crawling ≠ rendering (JavaScript runs in a separate step that can lag).
• Indexing ≠ ranking (being in the index doesn’t win you queries).

5. The leverage rule at scale. On big sites everything is template-level, so impact multiplies. One error can keep millions of pages out of the index; one canonical fix can recover a fortune. The boring structural work outranks the shiny stuff.

6. Prioritization is the real skill. The hardest and most valuable thing in technical SEO is knowing what to ignore. Fix indexing, canonicalization, internal links, and migrations; don’t sweat crawl budget, Core Web Vitals, or duplicate content without a diagnosed problem.

7. AI raises the bar, not lowers it. AI search decides eligibility (clean canonicalization, schema, fast renderable pages) before it ranks or cites. Messy technical signals can remove you from the answer entirely.

Technical SEO foundations checklist

A first pass to confirm search engines can find, read, and understand your site:

• Crawlable. Important pages are linked from somewhere crawlable (no orphans); robots.txt doesn’t block anything you want indexed.
• Discoverable. An XML sitemap is submitted in Google Search Console and Bing Webmaster Tools, listing only canonical, indexable URLs with accurate lastmod.
• Renderable. JS-dependent content is reachable via real <a href> links, not click-only navigation; critical content doesn’t depend on a slow render.
• Indexable. No stray noindex on pages you want found; check the GSC Page Indexing report for excluded statuses.
• Canonical. One canonical URL per piece of content; duplicates point to it; no canonical chains or canonicals pointing at the homepage; signals agree (redirect, canonical, internal links, sitemap all point the same way).
• Healthy server. Fast, stable responses — minimal 5xx/timeouts (bots slow down when your server struggles).
• Clean redirects. No long redirect chains or loops; permanent moves use 301/308; gone pages return 404/410.
• No URL waste. Parameters, faceted nav, and session IDs aren’t generating infinite or duplicate URL spaces (spider traps).
• On-page meta. Unique, accurate title tags and meta descriptions; correct meta robots directives.
• Migration-safe. If you’re moving anything (domain, HTTPS, platform, URLs), map old→new 1:1 redirects and use the GSC Change of Address tool where it applies.
• AI-eligible. Valid schema, consistent entity signals, AI-crawler access decided deliberately, and fast renderable pages so AI search can parse and cite you.

How much should you actually care?

The most useful thing I can give a technical SEO isn’t a checklist — it’s a sense of proportion. A lot of “best practices” get far more attention than they earn. Here’s my honest ranking of where the leverage is, and where it isn’t.

The pattern: spend your time on indexing, consolidation, and clean migrations; don’t lose sleep over crawl budget, Core Web Vitals, or duplicate content unless you have a specific, diagnosed problem. And don’t chase technical perfection — I doubt there’s a major website that’s technically perfect, and if there were, I’d worry they were wasting resources on things that don’t matter instead of things that do.

Which control does what

The other thing people mix up constantly — these are four separate tools for four separate jobs:

The classic mistake is using robots.txt to deindex — if you block crawling, Google can’t see the noindex tag and may keep the page in results via links from other sites. Want a page gone? Allow crawling and add noindex.

Tools for technical SEO

• Google Search Console — your ground truth for how Google treats the site: the Page Indexing report, Crawl Stats, the URL Inspection tool (crawl/render/index status for a single URL), and the Change of Address tool for migrations.
• Bing Webmaster Tools — crawl info, Crawl Control, Site Scan, and IndexNow — and it matters more than its share suggests because Bing’s index feeds many LLM answers.
• Site crawlers / audits — Ahrefs Site Audit and Screaming Frog SEO Spider simulate a crawl and surface redirect chains, duplicate URLs, blocked pages, broken canonicals, and trap-like patterns at scale.
• Server log file analysis — the only place you see exactly what bots crawled and where they wasted budget — and increasingly the clearest way to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot) are hitting you. Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.
• PageSpeed Insights / Lighthouse — page-experience and rendering checks; pair with field (CrUX) data for how real users and the renderer experience the page.
• IndexNow — push changed URLs to Bing (and others) in real time instead of waiting to be re-crawled.

Durable technical SEO metrics

Intended index coverage trend

• What it measures: Whether canonical, indexable URLs you intend search engines to serve are actually represented in the index over time.
• How to pull it: Compare the XML sitemap or approved indexable URL inventory with Search Console Page Indexing data, then inspect excluded reasons by template.
• Cadence: Review after material releases and on a recurring schedule suited to the site’s publishing and crawl rate.
• Healthy direction: Intended indexed URLs remain stable or grow with approved content while unexplained exclusions and duplicate canonical conflicts decline.
• Decision triggered: Investigate template-level crawling, canonical, rendering, duplication, or noindex problems before investing in more content for the same affected area.

Core Web Vitals field pass rate by template

• What it measures: The share of real-user page groups passing field Core Web Vitals, separated by template and device rather than averaged across the site.
• How to pull it: Use Search Console Core Web Vitals and CrUX/PageSpeed Insights field data; map affected URL groups to the responsible template or component.
• Cadence: Track through the field-data window and compare before and after major performance releases.
• Healthy direction: More important templates move into passing groups without regressions shifting to another device, region, or page type.
• Decision triggered: Prioritize a shared template or component fix when a poor group affects meaningful traffic or user experience; do not chase tiny lab-only changes as a ranking trick.

Crawl reliability and response mix

• What it measures: Whether search-engine crawling reaches useful URLs reliably instead of spending capacity on errors, chains, traps, or low-value URL spaces.
• How to pull it: Combine Search Console Crawl Stats with server logs and crawler reports, segmented by status, host, directory/template, and bot.
• Cadence: Monitor continuously for incidents and review trends after platform, CDN, redirect, faceting, or migration changes.
• Healthy direction: Stable successful responses for important URLs, fewer 5xx failures and redirect chains, and less repeated crawling of known trap spaces relative to the site’s own baseline.
• Decision triggered: Fix availability or routing first; then adjust internal discovery, parameter handling, or crawl controls for persistent waste.

Resources worth your time

My technical SEO writing (Ahrefs)

• The Beginner’s Guide to Technical SEO — the full framework: find, crawl, understand, and index.
• Enterprise Technical SEO — what technical SEO looks like at scale (and why perfection is the wrong goal).
• Crawl Budget: Everything You Need to Know — when it matters, and the (many) cases where it doesn’t.
• JavaScript SEO: Issues & Best Practices — “JavaScript is not bad for SEO… it’s just different.”
• Canonicalization: A Beginner’s Guide — the ~40 signals Google weighs, and why most duplicates aren’t nefarious.
• Redirects for SEO — the 11 types and how long to keep them (longer than you think).
• Website Migration: A Complete Guide — “you can fix almost anything that goes wrong.”
• Core Web Vitals & PageSpeed — why I don’t prioritize CWV for rankings.

My speaking

• How Search Works (SlideShare) — my full walkthrough of crawling, rendering, indexing, and ranking.
• Enterprise SEO Chaos (SMX) — the IBM-scale stories: 24 URL variations, 14-hop redirect chains, and signals all pointing different directions.

From around the industry

• web.dev — Core Web Vitals — Google’s web-platform docs on LCP, INP, and CLS: what each measures and how to fix it, separate from search.
• Onely blog — a technical-SEO agency known for deep rendering and indexing research (their two waves of indexing write-up is a good example).
• Search Engine Roundtable — Barry Schwartz’s near-daily log of what Google and Bing reps actually say; the fastest way to track statement changes.
• Search Engine Journal — Technical SEO — ongoing coverage and explainers, and the source for several rep statements quoted above.
• Search Engine Land — SEO — industry news and conference coverage (Pubcon/SMX AMAs with Google’s team).
• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• r/TechSEO — the community for crawl/index/render debugging.

Podcasts

• Search Off the Record (Google Search Relations) — Gary Illyes and Martin Splitt on how Googlebot crawls, renders, and indexes. The closest thing to official commentary on the pipeline. Listen
• Voices of Search — Prioritizing SEO Efforts, my conversation on why prioritization is the hardest part of the job and how I triage high-impact / low-effort work. Listen
• TheeDigital — Debunking SEO Myths — me on the technical myths that won’t die (duplicate-content penalties, keyword density, subdomains vs. subfolders). Listen

Videos

• Google Search Central (YouTube) — the How Google Search Works series and Martin Splitt’s crawling/rendering and JavaScript SEO explainers. Channel

Stats worth citing

A mix of my own research and third-party figures (Google, Microsoft, and Cloudflare):

• Rich results can lift CTR. Google’s own published case studies report a 25% higher CTR for Rotten Tomatoes’ marked-up pages, a 35% increase in visits for Food Network, and an 82% higher CTR on Nestlé’s rich-result pages vs. non-rich. Google — structured data intro
• Rendering costs ~20× more than crawling. From my JavaScript SEO talks: Ahrefs crawled ~7B pages a day but rendered ~80M JavaScript pages using around 600 servers — a useful sense of why rendering is rationed and can lag. Deck
• Roughly 60% of the web is duplicate content — Gary Illyes (Google) — which is exactly why some duplication is “normal” and canonicalization, not panic, is the right response. Google — consolidate duplicate URLs
• Bing discovers tens of billions of new URLs every day — Fabrice Canel (Microsoft) — the scale of the discovery-and-filtering problem behind “less is more.” Coverage (Search Engine Roundtable)
• AI bots are a clear #2 and closing on search-engine bots — from Cloudflare Radar crawl data (my analysis of it): search bots still crawl the most, but AI crawlers are on pace to overtake them within a couple of years. Source
• 95.2% of sites have 3XX redirects; 72.9% are missing meta descriptions (my study) — across 1M+ domains in Ahrefs Site Audit. But don’t over-react to the meta-description number: Google rewrites them about 62.78% of the time, and they aren’t a ranking factor. Source

Test yourself: Technical SEO

Five quick questions on what technical SEO is and the crawl → index → serve pipeline. Pick an answer for each, then check.

Technical SEO

Technical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own.

Related: Crawling, Indexing, Rendering, Site Migration

Technical SEO

Technical SEO is the work of making a website easy for search engines to crawl, render, index, and serve. It’s the plumbing layer of SEO: the part that determines whether your pages can be found and understood at all, before anything about content quality or links even comes into play. There’s no separate “technical SEO algorithm” — search engines run the same crawl → render → index → serve pipeline on every site. Technical SEO is just the discipline of keeping that pipeline clear.

The reason it matters is simple: you can’t rank a page Google won’t index. If crawling, rendering, indexing, or canonicalization are broken, even excellent content stays invisible. That’s why technical SEO is best thought of as a foundation rather than a tactic — it doesn’t make a page rank by itself, but its absence can stop everything else from working.

The same pipeline applies whether you have fifty pages or fifty million, but scale changes the stakes. On a large site, fixes happen at the template level, so one error can keep millions of pages out of the index and one fix can recover a fortune in link equity. And from 2025 on, AI search systems decide eligibility — clean canonicalization, valid schema, fast and renderable pages — before they ever rank or cite you. That makes technical hygiene more important, not less.

Related: Crawling, Indexing, Rendering, Site Migration

Changelog