Technical SEO

TL;DR — Technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. is the behind-the-scenes work that lets search engines find, read, and understand your pages. It isn’t a trick that pushes you up the rankings — it’s the foundation that lets your content and links do that. If the technical side is broken, even great pages can’t show up. The whole thing collapses to one sentence: you can’t rank a page Google won’t indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. This guide walks you through what to care about, how to check your own site, and what to safely ignore.

What technical SEO actually is

When people first learn SEO, they think about two things: content (writing good pages) and links (getting other sites to point at yours). Technical SEO is the third leg, and it comes before the other two can matter. It’s everything that decides whether a search engine can reach your page, read it correctly, and file it away in its index.

In my beginner’s guide to technical SEO I’ve described it for years as the practice of helping search engines find, crawl, understand, and index your pages. That’s the whole job in four verbs. Notice what’s not in there: writing great copy, choosing keywords, earning links. Those are real and important — they’re just not technical SEO.

The analogy I keep coming back to is a house. Content is the furniture and the paint. Links are the neighbors recommending the place. Technical SEO is the plumbing and the wiring — invisible, unglamorous, and the thing that ruins everything when it breaks. Nobody compliments your plumbing. They sure notice when it stops working.

The pipeline everything runs on

Google Search describes the process in three stages. Its documentation is blunt about it: “Google Search works in three stages, and not all pages make it through each stage.” Evidence for this claim Google Search describes crawling, indexing, and serving as three stages; discovery is part of the crawling stage, and not every page advances through each stage. Scope: Google Search documentation; conceptual explanation, not a promise of ranking outcomes. Confidence: high · Verified: 2026-07-14 Google: How Search Works

1. Crawl — a botA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. (GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. for Google, BingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. for Bing) discovers your URL and downloads the page.
2. Index — the engine works out what the page is about and stores it in a giant database of everything it might show.
3. Serve (rank) — when someone searches, the engine pulls the best matches from that database and puts them in order.

That clause — not all pages make it through each stage — is the entire reason technical SEO exists. A page can be crawled but not indexed, or indexed but never shown for a query. Most technical SEO is just removing whatever is stopping your pages from passing through those gates.

Why it’s a foundation, not a trick

Here’s the part that surprises people: technical SEO usually doesn’t make a page rank higher. There’s no “technical SEO algorithm” to game. What it does is clear the roadblocks so your good content and links can actually count. Fixing a broken canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content. or a slow server doesn’t add points — it stops you from losing them.

I sometimes put it this way: technical SEO is the most important part of SEO until it isn’t. The moment your pages can be crawled and indexed, the leverage shifts to content and links, which move rankings far more than most technical projects ever will. So the goal isn’t technical perfection — it’s clearing the gates and then getting out of your own way.

The foundations every site needs

You can ignore a surprising amount of technical SEO. But there’s a short list every site should get right:

• Crawl access. Make sure you’re not accidentally blocking pages you want foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.. Your robots.txt file controls which URLs bots may request. The classic beginner mistake is using it to try to hide a page — that’s not what it’s for (more on that in the myths section). Evidence for this claim A robots.txt rule controls crawling rather than guaranteeing removal from Google Search; a URL can still appear when Google cannot crawl it. Scope: Google Search crawler behavior. Other crawlers can interpret robots.txt differently. Confidence: high · Verified: 2026-07-14 Google: Introduction to robots.txt
• A sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing.. An XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. is a list of your important URLs handed straight to search engines. Submit it in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility.. It matters most for big or brand-new sites.
• One clear version of each page. If the same content lives at several URLs (with and without www, http vs https, tracking parameters), tell search engines which one is the canonical — the real one — with a rel="canonical" tag. This is canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., and it’s most of what beginners need to know about duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling..
• HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.'. Serve your site over a secure connection. It’s a small ranking signal and basic table stakes for trust.
• Reasonable speed and a mobile-friendly layout. Google indexes the mobile version of your site, so it has to work on a phone. Don’t obsess over speed scores (see the myths) — just don’t be painfully slow.

That’s genuinely most of it for a normal site. Crawlable, has a sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., one canonical version per page, HTTPS, works on mobile, not unbearably slow.

How to check your own technical SEO

You don’t need expensive tools to start. The free ones the search engines give you are the ground truth:

• Google Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. → Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. This tells you which pages are indexed and why the others aren’t. It’s the single most useful screen in technical SEO. If an important page isn’t indexed, this is where you find out.
• The URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. (also in Search Console). Paste any URL and Google tells you exactly how it crawled, rendered, and indexed that page — and lets you request indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• Bing Webmaster Tools. Bing’s equivalent, and worth setting up — its index now feeds a lot of AI answers, so it matters more than its search market share suggests.

Start there. If your important pages are indexed and your Page Indexing report isn’t full of surprises, your technical SEO is probably fine.

What technical SEO is not

A lot of confusion comes from people filing everything under “technical SEO.” It’s not:

• Content quality or keyword research — that’s on-page SEO and content strategy.
• Link building — that’s off-page SEO.
• Writing title tagsThe title tag is the HTML title element in a page's head that specifies the document's title. It's the primary source for the SERP title link and a confirmed light ranking factor — but since August 2021 Google doesn't always show it verbatim. and meta descriptionsThe meta description is an HTML head tag — `<meta name=\"description\" content=\"…\">` — that suggests a short summary of the page for the search snippet. It's not a Google ranking factor, and Google rewrites it the majority of the time, but a good one can still lift click-through. — that’s on-page, though it lives right on the boundary.

The boundaries genuinely blur in places — schema markupSchema markup is code that uses the schema.org vocabulary to label what your content means so search engines can understand it and show rich results. It's most often written in JSON-LD, and it's not a direct ranking factor., internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., and page speed all straddle technical and on-page. Don’t worry about which bucket they’re in. The label matters less than getting the work done.

Common myths to ignore

Half of being good at technical SEO is not wasting time on things that don’t matter. The big ones:

• “Blocking a page in robots.txt removes it from Google.” It doesn’t. Google says plainly that robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. “is not a mechanism for keeping a web page out of Google.” A blocked page can still show up if other sites link to it — Google just can’t see what’s on it. To actually remove a page, let it be crawled and add a noindex tag.
• “I need to worry about crawl budgetThe number of URLs an engine will crawl in a timeframe..” Almost certainly not. Most sites never need to think about it; it only matters for very large or rapidly changing sites. Evidence for this claim Google says crawl-budget guidance is mainly relevant to very large sites, including sites with over one million unique pages or over ten thousand rapidly changing pages. Scope: Google Search guidance; the page-count examples are diagnostic starting points, not hard eligibility thresholds. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide
• “Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. are a huge ranking factor.” They’re a real but minor signal. I generally don’t prioritize them for rankings unless a site is extremely slow. Improve them for your users, not for an imagined ranking boost. Evidence for this claim Google says Core Web Vitals are used by ranking systems, while page experience does not override more relevant content. Scope: Google Search ranking guidance; no fixed weight or ranking-position effect is promised. Confidence: high · Verified: 2026-07-14 Google: Core Web Vitals and Search Google: Page experience
• “You need a perfect 100 PageSpeed score.” No. Those lab scores aren’t what Google ranks on — it uses real-world field dataPerformance metrics captured from real users, not lab tests.. Chasing 100 is a waste.
• “Duplicate content is a penalty.” It isn’t. Google calls some duplication “normal.” It’s a canonicalization problem, not a danger.
• “IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. tells Google about my pages.” It doesn’t — Google doesn’t use IndexNow. It’s a Bing-and-others thing.

Technical SEO in the age of AI search

AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. — Google’s AI OverviewsAI Overviews are the AI-generated summary box Google shows above or within its regular search results, written by Gemini models from pages retrieved out of Google's normal Search index. It's a Search feature, not a separate platform or index., ChatGPT, Perplexity — runs on the same plumbing. These systems still have to crawl, read, and understand your pages before they can cite you, and there’s a new wave of AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. (GPTBot, ClaudeBot, PerplexityBot) fetching the web. So the foundations in this guide aren’t going away in the AI era — if anything, being technically clean is now what makes you eligible to be quoted in an AI answer at all.

Where to go next

When you’re ready for the practitioner version — the pipeline in real detail, crawl budget thresholds, the renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. decision, canonicalization’s many signals, and how to run technical SEO as an ongoing system — switch to the Advanced Guide tab.

And this page is the hub for the whole Technical SEO pillar. The deep dives are organized into How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. (crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., discovery, indexing, rendering), Site MigrationsA site migration is any significant change to a website's URL structure, domain, platform, protocol, or hosting that can affect how search engines crawl, index, and rank it. The risk scales with how much you change at once., On-Page meta tagsMeta tags are HTML elements in a page's head that pass metadata about the page to search engines and browsers. For SEO only a few matter — the title element, the meta description, and the robots meta tag — while meta keywords and most others are ignored., Search Engine ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.' (Search Console and Bing Webmaster Tools), and JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript. — all in the sidebar.

TL;DR — Technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. is the same crawl → render → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline on every site — there’s no separate “technical SEO algorithm” — and it’s a foundation, not a ranking factor of its own. Treat the pipeline as a series of gates and diagnose which one a page is stuck at before changing anything. The leverage is mostly negative (not losing what you’ve earned), so the boring structural work — canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. — pays best, and it compounds at scale. Most sites don’t need to manage crawl budgetThe number of URLs an engine will crawl in a timeframe.; renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is a separate step that can lag; Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. are three distinct problems and a minor ranking lever; canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. is a weighted decision across ~40 signals; and from 2025 on, AI search gates eligibility on clean technical signals before it ranks or cites you. The highest skill here is prioritization — knowing what to ignore.

Technical SEO decides eligibility, not position

Technical SEO is the one part of SEO whose payoff is almost entirely negative: its job is to keep you from losing rankings, not to win them. Google doesn’t hand out positions for clean plumbing. The same crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking systems run whether your site is pristine or a disaster — there’s no separate “technical SEO algorithm” sitting behind them. What technical SEO actually decides is whether your pages can enter those systems at all, and whether the engine understands them correctly once they’re in.

So the right mental model isn’t “do technical SEO to rank.” It’s “do technical SEO so your content and links are allowed to rank.” That inversion is the whole reason the unglamorous work — canonicalization, redirects, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them. — is the highest-value work, and why the single most useful skill in this discipline is prioritization: knowing what to fix and, just as often, what to leave alone.

The pipeline, as gates

Everything hangs off one pipeline, and the operative clause from Google is “not all pages make it through each stage.” Evidence for this claim Google Search describes crawling, indexing, and serving as three stages; discovery is part of the crawling stage, and not every page advances through each stage. Scope: Google Search documentation; conceptual explanation, not a promise of ranking outcomes. Confidence: high · Verified: 2026-07-14 Google: How Search Works Don’t picture a conveyor belt that carries every page to the finish. Picture a series of gates, each with its own pass/fail:

• Crawl — discovery (links + sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. + push protocols) plus the fetch. A page nothing links to, or one disallowed in robots.txt, may never arrive.
• Render — Google runs your JavaScript in a recent headless Chrome (the Web RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. Service) before it can fully understand the page. This is a separate step from the fetch, it’s stateless, and it can lag.
• Index — the engine processes the page, picks a canonical among duplicates, and decides whether to store it. “Indexing isn’t guaranteed” even when crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and rendering succeed.
• Serve — query understanding, then ranking across many automated systems, then the search features layered on top.

Keep crawl ≠ render ≠ index ≠ rank separate in your head and most of technical SEO stops being mysterious. When a page underperforms, you don’t guess and you don’t change ten things — you find which gate it failed and fix that stage.

One honest caveat before you treat any pipeline description as gospel, including mine: it’s a model, not the source code. How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. is a talk I give at conferences that walks through this whole pipeline (slides on SlideShare), and I open it with a warning I’ll repeat here: “this is my understanding of systems… not going to be 100% complete or accurate.” Hold it loosely and use it to reason about problems.

Who actually does the crawling

“GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer.” sounds like one program. It’s a family — desktop, mobile (the one that matters, since indexing is mobile-first), image, news, video, and ads crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — all drawing from the same crawl budgetThe number of URLs an engine will crawl in a timeframe. pool, which is why a runaway image or parameter crawl can starve the crawling of your real content.

And it’s not just search engines anymore. When I analyzed Cloudflare Radar crawl data (an Ahrefs piece I wrote on the new wave of bots), search-engine crawlers still crawled the most, but AI bots were firmly in second place and on track to overtake them. If you read your logs, the cast of characters has changed — and managing it (which AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. you allow, and confirming the ones hitting you are who they claim) is now part of the job.

Crawl budget: when it matters, and when it doesn’t

Google defines crawl budget as “the set of URLs that Google can and wants to crawl,” set by crawl capacity (your server’s health) and crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. (popularity and staleness). You raise effective budget two ways: give bots more capacity, or — far more often — stop wasting it. Consolidate duplicates, block low-value spaces, return 404/410 for permanently gone pages, fix soft 404sA soft 404 is a URL that returns a success status code (usually 200 OK) even though the page is empty, missing, or shows a 'not found' message. It isn't a status code a server sends — it's a label search engines apply after comparing the response code against the rendered content, and they treat the page like a 404 for indexing., keep sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. current with accurate lastmod, and avoid long redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency..

The reassuring part, and I’ll keep saying it: most sites don’t need to worry about crawl budget. Google itself tells you that if your pages are generally crawled the same day they’re published, “you don’t need to read this guide.” It starts to bite around 1M+ pages, or 10k+ pages that change rapidly. Below that, spend your energy elsewhere. Evidence for this claim Google says crawl-budget guidance is mainly relevant to very large sites, including sites with over one million unique pages or over ten thousand rapidly changing pages. Scope: Google Search guidance; the page-count examples are diagnostic starting points, not hard eligibility thresholds. Confidence: high · Verified: 2026-07-14 Google: Large site crawl budget guide Bing’s Fabrice Canel frames the same idea more bluntly: less is more — fewer URLs to crawl is better for SEO.

robots.txt: crawl control, not index control

The most important distinction in this whole file: robots.txt controls crawling, not indexing. Disallowing a URL stops bots from fetching it — it does not keep it out of the index. A disallowed page can still be indexed (URL only, no content) if other pages link to it, and worse, if you disallow a page you also stop Google from ever seeing a noindex tag on it.

So the rules are:

• Want a page gone from search? Allow crawling and add noindex. Never use robots.txt to deindex.
• Want bots to skip a low-value URL space (internal search, infinite faceted combinations) and don’t care about indexing? robots.txt disallow is correct.
• Managing AI crawlers? This is also where you allow or block GPTBot, ClaudeBot, PerplexityBot, CCBot and friends — a strategic decision, not a default.

Canonicalization: a weighted decision, not a command

Canonicalization is where a lot of advanced technical SEO lives, and it’s widely misunderstood. rel="canonical" is a hint, not a directive. Google weighs it against many other signals — redirects, internal links, sitemap inclusion, HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.', URL structureURL structure is how the parts of a web address — scheme, domain, path, query string, and fragment — are organized and formatted. It mostly affects crawling, usability, and how engines understand a page, not rankings directly. — when it picks the representative URL. My deep dive on canonicalization puts the count around 40 signals that feed canonical selection, which is why you sometimes see “Duplicate, Google chose different canonical than userA Google Search Console Page Indexing status: you declared a canonical for this URL, but Google overrode your choice, picked a different page as the canonical, and indexed that one instead.” in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance.: your tag was outvoted.

The practical implications:

• Don’t send conflicting signals. I spent years on enterprise sites (I ran technical SEO in-house at IBM), and in a talk I give called Enterprise SEO Chaos I show real pages that “redirected to one version, canonicaled to a second, and internally linked to a third.” Pick one URL and make every signal agree.
• Signal strength roughly ranks redirect > rel="canonical" > internal links > sitemap. A 301 is a much stronger statement than a canonical tagA rel=\"canonical\" annotation — in the HTML <head> or an HTTP Link header — that tells search engines which URL is the preferred version of duplicate or near-duplicate content..
• Duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. isn’t a penalty. Google’s Gary Illyes has said roughly 60% of the web is duplicate content, and Google treats some of it as normal — not a spam violation. The cost is split signals and wasted crawling, not a punishment. The fix is consolidation, not panic.

And a note on JavaScript: I once ran a test — injecting a rel="canonical" via JavaScript on a page that had none in the HTML — and Google honored it, even though it had publicly said it wouldn’t. After that surfaced, Google updated its JavaScript SEO documentation. The lesson isn’t “use JS canonicals”; it’s that this stuff is testable, and the docs aren’t always the last word.

The rendering decision

Rendering is the step most overviews skip, and it’s where JavaScript sites get into trouble. “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Evidence for this claim Google processes JavaScript pages in crawling, rendering, and indexing phases and uses a recent version of Chrome for rendering. Scope: Google Search JavaScript processing; rendering and indexing remain subject to technical and quality constraints. Confidence: high · Verified: 2026-07-14 Google: JavaScript SEO basics It’s a separate, stateless service, it can cache resources for weeks, and it can lag the initial fetch — so a JS-dependent change can take a while to be reflected.

JavaScript isn’t the enemy here. As I put it in my guide to JavaScript SEO, JavaScript is not bad for SEO, and it’s not evil — it’s just different from what many SEOs are used to. The real decision is how you render:

• Server-side rendering (SSR) — safest for SEO; the HTML arrives complete.
• Static generation (SSG/pre-rendering) — best of both worlds for content that doesn’t change per request.
• Client-side rendering (CSR) — highest risk; the content only exists after JS runs, so you’re betting on the render step.
• Dynamic rendering — Google calls it a workaround, not a recommendation; Bing is more favorable. Treat it as a bridge, not a destination.

Two traps to know cold. First, lazy-loading: Googlebot doesn’t scroll or click, so content that only loads on interaction can stay invisible — make sure it loads when it’s in the viewport. Second, links: Google can only follow a link that’s a real <a href> element. A routerLink or a click handler with no href is not a crawlable link. Verify the rendered output against the raw HTML with the URL Inspection tool whenever you suspect a gap.

Site architecture and internal linking

Internal links do three jobs at once: they help bots discover pages, they distribute PageRankPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems., and they pass topical context through anchor textAnchor text is the visible, clickable text of a hyperlink. It tells readers what they'll find on the other end and gives search engines context about the linked page.. John Mueller has called internal linkingLinks between pages on the same site. “super critical for SEO” and one of the biggest levers you have on your own site — and I agree. It’s one of the highest-ROI things you control directly.

A few systems-level points:

• Orphan pagesAn orphan page is a page on your site that no other page links to internally. Because crawlers discover pages by following links, an orphan page is effectively invisible to search engines unless it's in an XML sitemap or linked from an external site. — pages nothing links to — are the first thing to hunt for. If it’s not linked, it’s barely discoverable and gets almost no equity.
• Architecture is crawl-funnel management. Important pages belong close to the home page; deep, click-distant pages get crawled less and rank worse.
• PageRank sculpting with nofollow is dead (since 2009). Nofollowing internal links makes that equity evaporate rather than redistribute. Manage flow with real architecture, not nofollowrel=\"nofollow\" is a value of the HTML link rel attribute that tells search engines you don't vouch for a linked page and don't want to pass ranking signals to it. Since 2019–2020 Google treats it as a hint, not a directive — and it does not reliably block crawling or indexing. tricks.

Core Web Vitals: three problems, not one

The biggest practitioner error with page experience is treating it as a single “make the site faster” problem. Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. are three distinct problems with different root causes and different fixes:

• LCPLargest Contentful Paint — render time of the largest visible image or text block, relative to when the page started loading. ≤2.5 s (at the 75th percentile) is good. (Largest Contentful Paint) — loading. Driven by server response time, render-blocking resourcesRender-blocking resources are CSS and synchronous JavaScript files a browser must download and process before it can paint any visible content. They sit on the critical rendering path and delay First Contentful Paint and Largest Contentful Paint., and how fast the main content asset loads. Target under 2.5 seconds.
• INPInteraction to Next Paint — the input-to-paint latency at the 75th percentile of a page's interactions. ≤200 ms is good. (Interaction to Next Paint) — interactivity. Driven by JavaScript execution blocking the main thread. Target under 200 milliseconds. (INP replaced FID in 2024 — if you still see FID anywhere, the advice is stale.)
• CLSCumulative Layout Shift — a unitless score for unexpected visual movement, taken from the largest burst (session window) of layout shifts, not the lifetime sum. ≤0.1 is good. (Cumulative Layout Shift) — visual stability. Driven by images without dimensions, late-loading fonts, and injected content. Target under 0.1.

Two things matter beyond the definitions. Field dataPerformance metrics captured from real users, not lab tests., not lab data: Google ranks on real-user CrUXChrome User Experience Report — Google's public dataset of real-world (field) performance data from eligible Chrome users. It's the official field-data source behind the Core Web Vitals program. data, not your LighthouseLighthouse is Google's free, open-source tool that audits a page under simulated lab conditions and scores it 0–100 across Performance, Accessibility, Best Practices, and SEO. It's lab data — useful for debugging, not a ranking signal. score, so a Lighthouse 65 with good field data beats a Lighthouse 100 with bad field data. And proportion: I’ll be honest — I don’t think Core Web VitalsWeb Vitals is Google's initiative (launched May 2020) for unified page-experience quality signals. Core Web Vitals — LCP, INP, and CLS — are the subset used in ranking; the rest (TTFB, FCP, TBT, Speed Index) are diagnostic, not ranking factors. have much impact on SEO, and unless a site is extremely slow I generally won’t prioritize fixing them for rankings. Do the work for users and conversions; just don’t oversell it as a ranking lever.

Structured data: signals for search and AI

Structured dataStructured data is a standardized way of labeling page content (using the schema.org vocabulary in JSON-LD, Microdata, or RDFa) so search engines can understand its meaning. It's not a direct ranking factor — its value is rich results and entity understanding. (use JSON-LDJSON-LD (JavaScript Object Notation for Linked Data) is a script-based structured data format, typically paired with the schema.org vocabulary to describe page content for search engines and AI systems. Google recommends it over Microdata and RDFa because it's the easiest format to implement and maintain at scale — but all three work, and structured data isn't a ranking signal.) doesn’t rank you, but it makes pages eligible for rich resultsRich results (formerly 'rich snippets') are enhanced search listings — stars, images, prices, breadcrumbs, video thumbnails, and more — that Google and Bing build from structured data. They're a display feature, not a ranking factor, and eligibility never guarantees they'll show. and increasingly helps AI systems parse your content for citation. It’s genuinely useful — and genuinely oversold as a ranking signal. My honest framing: most of SEO is doing the basics well, and content and links move the needle more than schema does. Implement it where it unlocks a rich result or clarifies an entity; don’t expect it to lift rankings on its own. (And note: schema markupSchema markup is code that uses the schema.org vocabulary to label what your content means so search engines can understand it and show rich results. It's most often written in JSON-LD, and it's not a direct ranking factor. URLs are not crawlable internal links — Mueller has confirmed this.)

International, briefly

If you serve multiple languages or regions, use distinct URLs per version and hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. annotations to map them, and prefer ccTLDsCountry-code top-level domain like .de or .co.uk — a strong geotargeting signal. or subdirectories over URL parameters. Don’t auto-redirect by IP — Google explicitly warns against it and it breaks crawling. International SEOInternational SEO is the practice of optimizing a site so search engines understand which countries and/or languages it targets, and serve the right version to each user. It spans URL structure, hreflang, and on-page localization. is deep enough to be its own pillar; this is just the technical handshake.

Technical SEO is an ongoing system, not a one-time audit

The framing every competitor guide gets wrong: technical SEO is not a checklist you complete once. Sites change constantly — deploys break canonical tags, a release slips a noindex into a template, a new ad script tanks INP, redirect chains accumulate. The mature practice is monitoring and regression detection:

• Watch GSC Page Indexing for sudden changes in indexed counts and excluded statuses.
• Watch Crawl Stats and your logs for response-code spikes and crawl-pattern shifts.
• Re-validate crawl, render, and redirects after every significant deployment.

On log files specifically: I used to treat them as a once-every-few-years troubleshooting tool. That’s changed. Logs are now the clearest place to see which AI crawlers are actually hitting you and how often — something no other tool shows you as directly — so for anyone who cares about AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity., they’ve become a lot more useful than they were.

Site migrations: the highest-stakes event

A migration — new domain, HTTP to HTTPS, a replatform, a URL restructure — is the single highest-stakes technical event, because it touches every URL at once. Map old to new 1:1, use 301/308 permanent redirects, keep them in place indefinitely (I would not rush to remove them — a couple of redirect hops is nothing to worry about), and use the GSC Change of Address toolA setting in Google Search Console that tells Google you've moved your whole site to a new domain or subdomain. It's a supporting signal for a domain migration — the 301 redirects do the real work of transferring rankings. where it applies. Migrations can be complex and involve a lot of people, but don’t panic — you can fix almost anything that goes wrong. There’s a full Site MigrationsA site migration is any significant change to a website's URL structure, domain, platform, protocol, or hosting that can affect how search engines crawl, index, and rank it. The risk scales with how much you change at once. cluster under this pillar.

Technical SEO for AI search

The modern shift, and it cuts against the lazy “technical SEO is dead” take: from 2025 on, AI search systems decide eligibility before they ever rank or cite. To be quoted in an AI answer your page generally has to be cleanly canonicalized, fast enough, renderable without heroics, and structured enough to be parsed with confidence. Messy signals don’t just lower a ranking now — they can remove you from the answer entirely. Because Bing’s index feeds many LLM answers, Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. matter more than Bing’s search share suggests. Technical hygiene matters more in the AI era, not less.

Where the leverage actually is

If you take one thing from this guide, make it prioritization. Spend your time on indexing, canonicalization, internal links, and clean migrations — the work that decides whether pages exist in search and consolidate their equity. Don’t lose sleep over crawl budget, Core Web Vitals, duplicate content, or short redirect chains unless you have a specific, diagnosed problem. And don’t chase perfection — I doubt there’s a major website that’s technically perfect, and if there were, I’d worry they were wasting resources on things that don’t matter instead of things that do.

This hub maps the rest of the pillar: How Search Works, Site Migrations, On-Page, Search Engine ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.', and JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript.. Start wherever your site is breaking — the pipeline tells you which gate to look at first.

AI summary

A condensed take on the Advanced Guide:

• No separate algorithm. Same crawl → render → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline on every site. Technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. decides whether pages can enter the system and be understood — it’s a foundation, not a ranking trick. The leverage is mostly negative: not losing what content and links earned.
• Treat the pipeline as gates. “Not all pages make it through each stage.” Diagnose which gate (crawl, render, index, serve) a page failed before changing anything.
• The one-liner: you can’t rank a page Google won’t index — so the boring structural work (canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them.) pays best and compounds at scale.
• Crawl budgetThe number of URLs an engine will crawl in a timeframe.: capacity + demand. Most sites never need to manage it (matters ~1M+ pages, or 10k+ rapidly changing). Bing: less is more.
• robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. controls crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., not indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. To remove a page: allow crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. + noindex. It’s also where you allow/block AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls..
• CanonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. is a weighted decision across ~40 signals; rel=canonical is a hint, not a command. Don’t send conflicting signals; duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. isn’t a penalty.
• RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is separate and can lag. JS isn’t evil, just different. Choose SSR / static / CSR / dynamic deliberately; watch the lazy-loading and <a href> link traps.
• Internal linkingLinks between pages on the same site. is “super critical” (Mueller). Hunt orphans; architecture is crawl-funnel management; nofollow sculpting is dead.
• Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. = three problems (LCPLargest Contentful Paint — render time of the largest visible image or text block, relative to when the page started loading. ≤2.5 s (at the 75th percentile) is good./INPInteraction to Next Paint — the input-to-paint latency at the 75th percentile of a page's interactions. ≤200 ms is good./CLSCumulative Layout Shift — a unitless score for unexpected visual movement, taken from the largest burst (session window) of layout shifts, not the lifetime sum. ≤0.1 is good.), judged on field dataPerformance metrics captured from real users, not lab tests., not LighthouseLighthouse is Google's free, open-source tool that audits a page under simulated lab conditions and scores it 0–100 across Performance, Accessibility, Best Practices, and SEO. It's lab data — useful for debugging, not a ranking signal. — and a minor ranking lever (Patrick doesn’t prioritize them for rankings).
• Monitoring, not a one-time audit. Watch Page Indexing, Crawl Stats, and logs; re-validate after deploys. Logs are newly useful for spotting AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls..
• Migrations are the highest-stakes event — map 1:1, 301, keep redirects.
• AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. gates eligibility on clean technical signals before ranking/citing; Bing’s index feeds LLMs, so Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. + IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. matter more than its share.

Official documentation

The primary-source docs that anchor the whole technical-SEO pillar.

Google

• In-Depth Guide to How Google Search Works — the crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline, URL discoveryURL discovery is how search engines find URLs to crawl — by pull (following links and reading sitemaps) and by push (you notify them via IndexNow, the Indexing API, or WebSub). It's the find step that comes before a page is ever fetched., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and serving. The single most important page in technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own..
• SEO Starter Guide — Google’s own beginner orientation and the Search Essentials baseline.
• Crawling and Indexing — the hub for robots.txt, sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing., canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., and crawl controls.
• Optimize your crawl budget — crawl capacityThe number of URLs an engine will crawl in a timeframe. + demand, and who actually needs to care.
• JavaScript SEO basics — renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. as part of indexing and how to keep JS content indexable.
• Core Web Vitals & Page Experience — what the page-experience signals are and how they’re used.
• Site moves with URL changes — Google’s migration playbook: redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., Change of AddressA setting in Google Search Console that tells Google you've moved your whole site to a new domain or subdomain. It's a supporting signal for a domain migration — the 301 redirects do the real work of transferring rankings., and what to monitor.
• Inside Googlebot (March 2026) — current crawl economics and byte limits.

Bing / Microsoft

• How Bing delivers search results — Bing’s crawl → index → rank pipeline and its named ranking factors.
• bingbot Series: Maximizing Crawl Efficiency — Bing’s definition of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. and its “crawl efficiency north star.”
• IndexNow / indexnow.org — the push protocol for instantly signaling changed URLs; pairs with sitemapsA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing..

Quotes from the source

On-the-record statements from Google and Bing that anchor the technical-SEO pillar. Each link is a deep link that jumps to the quoted passage on the source page.

Google — the pipeline

• “Google Search works in three stages, and not all pages make it through each stage.” — Google Search Central docs. Jump to quote
• “GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. uses an algorithmic process to determine which sites to crawl, how often, and how many pages to fetch from each site.” Jump to quote
• “During the crawl, Google renders the page and runs any JavaScript it finds using a recent version of Chrome.” Jump to quote
• “IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. isn’t guaranteed; not every page that Google processes will be indexed.” Jump to quote

Google — crawl budgetThe number of URLs an engine will crawl in a timeframe. and robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere.

• “Taking crawl capacity and crawl demandCrawl demand is the 'want' side of crawl budget — how much a search engine wants to crawl a site or URL, driven by popularity, staleness, and perceived inventory (plus temporary spikes from site moves). It's distinct from crawl rate/capacity, the 'can' side. together, Google defines a site’s crawl budgetThe number of URLs an engine will crawl in a timeframe. as the set of URLs that Google can and wants to crawl.” Jump to quote
• “If your site doesn’t have a large number of pages that change rapidly, or if your pages seem to be crawled the same day that they are published, you don’t need to read this guide.” Jump to quote
• “A robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. file tells search engine crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. which URLs the crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. can access on your site.” Jump to quote

Bing — Fabrice Canel, Microsoft

• “CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. is the process by which bingbotBingbot is Microsoft Bing's primary web crawler — the bot that discovers, fetches, and renders pages to build the Bing index. That index also powers Yahoo, DuckDuckGo, Ecosia, and Microsoft Copilot, so Bingbot's reach is far wider than Bing's own search-market share. discovers new and updated documents or content to be added to Bing’s searchable index.” Jump to quote
• “Less is more for SEO. Never forget that. Less URLs to crawl, better for SEO.” Read the interview

Google reps, on the record (reproduced through industry coverage)

• John Mueller: “Technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. is not going away, it continues to be the foundation of everything built on the open web.” Coverage (Search Engine Journal)
• Gary Illyes, on crawl budget: “the vast majority of the people don’t have to care about it.” Coverage (Search Engine Journal)
• John Mueller, on internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them.: “internal linkingLinks between pages on the same site. is super critical for SEO… one of the biggest things that you can do on a website.” Coverage (Search Engine Journal)
• John Mueller, on Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data.: “it’s more than a tie-breaker, but it also doesn’t replace relevance.” Coverage (Search Engine Journal)
• Gary Illyes, on Core Web VitalsWeb Vitals is Google's initiative (launched May 2020) for unified page-experience quality signals. Core Web Vitals — LCP, INP, and CLS — are the subset used in ranking; the rest (TTFB, FCP, TBT, Speed Index) are diagnostic, not ranking factors. priority: “If you don’t have anything better to do on your site, go do Core Web Vitals.” (Pubcon AMA) Coverage (Search Engine Land)
• Martin Splitt, on renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM.: “two-wave indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.… plays less and less of a role.” Coverage (Onely)

Patrick Stox (my own work)

• “This is my understanding of systems and is based on a lot of public statements from Google and my own knowledge. Warning: It’s not going to be 100% complete or accurate.” — slide 3 of my How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. deck. See the slide

The mental models

1. There is no technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. algorithm. Same crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline, same ranking systems, on every site. Technical SEO decides whether your pages can enter the system and be understood — it doesn’t hand you ranking points. Before you hunt for a “technical trick,” ask which ordinary stage is failing.

2. Foundation, not factor. The leverage of technical SEO is mostly negative: it stops you from losing what content and links earned. You can’t rank a page Google won’t index — so most of the job is clearing obstacles, not adding signals.

3. Four gates, not a conveyor belt. Crawl → render → index → serve. Each is a filter, and “not all pages make it through each stage.” When a page underperforms, locate which gate it failed before you change anything.

4. The four “not equals.”

• CrawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. ≠ indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. (a blocked page can still be indexed; indexing isn’t guaranteed even when crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. succeeds).
• Crawling ≠ ranking (crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor. is not a ranking signal).
• Crawling ≠ renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. (JavaScript runs in a separate step that can lag).
• Indexing ≠ ranking (being in the index doesn’t win you queries).

5. The leverage rule at scale. On big sites everything is template-level, so impact multiplies. One error can keep millions of pages out of the index; one canonical fix can recover a fortune. The boring structural work outranks the shiny stuff.

6. Prioritization is the real skill. The hardest and most valuable thing in technical SEO is knowing what to ignore. Fix indexing, canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., and migrations; don’t sweat crawl budgetThe number of URLs an engine will crawl in a timeframe., Core Web VitalsGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data., or duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. without a diagnosed problem.

7. AI raises the bar, not lowers it. AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. decides eligibility (clean canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., schema, fast renderable pages) before it ranks or cites. Messy technical signals can remove you from the answer entirely.

Technical SEO foundations checklist

A first pass to confirm search engines can find, read, and understand your site:

• Crawlable. Important pages are linked from somewhere crawlable (no orphans); robots.txt doesn’t block anything you want indexedStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• Discoverable. An XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. is submitted in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility., listing only canonical, indexable URLs with accurate lastmod.
• Renderable. JS-dependent content is reachable via real <a href> links, not click-only navigation; critical content doesn’t depend on a slow render.
• Indexable. No stray noindex on pages you want foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore.; check the GSC Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. report for excluded statuses.
• Canonical. One canonical URLHow search engines pick one canonical URL among duplicates and consolidate signals onto it. per piece of content; duplicates point to it; no canonical chains or canonicals pointing at the homepage; signals agree (redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., canonical, internal linksAn internal link is a hyperlink from one page on a website to another page on the same website. Internal links help search engines discover your pages and pass ranking signals (PageRank and anchor-text context) between them., sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. all point the same way).
• Healthy server. Fast, stable responses — minimal 5xx/timeouts (bots slow down when your server struggles).
• Clean redirects. No long redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency. or loops; permanent moves use 301/308; gone pages return 404/410.
• No URL waste. Parameters, faceted nav, and session IDs aren’t generating infinite or duplicate URL spaces (spider trapsA spider trap (also called a crawler trap) is a site structure that generates an effectively infinite number of URLs — from faceted filters, calendars, session IDs, or redirect loops — so crawlers waste their budget on low-value, near-duplicate pages instead of your real content.).
• On-page meta. Unique, accurate title tagsThe title tag is the HTML title element in a page's head that specifies the document's title. It's the primary source for the SERP title link and a confirmed light ranking factor — but since August 2021 Google doesn't always show it verbatim. and meta descriptionsThe meta description is an HTML head tag — `<meta name=\"description\" content=\"…\">` — that suggests a short summary of the page for the search snippet. It's not a Google ranking factor, and Google rewrites it the majority of the time, but a good one can still lift click-through.; correct meta robots directives.
• Migration-safe. If you’re moving anything (domain, HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.', platform, URLs), map old→new 1:1 redirects and use the GSC Change of Address toolA setting in Google Search Console that tells Google you've moved your whole site to a new domain or subdomain. It's a supporting signal for a domain migration — the 301 redirects do the real work of transferring rankings. where it applies.
• AI-eligible. Valid schema, consistent entity signals, AI-crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. access decided deliberately, and fast renderable pages so AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. can parse and cite you.

How much should you actually care?

The most useful thing I can give a technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. isn’t a checklist — it’s a sense of proportion. A lot of “best practices” get far more attention than they earn. Here’s my honest ranking of where the leverage is, and where it isn’t.

The pattern: spend your time on indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., consolidation, and clean migrations; don’t lose sleep over crawl budget, Core Web VitalsWeb Vitals is Google's initiative (launched May 2020) for unified page-experience quality signals. Core Web Vitals — LCP, INP, and CLS — are the subset used in ranking; the rest (TTFB, FCP, TBT, Speed Index) are diagnostic, not ranking factors., or duplicate content unless you have a specific, diagnosed problem. And don’t chase technical perfection — I doubt there’s a major website that’s technically perfect, and if there were, I’d worry they were wasting resources on things that don’t matter instead of things that do.

Which control does what

The other thing people mix up constantly — these are four separate tools for four separate jobs:

The classic mistake is using robots.txt to deindex — if you block crawling, Google can’t see the noindex tag and may keep the page in results via links from other sites. Want a page gone? Allow crawling and add noindex.

Tools for technical SEO

• Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — your ground truth for how Google treats the site: the Page Indexing reportThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason., Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root)., the URL Inspection toolA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. (crawl/render/indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. status for a single URL), and the Change of Address toolA setting in Google Search Console that tells Google you've moved your whole site to a new domain or subdomain. It's a supporting signal for a domain migration — the 301 redirects do the real work of transferring rankings. for migrations.
• Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility. — crawl info, Crawl Control, Site Scan, and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — and it matters more than its share suggests because Bing’s index feeds many LLM answers.
• Site crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. / audits — Ahrefs Site Audit and Screaming Frog SEO Spider simulate a crawl and surface redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency., duplicate URLs, blocked pages, broken canonicals, and trap-like patterns at scale.
• Server log file analysisLog file analysis is reading a web server's raw access logs to see exactly which URLs search engine crawlers actually requested, when, how often, and what status code they got. Unlike crawl tools or Search Console, logs are the unsampled, ground-truth record of what really happened. — the only place you see exactly what bots crawled and where they wasted budget — and increasingly the clearest way to see which AI crawlers (GPTBot, ClaudeBot, PerplexityBot) are hitting you. Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform.
• Ahrefs Webmaster Tools — free crawl + audit for sites you verify.
• PageSpeed InsightsPageSpeed Insights (PSI) is a free Google tool at pagespeed.web.dev that reports two kinds of data for a URL: real-user field data from the Chrome UX Report and a single Lighthouse lab run with the 0–100 Performance score. Only the field Core Web Vitals are what Google uses for ranking. / LighthouseLighthouse is Google's free, open-source tool that audits a page under simulated lab conditions and scores it 0–100 across Performance, Accessibility, Best Practices, and SEO. It's lab data — useful for debugging, not a ranking signal. — page-experience and renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. checks; pair with field (CrUXChrome User Experience Report — Google's public dataset of real-world (field) performance data from eligible Chrome users. It's the official field-data source behind the Core Web Vitals program.) data for how real users and the renderer experience the page.
• IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — push changed URLs to Bing (and others) in real time instead of waiting to be re-crawled.

Durable technical SEO metrics

Intended index coverage trend

• What it measures: Whether canonical, indexable URLs you intend search engines to serve are actually represented in the indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. over time.
• How to pull it: Compare the XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. or approved indexable URL inventory with Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason. data, then inspect excluded reasons by template.
• Cadence: Review after material releases and on a recurring schedule suited to the site’s publishing and crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor..
• Healthy direction: Intended indexed URLs remain stable or grow with approved content while unexplained exclusions and duplicate canonical conflicts decline.
• Decision triggered: Investigate template-level crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., canonical, renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., duplication, or noindex problems before investing in more content for the same affected area.

Core Web Vitals field pass rate by template

• What it measures: The share of real-user page groups passing field Core Web Vitals, separated by template and device rather than averaged across the site.
• How to pull it: Use Search Console Core Web VitalsThe Google Search Console report (under Experience) that shows how your indexed URLs perform on the Core Web Vitals — LCP, INP, and CLS — using real-user field data from CrUX, grouped by device, status, and clusters of similar-performing URLs. and CrUXChrome User Experience Report — Google's public dataset of real-world (field) performance data from eligible Chrome users. It's the official field-data source behind the Core Web Vitals program./PageSpeed InsightsPageSpeed Insights (PSI) is a free Google tool at pagespeed.web.dev that reports two kinds of data for a URL: real-user field data from the Chrome UX Report and a single Lighthouse lab run with the 0–100 Performance score. Only the field Core Web Vitals are what Google uses for ranking. field dataPerformance metrics captured from real users, not lab tests.; map affected URL groups to the responsible template or component.
• Cadence: Track through the field-data window and compare before and after major performance releases.
• Healthy direction: More important templates move into passing groups without regressions shifting to another device, region, or page type.
• Decision triggered: Prioritize a shared template or component fix when a poor group affects meaningful traffic or user experience; do not chase tiny lab-only changes as a ranking trick.

Crawl reliability and response mix

• What it measures: Whether search-engine crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. reaches useful URLs reliably instead of spending capacity on errors, chains, traps, or low-value URL spaces.
• How to pull it: Combine Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. Crawl Stats with server logs and crawlerA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. reports, segmented by status, host, directory/template, and bot.
• Cadence: Monitor continuously for incidents and review trends after platform, CDN, redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., faceting, or migration changes.
• Healthy direction: Stable successful responses for important URLs, fewer 5xx failures and redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency., and less repeated crawling of known trap spaces relative to the site’s own baseline.
• Decision triggered: Fix availability or routing first; then adjust internal discovery, parameter handling, or crawl controls for persistent waste.

Resources worth your time

My technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. writing (Ahrefs)

• The Beginner’s Guide to Technical SEO — the full framework: find, crawl, understand, and indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..
• Enterprise Technical SEO — what technical SEO looks like at scale (and why perfection is the wrong goal).
• Crawl Budget: Everything You Need to Know — when it matters, and the (many) cases where it doesn’t.
• JavaScript SEO: Issues & Best Practices — “JavaScript is not bad for SEO… it’s just different.”
• Canonicalization: A Beginner’s Guide — the ~40 signals Google weighs, and why most duplicates aren’t nefarious.
• Redirects for SEO — the 11 types and how long to keep them (longer than you think).
• Website Migration: A Complete Guide — “you can fix almost anything that goes wrong.”
• Core Web Vitals & PageSpeed — why I don’t prioritize CWVGoogle's three real-user UX metrics — LCP (loading), INP (responsiveness), and CLS (visual stability) — used by Google's ranking systems, with no official weight attached, measured on field data. for rankings.

My speaking

• How Search Works (SlideShare) — my full walkthrough of crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and ranking.
• Enterprise SEO Chaos (SMX) — the IBM-scale stories: 24 URL variations, 14-hop redirect chainsA → B → C instead of A → C. Each hop loses link equity and adds latency., and signals all pointing different directions.

From around the industry

• web.dev — Core Web Vitals — Google’s web-platform docs on LCPLargest Contentful Paint — render time of the largest visible image or text block, relative to when the page started loading. ≤2.5 s (at the 75th percentile) is good., INPInteraction to Next Paint — the input-to-paint latency at the 75th percentile of a page's interactions. ≤200 ms is good., and CLSCumulative Layout Shift — a unitless score for unexpected visual movement, taken from the largest burst (session window) of layout shifts, not the lifetime sum. ≤0.1 is good.: what each measures and how to fix it, separate from search.
• Onely blog — a technical-SEO agency known for deep renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. and indexing research (their two waves of indexing write-up is a good example).
• Search Engine Roundtable — Barry Schwartz’s near-daily log of what Google and Bing reps actually say; the fastest way to track statement changes.
• Search Engine Journal — Technical SEO — ongoing coverage and explainers, and the source for several rep statements quoted above.
• Search Engine Land — SEO — industry news and conference coverage (Pubcon/SMX AMAs with Google’s team).
• Google’s Crawling December series — the best concentrated set of official crawl explainers.
• r/TechSEO — the community for crawl/index/render debugging.

Podcasts

• Search Off the Record (Google Search Relations) — Gary Illyes and Martin Splitt on how GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. crawls, renders, and indexes. The closest thing to official commentary on the pipeline. Listen
• Voices of Search — Prioritizing SEO Efforts, my conversation on why prioritization is the hardest part of the job and how I triage high-impact / low-effort work. Listen
• TheeDigital — Debunking SEO Myths — me on the technical myths that won’t die (duplicate-content penalties, keyword density, subdomains vs. subfolders). Listen

Videos

• Google Search Central (YouTube) — the How Google Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank. series and Martin Splitt’s crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor./renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. and JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript. explainers. Channel

Stats worth citing

A mix of my own research and third-party figures (Google, Microsoft, and Cloudflare):

• Rich resultsRich results (formerly 'rich snippets') are enhanced search listings — stars, images, prices, breadcrumbs, video thumbnails, and more — that Google and Bing build from structured data. They're a display feature, not a ranking factor, and eligibility never guarantees they'll show. can lift CTR. Google’s own published case studies report a 25% higher CTR for Rotten Tomatoes’ marked-up pages, a 35% increase in visits for Food Network, and an 82% higher CTR on Nestlé’s rich-result pages vs. non-rich. Google — structured data intro
• RenderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. costs ~20× more than crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor.. From my JavaScript SEOMaking sure search engines can crawl, render, and index content that depends on JavaScript. talks: Ahrefs crawled ~7B pages a day but rendered ~80M JavaScript pages using around 600 servers — a useful sense of why renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM. is rationed and can lag. Deck
• Roughly 60% of the web is duplicate contentThe same or very similar primary content reachable at more than one URL. There's no general duplicate content penalty — the real costs are possible signal dilution, the wrong URL getting chosen, and less-efficient crawling. — Gary Illyes (Google) — which is exactly why some duplication is “normal” and canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., not panic, is the right response. Google — consolidate duplicate URLs
• Bing discovers tens of billions of new URLs every day — Fabrice Canel (Microsoft) — the scale of the discovery-and-filtering problem behind “less is more.” Coverage (Search Engine Roundtable)
• AI botsAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are a clear #2 and closing on search-engine botsA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — from Cloudflare Radar crawl data (my analysis of it): search bots still crawl the most, but AI crawlersAI crawlers are bots from AI companies that fetch web pages to train language models, build AI-search indexes, or answer live user questions. They come in three categories, each with its own user-agent tokens and its own robots.txt controls. are on pace to overtake them within a couple of years. Source
• 95.2% of sites have 3XX redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.; 72.9% are missing meta descriptionsThe meta description is an HTML head tag — `<meta name=\"description\" content=\"…\">` — that suggests a short summary of the page for the search snippet. It's not a Google ranking factor, and Google rewrites it the majority of the time, but a good one can still lift click-through. (my study) — across 1M+ domains in Ahrefs Site Audit. But don’t over-react to the meta-description number: Google rewrites them about 62.78% of the time, and they aren’t a ranking factor. Source

Test yourself: Technical SEO

Five quick questions on what technical SEOTechnical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own. is and the crawl → indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed. → serve pipeline. Pick an answer for each, then check.

Technical SEO

Technical SEO is the practice of making a site easy for search engines to crawl, render, index, and (now) be eligible for AI answers. It's the foundation that lets your content and links rank — not a ranking trick of its own.

Related: Crawling, Indexing, Rendering, Site Migration

Technical SEO

Technical SEO is the work of making a website easy for search engines to crawl, render, indexStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., and serve. It’s the plumbing layer of SEO: the part that determines whether your pages can be foundA 302 (\"Found\") is a temporary redirect: it forwards users to a new URL while telling search engines the original URL should stay in the index. It's a weak canonicalization signal, not the zero-equity dead end of SEO folklore. and understood at all, before anything about content quality or links even comes into play. There’s no separate “technical SEO algorithm” — search engines run the same crawl → render → index → serve pipeline on every site. Technical SEO is just the discipline of keeping that pipeline clear.

The reason it matters is simple: you can’t rank a page Google won’t index. If crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor., renderingTurning HTML, CSS, and JavaScript into the final visual page and DOM., indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., or canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it. are broken, even excellent content stays invisible. That’s why technical SEO is best thought of as a foundation rather than a tactic — it doesn’t make a page rank by itself, but its absence can stop everything else from working.

The same pipeline applies whether you have fifty pages or fifty million, but scale changes the stakes. On a large site, fixes happen at the template level, so one error can keep millions of pages out of the index and one fix can recover a fortune in link equityPageRank is Google's original recursive link-graph algorithm: a page's score depends on the scores of the pages linking to it, and in the published model each page's score is split across its outbound links (the simplified version: links are weighted votes). Google says it's evolved since launch but still part of its core ranking systems.. And from 2025 on, AI searchAI search uses large language models and retrieval-augmented generation (RAG) to synthesize an answer from multiple sources rather than returning a ranked list of links. Examples include Google AI Overviews, ChatGPT Search, and Perplexity. systems decide eligibility — clean canonicalizationHow search engines pick one canonical URL among duplicates and consolidate signals onto it., valid schema, fast and renderable pages — before they ever rank or cite you. That makes technical hygiene more important, not less.

Related: Crawling, Indexing, Rendering, Site Migration

Changelog