Sitemaps

What a sitemap is, whether you need one, the types (XML, HTML, image, video, news), how to find, create, and submit one, and why sitemaps help discovery but never guarantee indexing.

First published: Jun 22, 2026 · Last updated: Jul 15, 2026 · Advanced
demand #2 in Discovery#11 in How Search Works#71 in Technical SEO#95 on the site
AIAuthored byAI v2
1 evidence signal on this page

A sitemap is a file that lists the URLs (and media) on your site so search engines can discover them. It helps discovery and is a great coverage/diagnostic tool — but submitting one never guarantees crawling or indexing. A single sitemap caps at 50,000 URLs or 50MB uncompressed; Google ignores <priority>/<changefreq> but reads an accurate <lastmod>. Automate it from the pages you actually have, list only canonical indexable URLs, and submit it in Search Console + Bing Webmaster Tools (the old ping endpoint is dead). This hub routes you to the deep dives: XML sitemap, sitemap index, image sitemap, and video sitemap.

TL;DR — A sitemapA sitemap is a file that lists the pages, images, videos, and other files on your site so search engines can discover them. It helps discovery, but submitting a sitemap doesn't guarantee crawling or indexing. is a file listing your URLs (and optionally images/videos) so engines can discover them — a coverage and diagnostic tool, not a discovery silver bullet, because submitting never guarantees crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. or indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. A single sitemap caps at 50,000 URLs or 50MB uncompressed (gzipCompression (HTTP content encoding) shrinks text-based responses — HTML, CSS, JS, JSON, SVG, XML sitemaps — before they're sent over the network, using an algorithm like Gzip, Brotli, or Zstd, so the browser or crawler downloads fewer bytes. It's not a ranking factor, but it speeds up page loads and helps pages stay under crawler fetch limits. is allowed; the cap is the uncompressed size); UTF-8; fully-qualified absolute URLs; entity-escape &, ', ", <, >. Google ignores <priority> and <changefreq> and only uses <lastmod> if it’s consistently accurate. Submit via Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results., Bing Webmaster ToolsThe free, first-party consoles search engines give site owners — Google Search Console and Bing Webmaster Tools — to see how their pages are crawled, indexed, and ranked, and to fix problems. The legacy name is 'webmaster tools.', and the robots.txt Sitemap: line — the old ping endpoint has 404’d since Jan 2024, and IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. notifies Bing and others (not Google). Build it automatically, list only canonical indexable URLs, and use the submitted-vs-indexed signal in Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. to find what’s missing.

What a sitemap is — and what it’s for

Google’s definition is the right anchor: “A sitemap is a file where you provide information about the pages, videos, and other files on your site, and the relationships between them.” Evidence for this claim A sitemap supplies search engines with information about pages, videos, and other files on a site and their relationships. Scope: Google's general definition of sitemaps; supported formats and extensions have additional requirements. Confidence: high · Verified: Google Search Central: Learn about sitemaps It’s an inventory you hand to search engines.

The most important framing I can give you: a sitemap is a coverage and diagnostic tool, not a discovery silver bullet. Google is explicit that “A sitemap helps search engines discover URLs on your site, but it doesn’t guarantee that all the items in your sitemap will be crawled and indexed.” Submitting a URL doesn’t index it. The real day-to-day payoff is the submitted-vs-indexed comparison in Search Console: split your sitemaps by section or content type and you can see which part of the site isn’t getting picked up. During a migration I’ll keep a sitemap of the old URLs around for a while on purpose, specifically so I can watch them drop out of the index in GSC.

Do you need one? Resolving the contradiction

You’ll see two pieces of advice that look like they conflict:

  • Google’s threshold: you likely don’t need a sitemap if the site is small (around 500 pages) and well internally linked; you likely do if it’s large, new with few backlinks, or media/News-heavy.
  • Mueller’s baseline: “Making a sitemap file automatically seems like a minimal baseline for any serious website, imo.”

Both are right, and here’s how I reconcile them: a sitemap isn’t strictly required for a tiny, well-linked site, but it’s cheap insurance and it’s now the expected baseline — so always generate one, automatically. Which brings me to the one rule I won’t bend on.

Automate it, or it rots

Sitemaps should be generated automatically from the pages you actually have. If someone hands me a manually-built sitemap, I already know it’ll fall out of date fast — somebody adds pages, nobody updates the file, and now it’s lying to Google. And there’s a subtler point: if you’re generating the sitemap from your real, crawlable pages anyway, search engines can usually reach those pages on their own through links. The sitemap earns its keep by being current and complete, not just by existing. Automation is what makes it current.

The types of sitemap

  • XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. — the workhorse, written for crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index.. A list of <loc> URLs, optionally with <lastmod>. This is what people usually mean by “sitemap.”
  • HTML sitemap — a human-facing page of links. Useful for users and for internal linking, but it isn’t the file you submit to Search Console.
  • Image sitemapAn image sitemap is a sitemap (or an extension added to an existing sitemap) that lists the images on your pages using Google's image namespace, helping Google discover images it might otherwise miss. — surfaces images search engines might otherwise miss (for example, images loaded by JavaScript or served dynamically). Can be a standalone file or image tags added to an existing sitemap.
  • Video sitemapA video sitemap is an XML sitemap (or mRSS feed) that uses the video:video extension to tell Google about videos hosted on your pages — the landing page, a thumbnail, a title, a description, and a link to the video file or player. It helps Google discover and surface video content, but it doesn't guarantee indexing. — gives search engines the metadata they need to understand and index your videos (thumbnail, title, description, and the media or player URL).
  • News sitemapA news sitemap is a sitemap (a standalone file or a section of an existing sitemap) that uses Google's news: XML namespace to tell Google about your recent articles — only those published in the last two days. It speeds discovery of breaking content; it isn't a ranking factor and isn't required for Google News eligibility. — for sites in Google News, listing recent articles.

And XML isn’t the only accepted format. Google also takes RSS/Atom feeds and a plain-text .txt file (one URL per line). RSS and text formats can only list page URLs — no image or video metadata — so XML is the most versatile.

XML vs HTML

These get confused constantly, so keep them separate: an XML sitemapAn XML sitemap is a UTF-8 file listing the canonical URLs on your site (with optional lastmod) so search engines can discover and prioritize them. It's a discovery and diagnostic aid, not a guarantee of indexing — and Google ignores its priority and changefreq tags. is for crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. — machine-readable, submitted to Search Console, the thing search engines parse. An HTML sitemap is for people — a regular web page linking to your sections, which can also help internal linkingLinks between pages on the same site.. If you only build one, build the XML sitemap; that’s the one search engines actually use.

How to find a sitemap (and why some sites hide theirs)

To find a sitemap, try, in order:

  • /sitemap.xml — the default location on most platforms.
  • robots.txt — a public sitemap is usually named here with a Sitemap: line, which is also how engines auto-discover it.
  • site: operators / Search Console — for your own property, the Search Console Sitemaps reportThe Google Search Console report where you submit sitemaps and watch how Google processes them — type, last read date, status, and how many URLs were discovered. It confirms Google read your list; it doesn't prove anything got indexed. is the authoritative view of what’s submitted and how it’s doing.

But here’s the part most guides skip: not every sitemap is meant to be found. Some sites deliberately omit the Sitemap: line from robots.txt and submit the file only through Search Console and Bing Webmaster ToolsMicrosoft's free portal for monitoring and improving how a site appears in Bing search — the peer to Google Search Console, plus IndexNow instant indexing, richer backlink data, and keyword volumes. Because Bing's index also feeds Microsoft Copilot, it doubles as a window into AI-search visibility.. A sitemap referenced in robots.txt is readable by anyone — including competitors who’d love a tidy list of every URL you publish (unlinked pages, new launches, strategically important pages) and a way to watch how fast you ship. A sitemap submitted only through the consoles is effectively private. So if you can’t find a site’s sitemap at the usual paths, it may just be unlisted, not absent.

The trade-off is real, though: hiding the sitemap from robots.txt also means any engine that relies on robots.txt auto-discovery won’t see it unless you’ve submitted it in each console. If you go the private route, you have to do the submission legwork everywhere you care about.

How to create one

Almost every platform can generate a sitemap for you — and on most of them it’s automatic. I’ve put the per-platform paths (WordPress, Shopify, Wix, Squarespace, Webflow, Drupal, JS frameworks, and manual generators) in the Cheat Sheets tab so you can jump straight to yours. The short version for JS frameworks: search the framework name plus “sitemap” (for example, “Gatsby sitemap” or “Next.js sitemap”) — there’s almost always an existing module so you don’t hand-roll it. This site runs on Astro and uses @astrojs/sitemap, configured in astro.config.mjs.

How to submit it

The live methods:

  • Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results. — the Sitemaps report (and the Search Console APIA set of REST APIs that let you programmatically read and manage Google Search Console data for properties you've verified — Performance data, URL index status, sitemaps, and properties — authorized with OAuth 2.0.). The primary method.
  • Bing Webmaster Tools — submit there too; Bing fetches it immediately, then rechecks roughly daily.
  • robots.txt Sitemap: line — works for any engine that reads robots.txt, and is how a public sitemap gets auto-discovered. Use the full absolute URL.

What’s dead: the old standalone ping endpoint (google.com/ping?sitemap=). It’s been deprecated since 2023 and returns a 404 since January 2024 — Google killed it because the vast majority of submissions were spam. If you’ve got a plugin or cron job still hitting it, strip that out. For fast, URL-level notification of single changes to Bing and others, use IndexNowIndexNow is an open push protocol that lets you instantly tell participating search engines (Bing, Yandex, Naver, Seznam, and Yep) which URLs you've added, changed, or removed via a simple HTTP request — and one submission is shared across all of them. Google does not use it. — but note Google does not participate in IndexNow, so it won’t speed up Google indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed.. Bing’s own framing is that the two are complementary: sitemaps for comprehensive coverage, IndexNow for fast per-URL pushes.

Submission problems

When something goes wrong, the Search Console Sitemaps report will tell you — but the error labels aren’t always self-explanatory. I’ve put a full error → meaning → fix table in the Cheat Sheets tab. The greatest hits: “Couldn’t fetch” (wrong URL, robots block, or just not processed yet — often transient), “Unsupported format” (you submitted an HTML page instead of a real XML/RSS/Atom/txt file), and “URL not allowed” (the classic HTTP-vs-HTTPSHTTPS is the encrypted version of HTTP — it uses TLS to authenticate the server and protect data in transit between a browser and a website. Google announced it as a lightweight ranking signal in 2014 and today conditionally prefers HTTPS pages as canonical; Chrome marks plain HTTP pages 'Not Secure.' / www mismatch, or URLs above the sitemap’s own path). A couple of non-error gotchas worth knowing: Google may serve you a stale cached copy, so changes aren’t instant, and submitting both the children and the sitemap indexA sitemap index is a sitemap of sitemaps — a single file that lists your other sitemap files instead of listing URLs directly. It's how large sites stay under the 50,000-URL / 50MB-per-sitemap limit while submitting just one file. is harmless but unnecessary.

TIP

Trace a 'URL not allowed' report back to the exact host or path mismatch with the XML Sitemap Validator Free

  1. Fetch the deployed sitemap or paste the XML that produced the submission error.
  2. Compare each URL’s host and path with the sitemap location and the verified property.
  3. Move disallowed URLs to the correct sitemap or complete the documented cross-site verification, then resubmit and monitor the live report.
Google generally ignores a cross-host URL unless the submission setup proves the required ownership. Keep ordinary sitemaps and their URLs on the same host.

The completed XML Sitemap Validator result warns that one URL uses a different host or scheme from the sitemap. It says Google generally ignores such URLs unless ownership is verified through Search Console or robots.txt cross-submission.

Best practices — and “exclude ≠ noindex”

The rules that matter, all from Google’s own spec:

  • Size: a single sitemap holds at most 50,000 URLs or 50MB uncompressed, whichever comes first. You can gzip it — the 50MB cap is the uncompressed size. Evidence for this claim Google limits a single sitemap to 50,000 URLs or 50 MB uncompressed. Scope: Google-supported sitemap files; larger inventories must be split across multiple sitemaps, optionally joined by an index. Confidence: high · Verified: Google Search Central: Build and submit a sitemap Over the limit, split into multiple sitemaps and tie them together with a sitemap index.
  • Encoding: UTF-8.
  • URLs: fully-qualified, absolute URLs. Entity-escape &, ', ", <, and >.
  • Tags Google ignores: <priority> and <changefreq> do nothing — don’t bother with them.
  • lastmod done honestly: Google only trusts lastmod if it’s consistently and verifiably accurate. Set it on significant updates (main content, structured data, or links) — not a blanket copyright-year or “today” stamp on every URL. Lie about it and Google stops believing the field. Done right, an accurate lastmod genuinely helps re-crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. (Bing leans on it even harder than Google does).
  • Contents: list only canonical, indexable, 200-status URLs. No redirectsA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't., no non-canonical URLs, no noindex’d pages. Any page you want indexed should be in the sitemap; nothing you don’t.

That last point hides the single most common conceptual error: exclude ≠ noindexNoindex is a directive that tells search engines to keep a page out of their index, so it won't appear in search results. It works only on pages a crawler can actually fetch — a page blocked in robots.txt can never be noindexed.. Leaving a URL out of your sitemap does not deindex it. The sitemap is an advertisement, not a gate — dropping a URL just stops you advertising it; it doesn’t remove it from Google. If you want a page gone, allow crawling and use noindex. The sitemap is the wrong tool for that job.

Where to go next

This page is the overview. Each of these is its own deep dive nested under the discovery topic:

  • XML sitemap — the format and anatomy: <urlset>, <loc>, <lastmod>, the ignored tags, what to include and exclude, and hreflangHreflang is an annotation (in HTML, HTTP headers, or XML sitemaps) that tells search engines which language and optional region a page targets, and which alternate versions exist. It only works when every page in the cluster references all the others. in sitemaps.
  • Sitemap index — the “sitemap of sitemaps” for large sites, when to split, and the math on how many URLs you can cover.
  • Image sitemap — surfacing images search engines might miss, the current (not deprecated) tag list, and cross-domain rules.
  • Video sitemap — the required tags, accepted file types, and getting videos understood and indexed.

Sitemaps are one half of discovery — the other half is crawling, which is how engines actually fetch the URLs your sitemap points to. For the whole picture, see How Search WorksSearch works in three stages — crawling, indexing, and serving (ranking). A page has to clear each one to appear in results: getting crawled doesn't mean you're indexed, and getting indexed doesn't mean you rank.. Every topic above is in the sidebar too.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.