Host Status (Crawl Stats)

What the Host status indicator in Google Search Console's Crawl Stats report means — the three availability checks, the three states, and why a red status is an emergency.

First published: Jun 23, 2026 · Last updated: Jul 18, 2026 · Advanced
demand #13 in Search Engine Tools#16 in Tools#267 in Technical SEO#360 on the site
AIAuthored byAI v2

Host status is the availability indicator at the top of GSC's Crawl Stats report — a historical record of what Google observed, not a live uptime check. It checks three things — robots.txt fetching, DNS resolution, and server connectivity — over the last 90 days. Green means no significant issues; yellow means a problem happened more than a week ago (it typically ages out of the report on its own); red means one happened in the last week. The highest-stakes failure is an unreachable robots.txt: Google stops crawling for ~12 hours, then falls back to the last good cached copy — if one exists — for up to ~30 days, after which behavior depends on your site's broader availability. It's the underlying robots.txt, DNS, or server failure that throttles crawling of the whole site, not the host-status label itself, and the state aging back to green isn't proof crawling, indexing, or rankings have fully recovered.

TL;DR — Host statusThe at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity. is the availability lens at the top of GSCA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.’s Crawl Stats report — a historical record of what Google observed, not a live uptime check — assessed over the last 90 days across three checks: robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. fetching, DNS resolution, and server connectivity. Green = no significant issues; yellow = an issue more than a week ago (typically ages out of the report on its own); red = an issue in the last week. The highest-stakes failure is an unreachable robots.txt: Google stops crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. for the first 12 hours, then falls back to the last good cached copy — if one exists — for up to ~30 days, after which behavior depends on your site’s broader availability. A 4xx robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere. is fine (Google crawls freely); a 5xx/timeout/DNS failure is what pauses crawlingCrawling is how search engines use automated bots (like Googlebot and Bingbot) to discover URLs and download pages. A page has to be crawlable to be indexed, but crawling on its own isn't a ranking factor. — that underlying failure, not the host-status label, is what throttles crawling. There’s no reset button, and the state returning to green isn’t proof crawling, indexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed., or rankings have fully recovered.

What host status actually measures

Search ConsoleGoogle's free tool for monitoring crawling, indexing, and search performance. documents host status within Crawl StatsA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). and evaluates availability over the previous 90 days. Evidence for this claim Search Console Host status summarizes robots.txt availability, DNS resolution, and server connectivity over the previous 90 days. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: Google: Crawl Stats report Diagnose individual URL outcomes separately in URL InspectionA Google Search Console feature that reports how Google sees one specific URL on a property you own. By default it shows the last-indexed snapshot; a separate \"Test live URL\" mode fetches the current version. or Page IndexingThe Google Search Console report (formerly Index Coverage) showing how many of your URLs are indexed vs. not indexed, and grouping the not-indexed ones by reason.. Evidence for this claim Host status is a host-level Crawl Stats diagnostic rather than a per-URL Page Indexing decision. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: Google: Crawl Stats report

Host status answers a narrow question: over the last 90 days, could GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. reach the host at all? That’s a different question from “is this page indexed” or “does this URL throw an error.” It’s site/host-level availability, not per-URL crawl errors — which is exactly why I keep it mentally separate from Page IndexingStoring a crawled page in the search index so it can appear in results. Crawled is not the same as indexed — Google selects what to keep, and indexing isn't guaranteed..

It’s also worth being precise about what kind of “availability” this is. Host status is a historical record of what Google observed during crawling, not a live uptime monitor — a green state right now doesn’t prove your site is reachable this second, and a red state doesn’t prove it’s down now, only that a significant issue crossed Google’s threshold sometime in the last week. For a live check of the current moment, you need a direct probe, not this report.

It lives at the top of the Crawl Stats reportA Google Search Console report (under Settings) that shows how Google has crawled your site over the last 90 days — total requests, download size, and average response time, broken down by response code, file type, Googlebot type, and purpose. It's only available for root-level properties (a Domain property or a URL-prefix property verified at the site's root). (Settings → Crawl stats), which is only available for domain or root-level property access — not URL-prefix subfolders. The broader Crawl Stats report covers volume and performance (total crawl requests, download size, average response time, crawl purpose, file type, response codes); host status is the tighter availability slice. When I’m diagnosing a crawl slump, host status tells me whether Google could connect; the rest of the report and my server logs tell me what happened.

The three checks

Google assesses host availability across three categories, each with its own failure-rate graph in the host details:

  • robots.txt fetching — “The graph shows the failure rate for robots.txt requests during a crawl.” This is the highest-stakes check (more below).
  • DNS resolution — “The graph shows when your DNS server didn’t recognize your hostname or didn’t respond during crawling.”
  • Server connectivity — “The graph shows when your server was unresponsive or did not provide a full response for a URL during a crawl.” Think 5xx, timeouts, and truncated/partial responses.

The three states

The indicator summarizes those checks over a rolling 90-day window:

  • Green — “Google didn’t encounter any significant crawl availability issues on your site in the past 90 days—good job!”
  • Yellow / “had problems in the past” — Google encountered at least one significant crawl-availability issue in the last 90 days, but it occurred more than a week ago. This state usually self-heals: once a week passes with no recurrence, it returns to green. You generally don’t need to do anything beyond confirming the cause is gone.
  • Red / “problems now” — Google encountered at least one significant crawl-availability issue in the last week. This is the one to triage.

So the yellow/red split is really just a timing distinction: red = within the last week (“right now”), yellow = earlier in the 90-day window (“in the past”). The exact color names and icon labels are interface details Google can change; what’s durable is that recency-based meaning.

Each category only counts as a “significant” issue once its failure rate crosses a threshold Google draws as a dotted line on that category’s graph. Google gives DNS failures above 5% of a day’s requests as one example of where that line sits — it’s an illustration for DNS, not a documented universal threshold that applies the same way to robots.txt fetching and server connectivity.

Why robots.txt failures are the most dangerous

If Google can’t get an acceptable robots.txt response, “Google will slow or stop crawling your site until it can get an acceptable robots.txt response.” That’s a whole-site pause, not a one-URL problem — which is what makes a red host status driven by robots.txt an emergency.

The detailed timeline lives in Google’s robots.txt spec. When robots.txt returns 5xx, server errors, or network failures:

  • First 12 hours: Google stops crawling the site but keeps trying to fetch the robots.txt file.
  • Next ~30 days: if Google can’t fetch a new version, it falls back to the last good cached version while still trying to fetch a fresh one. A 503 in particular triggers fairly frequent retrying; if there’s no cached version available, Google assumes there are no crawl restrictions.
  • After 30 days: behavior diverges based on overall site health. If the site generally serves fine, Google behaves as if there’s no robots.txt (and keeps checking); if the site has broader availability problems, Google stops crawling the site while still periodically requesting robots.txt.

The critical nuance — and a common myth — is the 4xx vs 5xx distinction. Google’s HTTP status-code handling for robots.txt:

  • 2xx (success): crawlersA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. process the robots.txt as served.
  • 3xx (redirectA redirect sends browsers and crawlers from a requested URL to a different one. An HTTP redirect specifically is a 3xx status code paired with a Location header; meta refresh and JavaScript redirects achieve a similar navigation without being a 3xx response themselves. Permanent redirects (301/308) are Google's signal the target should be canonical; temporary ones (302/303/307) aren't.): Google follows at least five redirect hops, then treats it as a 404.
  • 4xx (except 429): treated as if a valid robots.txt doesn’t exist — i.e. Google crawls freely.
  • 5xx / 429 / network / DNS: treated as server errors → the crawl-pause timeline above.

So a missing robots.txt (a clean 404) does not pause crawling. An unreachable one (5xx, timeout, DNS failure) does. And critically, “a robots.txt file which cannot be fetched due to DNS or networking issues, such as timeouts, invalid responses, reset or interrupted connections, and HTTP chunkingChunking is splitting a document into smaller passages so AI systems can embed, index, and retrieve the single most relevant piece — not the whole page — in response to a query. It's a foundational step in RAG pipelines and the conceptual cousin of Google's passage ranking. errors, is treated as a server error.” A flaky DNS provider can stall your crawl exactly like a 5xx.

DNS and server connectivity

DNS and server-connectivity failures are more intuitive but no less real. DNS resolution failures mean Google’s resolvers couldn’t turn your hostname into an address at some point in the window. Treat the specific cause as a hypothesis to confirm, not a given — candidates include bad nameservers, an outage at your DNS provider, or DNS-level rate limiting of GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer., and telling them apart needs your DNS records, nameserver logs, and provider status history alongside the Crawl Stats graph. Server-connectivity failures are 5xx, 429, timeouts, and partial/truncated responses; likewise, an overloaded origin and a CDN/WAF throttling Googlebot’s IP ranges are both plausible explanations that your origin and edge logs — not the graph alone — have to confirm.

This ties straight back to crawl rateCrawl rate is how fast a search engine crawler fetches pages from your site — the number of simultaneous requests it makes and the delay between them. Google sets it automatically based on your server's health; it's the supply side of crawl budget, not a ranking factor.. As I’ve written in my crawl budget guide, “Google will slow down their crawling if they receive too many 5xx (server errors) or 429 (too many requests) HTTP status codesAn HTTP status code is the three-digit number a server returns with every response to tell a browser or crawler what happened to its request — success, redirect, client error, or server error. For SEO the code matters as much as the content: it tells Google and Bing whether to index a page, follow a redirect, retry later, or drop the URL from the index..” John Mueller made the same point about how fast the crawl reacts:

“I’d only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts, so I’d double-check what actually happened (404s are generally fine & once discovered, Googlebot will retry them anyway).” — John Mueller, Google.

There’s a subtle trap here. Returning 503/429 is the legitimate short-term way to tell Googlebot to slow down — but it’s the same signal that drives a red host status. It’s a temporary throttle, not a strategy: lean on it too long and you risk pages dropping out of the index. The signal that lets you ease a crawl is the signal that, sustained, looks like an outage.

TIP Split DNS failure from server and robots.txt failure

Search Console's host state is aggregated. A direct availability check helps identify the failing layer, but one probe cannot reproduce Google's 90-day host-status calculation.

Run a live availability trace with my free Website Down Checker Free

  1. Test the affected host and record DNS, connection, TLS, redirect, and final-response evidence separately.
  2. If DNS returns NXDOMAIN, confirm the intended records and delegation before changing crawl controls.
  3. Rerun after repair and use Crawl Stats to confirm Google’s host state recovers over time.
No HTTP response is the consequence here; the observed failure starts at DNS.

The result says the DNS name was not found from this location because the fallback DNS probe returned NXDOMAIN. TLS is unknown, no HTTP response was received, and the next checks listed are DNS Checker, Hosting Checker, and Whois or RDAP Lookup.

What you don’t need to do

A few things people overthink:

  • There’s no “reset crawl rate” button. Crawl rate typically recovers on its own once availability is restored — Google doesn’t document a fixed recovery window or SLA for that, so don’t expect it to happen on any particular schedule. The old manual crawl-rate limiter in Search Console was deprecated; you don’t (and can’t) manually kick crawling back up.
  • Yellow usually fades on its own. Once a week passes without the issue recurring, the state ages back to green. Confirm the root cause is gone, then verify recovery independently rather than trusting the color alone: check current reachability, watch the failing category’s graph trend back under its threshold, confirm crawl volume in the rest of Crawl Stats looks normal again, and, if it matters for a specific URL, check recrawl/index status in Page Indexing.
  • It’s not a ranking penalty. Host status affects crawling, not rankings directly, and on its own it doesn’t explain a traffic change — correlate the affected window with crawl, index, and performance data before assuming cause. The indirect risk is that a sustained inability to crawl eventually affects freshness and, downstream, indexing — but there’s no manual action attached to a yellow or red indicator.

Host status vs the broader Crawl Stats report

Keep the scopes straight. Host status is one section of the Crawl Stats report — the availability lens. The rest of the report (the part most people mean when they say “Crawl Stats”) is about volume and performance over time. As I’ve said elsewhere, when you’re chasing a crawling problem “the best place to look is the Crawl Stats report in Google Search ConsoleA free Google service that reports how a site performs in Google Search and surfaces problems with how Google crawls, indexes, and serves it. It's first-party data straight from Google — but you don't need it to appear in results.” — and host status is the first thing in it I check, because if Google can’t reach the host, nothing else in the report matters yet. The broader report is its own topic; this one is tightly about the availability indicator.

Add an expert note

Pin an expert quote

New person? Create their unclaimed profile at /admin/experts/ → Pin a quote first.