Host Status (Crawl Stats)

TL;DR — Host status is a little indicator at the top of Google Search Console’s Crawl Stats report that tells you whether Google could reach your site over the last 90 days — it’s a historical record, not a live “is my site up right now” check. Green is good. Yellow means there was a problem, but it was more than a week ago. Red means there was a problem in the last week. It checks three things: your robots.txt file, your DNS, and whether your server responds.

What host status is

Host status is the Crawl Stats availability summary covering robots.txt fetching, DNS resolution, and server connectivity over the report window. Evidence for this claim Search Console Host status summarizes robots.txt availability, DNS resolution, and server connectivity over the previous 90 days. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Crawl Stats report It is a host-level diagnostic rather than a per-URL indexing verdict. Evidence for this claim Host status is a host-level Crawl Stats diagnostic rather than a per-URL Page Indexing decision. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Crawl Stats report

When Google wants to crawl your site, a few things have to work before it can even start: it has to find your robots.txt file, your domain name has to resolve to a server, and that server has to answer. Host status is Google’s summary of whether all of that worked over the last 90 days.

You’ll find it in Google Search Console under Settings → Crawl stats, sitting right at the top of the Crawl Stats report. It’s a single colored indicator, and it’s checking three things:

1. robots.txt fetching — can Google download your robots.txt file?
2. DNS resolution — does your domain name point to a server?
3. Server connectivity — does your server actually respond?

The three colors

• Green — Google didn’t run into any significant problems reaching your site in the last 90 days. Nothing to do.
• Yellow (“Host had problems in the past”) — there was a problem, but it happened more than a week ago. Usually this fixes itself: once a week goes by without it happening again, it turns back to green on its own.
• Red — there was a problem in the last week. This one’s worth looking into, because if Google can’t reach your site, it crawls it less.

The exact colors and labels are an interface detail Google could tweak; what stays durable is the underlying meaning — how recently a significant issue happened.

Why it matters

Host status isn’t about one broken page — it’s about whether Google can connect to your whole site. The scariest version is when Google can’t download your robots.txt file. When that happens, Google doesn’t just shrug it off; it slows down or stops crawling your entire site until it can read that file again. So it’s that underlying robots.txt, DNS, or server failure — not the host-status label itself — that quietly chokes off crawling everywhere.

The good news: there’s no button to press to fix it. Once your site is reachable again, Google typically resumes normal crawling on its own — Google doesn’t publish a fixed recovery window, so don’t expect an instant flip back to green. Your job is just to figure out which of the three checks failed and fix the underlying issue.

One thing host status is not: a ranking penalty. It’s a crawling signal, not a strike against you. Want the deeper version — the exact crawl-pause timeline, the 4xx-vs-5xx distinction, and a troubleshooting tree? Switch to the Advanced tab.

Test yourself: host status

Five quick questions on reading and responding to the Crawl Stats host-status indicator. Pick an answer for each, then check.

TL;DR — Host status is the availability lens at the top of GSC’s Crawl Stats report — a historical record of what Google observed, not a live uptime check — assessed over the last 90 days across three checks: robots.txt fetching, DNS resolution, and server connectivity. Green = no significant issues; yellow = an issue more than a week ago (typically ages out of the report on its own); red = an issue in the last week. The highest-stakes failure is an unreachable robots.txt: Google stops crawling for the first 12 hours, then falls back to the last good cached copy — if one exists — for up to ~30 days, after which behavior depends on your site’s broader availability. A 4xx robots.txt is fine (Google crawls freely); a 5xx/timeout/DNS failure is what pauses crawling — that underlying failure, not the host-status label, is what throttles crawling. There’s no reset button, and the state returning to green isn’t proof crawling, indexing, or rankings have fully recovered.

What host status actually measures

Search Console documents host status within Crawl Stats and evaluates availability over the previous 90 days. Evidence for this claim Search Console Host status summarizes robots.txt availability, DNS resolution, and server connectivity over the previous 90 days. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Crawl Stats report Diagnose individual URL outcomes separately in URL Inspection or Page Indexing. Evidence for this claim Host status is a host-level Crawl Stats diagnostic rather than a per-URL Page Indexing decision. Scope: Google Search Console report terminology and documented Search behavior; the label alone may not prove the underlying root cause. Confidence: high · Verified: 2026-07-14 Google: Crawl Stats report

Host status answers a narrow question: over the last 90 days, could Googlebot reach the host at all? That’s a different question from “is this page indexed” or “does this URL throw an error.” It’s site/host-level availability, not per-URL crawl errors — which is exactly why I keep it mentally separate from Page Indexing.

It’s also worth being precise about what kind of “availability” this is. Host status is a historical record of what Google observed during crawling, not a live uptime monitor — a green state right now doesn’t prove your site is reachable this second, and a red state doesn’t prove it’s down now, only that a significant issue crossed Google’s threshold sometime in the last week. For a live check of the current moment, you need a direct probe, not this report.

It lives at the top of the Crawl Stats report (Settings → Crawl stats), which is only available for domain or root-level property access — not URL-prefix subfolders. The broader Crawl Stats report covers volume and performance (total crawl requests, download size, average response time, crawl purpose, file type, response codes); host status is the tighter availability slice. When I’m diagnosing a crawl slump, host status tells me whether Google could connect; the rest of the report and my server logs tell me what happened.

The three checks

Google assesses host availability across three categories, each with its own failure-rate graph in the host details:

• robots.txt fetching — “The graph shows the failure rate for robots.txt requests during a crawl.” This is the highest-stakes check (more below).
• DNS resolution — “The graph shows when your DNS server didn’t recognize your hostname or didn’t respond during crawling.”
• Server connectivity — “The graph shows when your server was unresponsive or did not provide a full response for a URL during a crawl.” Think 5xx, timeouts, and truncated/partial responses.

The three states

The indicator summarizes those checks over a rolling 90-day window:

• Green — “Google didn’t encounter any significant crawl availability issues on your site in the past 90 days—good job!”
• Yellow / “had problems in the past” — Google encountered at least one significant crawl-availability issue in the last 90 days, but it occurred more than a week ago. This state usually self-heals: once a week passes with no recurrence, it returns to green. You generally don’t need to do anything beyond confirming the cause is gone.
• Red / “problems now” — Google encountered at least one significant crawl-availability issue in the last week. This is the one to triage.

So the yellow/red split is really just a timing distinction: red = within the last week (“right now”), yellow = earlier in the 90-day window (“in the past”). The exact color names and icon labels are interface details Google can change; what’s durable is that recency-based meaning.

Each category only counts as a “significant” issue once its failure rate crosses a threshold Google draws as a dotted line on that category’s graph. Google gives DNS failures above 5% of a day’s requests as one example of where that line sits — it’s an illustration for DNS, not a documented universal threshold that applies the same way to robots.txt fetching and server connectivity.

Why robots.txt failures are the most dangerous

If Google can’t get an acceptable robots.txt response, “Google will slow or stop crawling your site until it can get an acceptable robots.txt response.” That’s a whole-site pause, not a one-URL problem — which is what makes a red host status driven by robots.txt an emergency.

The detailed timeline lives in Google’s robots.txt spec. When robots.txt returns 5xx, server errors, or network failures:

• First 12 hours: Google stops crawling the site but keeps trying to fetch the robots.txt file.
• Next ~30 days: if Google can’t fetch a new version, it falls back to the last good cached version while still trying to fetch a fresh one. A 503 in particular triggers fairly frequent retrying; if there’s no cached version available, Google assumes there are no crawl restrictions.
• After 30 days: behavior diverges based on overall site health. If the site generally serves fine, Google behaves as if there’s no robots.txt (and keeps checking); if the site has broader availability problems, Google stops crawling the site while still periodically requesting robots.txt.

The critical nuance — and a common myth — is the 4xx vs 5xx distinction. Google’s HTTP status-code handling for robots.txt:

• 2xx (success): crawlers process the robots.txt as served.
• 3xx (redirect): Google follows at least five redirect hops, then treats it as a 404.
• 4xx (except 429): treated as if a valid robots.txt doesn’t exist — i.e. Google crawls freely.
• 5xx / 429 / network / DNS: treated as server errors → the crawl-pause timeline above.

So a missing robots.txt (a clean 404) does not pause crawling. An unreachable one (5xx, timeout, DNS failure) does. And critically, “a robots.txt file which cannot be fetched due to DNS or networking issues, such as timeouts, invalid responses, reset or interrupted connections, and HTTP chunking errors, is treated as a server error.” A flaky DNS provider can stall your crawl exactly like a 5xx.

DNS and server connectivity

DNS and server-connectivity failures are more intuitive but no less real. DNS resolution failures mean Google’s resolvers couldn’t turn your hostname into an address at some point in the window. Treat the specific cause as a hypothesis to confirm, not a given — candidates include bad nameservers, an outage at your DNS provider, or DNS-level rate limiting of Googlebot, and telling them apart needs your DNS records, nameserver logs, and provider status history alongside the Crawl Stats graph. Server-connectivity failures are 5xx, 429, timeouts, and partial/truncated responses; likewise, an overloaded origin and a CDN/WAF throttling Googlebot’s IP ranges are both plausible explanations that your origin and edge logs — not the graph alone — have to confirm.

This ties straight back to crawl rate. As I’ve written in my crawl budget guide, “Google will slow down their crawling if they receive too many 5xx (server errors) or 429 (too many requests) HTTP status codes.” John Mueller made the same point about how fast the crawl reacts:

“I’d only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts, so I’d double-check what actually happened (404s are generally fine & once discovered, Googlebot will retry them anyway).” — John Mueller, Google.

There’s a subtle trap here. Returning 503/429 is the legitimate short-term way to tell Googlebot to slow down — but it’s the same signal that drives a red host status. It’s a temporary throttle, not a strategy: lean on it too long and you risk pages dropping out of the index. The signal that lets you ease a crawl is the signal that, sustained, looks like an outage.

What you don’t need to do

A few things people overthink:

• There’s no “reset crawl rate” button. Crawl rate typically recovers on its own once availability is restored — Google doesn’t document a fixed recovery window or SLA for that, so don’t expect it to happen on any particular schedule. The old manual crawl-rate limiter in Search Console was deprecated; you don’t (and can’t) manually kick crawling back up.
• Yellow usually fades on its own. Once a week passes without the issue recurring, the state ages back to green. Confirm the root cause is gone, then verify recovery independently rather than trusting the color alone: check current reachability, watch the failing category’s graph trend back under its threshold, confirm crawl volume in the rest of Crawl Stats looks normal again, and, if it matters for a specific URL, check recrawl/index status in Page Indexing.
• It’s not a ranking penalty. Host status affects crawling, not rankings directly, and on its own it doesn’t explain a traffic change — correlate the affected window with crawl, index, and performance data before assuming cause. The indirect risk is that a sustained inability to crawl eventually affects freshness and, downstream, indexing — but there’s no manual action attached to a yellow or red indicator.

Host status vs the broader Crawl Stats report

Keep the scopes straight. Host status is one section of the Crawl Stats report — the availability lens. The rest of the report (the part most people mean when they say “Crawl Stats”) is about volume and performance over time. As I’ve said elsewhere, when you’re chasing a crawling problem “the best place to look is the Crawl Stats report in Google Search Console” — and host status is the first thing in it I check, because if Google can’t reach the host, nothing else in the report matters yet. The broader report is its own topic; this one is tightly about the availability indicator.

AI summary

A condensed take on the Advanced version:

• Host status = “did Google’s crawl history show it could reach the host?” It’s the availability indicator at the top of GSC’s Crawl Stats report (Settings → Crawl stats), assessed over the last 90 days. It’s a historical record, not a live uptime check, and it’s host/site-level, not per-URL.
• Three checks: robots.txt fetching, DNS resolution, server connectivity — each with its own failure-rate graph, flagged once it crosses a threshold (Google’s own example: DNS failures above 5% of a day’s requests).
• Three states: green (no significant issues in 90 days); yellow (issue more than a week ago — typically ages back to green on its own); red (issue in the last week — triage it). Colors and labels are interface detail; recency is the durable meaning.
• robots.txt is the highest-stakes check. If Google can’t get an acceptable robots.txt response it’s the underlying failure — not the host-status label — that slows or stops crawling the whole site: ~12 hours fully stopped while retrying, then up to ~30 days on the last cached copy if one exists, then behavior depends on overall site health.
• 4xx vs 5xx matters: a 404 robots.txt = crawl freely; a 5xx/timeout/DNS failure = crawl pause. DNS/network errors count as server errors.
• 5xx and 429 throttle crawl rate (Mueller: 429/500/503/timeouts react fast; 404s are generally fine). The same 503/429 throttle is a short-term tool, not a strategy.
• Recovery typically happens on its own once availability is restored, but Google documents no fixed recovery window — no reset button; the manual crawl-rate limiter was deprecated. The state aging back to green isn’t proof crawling, indexing, or rankings have fully recovered; verify independently.
• Not a ranking penalty on its own. It’s a crawl-availability signal, distinct from Page Indexing errors and broader than the rest of the Crawl Stats report — correlate with crawl/index/performance data before assigning cause to a traffic change.

Official documentation

Primary-source documentation from Google.

Google

• Crawl Stats report — the report that contains host status; defines the three checks and the three states.
• How Google interprets the robots.txt specification — the robots.txt fetch-failure timeline (12-hour stop, 30-day cached fallback) and the HTTP status-code handling.
• Troubleshoot Google Search crawling errors — Google’s general crawl-error troubleshooting.
• Debug Network and DNS errors for Google’s crawlers — DNS and network-level diagnostics.

Quotes from the source

On-the-record statements from Google. Each link jumps toward the quoted passage on the source page.

Google — the three states (Crawl Stats report)

• “Google didn’t encounter any significant crawl availability issues on your site in the past 90 days—good job!” — Google Search Console Help (green state). Jump to quote
• “Google encountered at least one significant crawl availability issue in the last 90 days on your site, but it occurred more than a week ago.” — Google Search Console Help (yellow / “problems in the past” state). Jump to quote
• “Google encountered at least one significant crawl availability issue in the last week on your site.” — Google Search Console Help (red / “problems now” state). Jump to quote

Google — the three checks (Crawl Stats report)

• “The graph shows the failure rate for robots.txt requests during a crawl.” — Google Search Console Help (robots.txt fetching). Jump to quote
• “The graph shows when your DNS server didn’t recognize your hostname or didn’t respond during crawling.” — Google Search Console Help (DNS resolution). Jump to quote
• “The graph shows when your server was unresponsive or did not provide a full response for a URL during a crawl.” — Google Search Console Help (server connectivity). Jump to quote

Google — the robots.txt crawl-pause mechanic

• “Google will slow or stop crawling your site until it can get an acceptable robots.txt response.” — Google Search Console Help (robots.txt fetching section). Jump to quote
• “For the first 12 hours, Google stops crawling the site but keeps trying to fetch the robots.txt file.” — Google for Developers, robots.txt spec. Jump to quote
• “A robots.txt file which cannot be fetched due to DNS or networking issues, such as timeouts, invalid responses, reset or interrupted connections, and HTTP chunking errors, is treated as a server error.” — Google for Developers, robots.txt spec. Jump to quote

John Mueller, Google (relayed via Search Engine Journal)

• “I’d only expect the crawl rate to react that quickly if they were returning 429 / 500 / 503 / timeouts, so I’d double-check what actually happened (404s are generally fine & once discovered, Googlebot will retry them anyway).” Read the coverage

Host-status triage checklist

A pass to run when host status goes yellow or red. Work the failing check first.

If robots.txt fetching is failing

• Confirm robots.txt returns 200 (or a clean 404) — not 5xx, 429, or a timeout. A 404 is fine; a server error is what pauses crawling.
• Test the file in GSC’s robots.txt report and fetch it yourself from multiple locations.
• Check whether a CDN, WAF, or rate-limiter is blocking or throttling Googlebot.
• Make sure the file is reachable and reasonably small (a few hundred KB at most).

If DNS resolution is failing

• Confirm your domain resolves (dig/nslookup from outside your network).
• Check your DNS provider’s status/uptime for the failure window.
• Look for misconfigured or slow nameservers.
• Verify nothing is rate-limiting Googlebot at the DNS layer.

If server connectivity is failing

• Check logs for 5xx, 429, and timeouts during the flagged window.
• Confirm the origin isn’t overloaded (capacity, DB, upstream timeouts).
• Verify a firewall/CDN isn’t throttling Googlebot’s IP ranges.
• Look for partial/truncated responses.

Always

• Cross-reference the host-status graph timestamps with deploys, outages, and config changes.
• Corroborate with server logs and the Crawl Stats response-code breakdown.
• Once fixed, do nothing else — crawling recovers automatically; yellow self-heals after a clear week.

The mental models

1. Three checks, one question. Host status only asks “can Google reach the host?” via robots.txt fetching, DNS resolution, and server connectivity. Identify which check failed before you touch anything — the fix for a DNS outage looks nothing like the fix for a 5xx storm.

2. The state machine is a clock, not a severity dial. Green = clean for 90 days. Red = a problem in the last week. Yellow = a problem earlier in the window. Yellow → green happens on its own once a week passes without recurrence. So “red” isn’t “worse than yellow” — it’s “more recent.”

3. The robots.txt decision rule.

• 2xx → processed as served.
• 3xx → followed ~5 hops, then treated as 404.
• 4xx (except 429) → treated as no robots.txt → crawl freely.
• 5xx / 429 / timeout / DNS failure → server error → crawl pause (12h stop → ~30d cached fallback → depends on site health).

A missing robots.txt is safe. An unreachable one is the emergency.

4. Host status is upstream of everything else. If Google can’t connect to the host, no per-URL report tells you anything useful yet. Clear host status first, then read the rest of Crawl Stats and Page Indexing.

5. The throttle is also the symptom. 503/429 is the legitimate way to tell Googlebot to slow down — and the exact signal that lights up a red host status. Use it briefly and on purpose; sustained, it stops looking like a throttle and starts looking like an outage (with indexing risk).

Host status — cheat sheet

The three states (90-day window)

The three checks

robots.txt status-code handling

robots.txt crawl-pause timeline (on server error)

• First 12 hours: stops crawling, keeps retrying the file.
• Next ~30 days: uses last good cached copy, still retrying.
• After 30 days: depends on overall site health.

Fast facts

• Host status lives in Settings → Crawl stats (top of the report).
• Domain/root-property only — not URL-prefix subfolders.
• Recovery is typically hands-off once availability is restored — no reset button, no documented fixed timeline (manual crawl-rate limiter deprecated).
• It’s host-level, not per-URL — distinct from Page Indexing errors.
• It’s not a ranking penalty.

Tools for diagnosing host status

• Google Search Console — Crawl Stats report — where host status lives (Settings → Crawl stats), plus the three per-check failure-rate graphs and the response-code breakdown.
• GSC robots.txt report — confirm Google can fetch your robots.txt and see the status code it got.
• Server log file analysis — the ground truth for what Googlebot actually received (5xx/429/timeouts) and when. Tools: Screaming Frog Log File Analyser, or pipe logs into BigQuery / a log platform.
• DNS diagnostics — dig / nslookup, plus your DNS provider’s status page, to confirm resolution during the flagged window.
• Uptime / status monitoring — to line up host-status dips with real outages, deploys, and config changes.

What should I do with this host-status state?

Host-status mistakes that delay recovery

Treating a missing robots.txt file like an outage

Why it’s wrong: Google treats most 4xx responses, including a clean 404, as if no robots.txt restrictions exist. 5xx, 429, DNS failures, and timeouts are the responses that trigger crawl pausing. Do instead: test the actual response code and availability before creating a file simply to make the 404 disappear.

Waiting for a manual crawl-rate reset

Why it’s wrong: there is no reset button, and the old crawl-rate control is not the recovery path. Do instead: restore stable responses and let Googlebot increase crawling automatically.

Reading red and yellow as severity grades

Why it’s wrong: the colors mainly encode recency. Red means a significant problem occurred in the last week; yellow means it occurred earlier in the rolling 90-day window. Do instead: use the failing check and timestamps to judge severity.

Using 503 or 429 as a permanent crawl-control strategy

Why it’s wrong: those codes can be a legitimate short-term throttle, but they are also server-error signals that reduce crawling and can become an indexing risk when sustained. Do instead: use them briefly during real overload and fix the capacity or edge rule causing the pressure.

Diagnosing a host-level failure one URL at a time

Why it’s wrong: host status summarizes robots.txt, DNS, and connectivity failures at the host level. Do instead: correlate the failure-rate graph with server, CDN, WAF, DNS, and deployment logs before chasing isolated page templates.

Quick host-availability checks

Check robots.txt and its redirect chain

Run this in a macOS/Linux shell. Replace the hostname; -L follows redirects and the output exposes whether the final response is acceptable or a crawl-pausing server error.

curl -sS -L -o /dev/null -w 'final=%{url_effective} code=%{http_code} redirects=%{num_redirects} time=%{time_total}s\n' https://example.com/robots.txt

Run the equivalent in PowerShell to see each hop and final status.

$r = Invoke-WebRequest -Uri 'https://example.com/robots.txt' -MaximumRedirection 5
[pscustomobject]@{ Status = [int]$r.StatusCode; FinalUrl = $r.BaseResponse.ResponseUri.AbsoluteUri }

Check authoritative DNS from the command line

Run these in a macOS/Linux shell. A missing or intermittent answer points to the DNS path rather than the origin application.

dig +short A example.com
dig +short AAAA example.com
dig +trace example.com

Run the PowerShell equivalent on Windows.

Resolve-DnsName example.com -Type A
Resolve-DnsName example.com -Type AAAA

Summarize crawl-pause response codes from an access log

Run this in a shell against a standard combined access log. Adjust the log field if your format differs; the expression counts 429 and 5xx responses for Googlebot.

awk 'tolower($0) ~ /googlebot/ && ($9 == 429 || $9 ~ /^5[0-9][0-9]$/) { count[$9]++ } END { for (code in count) print code, count[code] }' access.log | sort

Resources worth your time

My related writing

• When Should You Worry About Crawl Budget? — where the 5xx/429 throttle, the Crawl Stats report, and host availability fit into the bigger crawl-budget picture.
• Robots.txt and SEO: Everything You Need to Know — the file whose unreachability drives the highest-stakes host-status failure.

From others

• Googlebot Crawl Slump? Mueller Points To Server Errors — Search Engine Journal, Aug 2025; the Mueller quote on 429/500/503/timeouts.
• 5 Top Crawl Stats Insights in Google Search Console — Search Engine Journal, on reading the broader report host status sits inside.
• How Google interprets the robots.txt specification — Google for Developers; the authoritative source for the 12-hour crawl-stop and 30-day cached-fallback timeline.
• Debug Network and DNS errors for Google’s crawlers — Google for Developers; DNS and network-level diagnostics directly relevant to the DNS resolution check.
• Troubleshoot Google Search crawling errors — Google for Developers; Google’s own guide for diagnosing the server-connectivity failures that drive a red host status.
• r/TechSEO — the community for crawl/availability debugging.

Host status

The at-a-glance availability indicator at the top of Google Search Console's Crawl Stats report. It shows whether Googlebot hit significant problems reaching your site in the last 90 days across three checks — robots.txt fetching, DNS resolution, and server connectivity.

Related: Crawl Stats report, Robots.txt, Crawling

Host status

Host status is the availability indicator at the top of Google Search Console’s Crawl Stats report. It answers one question: in the last 90 days, could Googlebot actually reach your host? It checks three things — whether Google can fetch your robots.txt, whether DNS resolves your hostname, and whether your server responds. Green means no significant problems, and the two warning states tell you whether a problem is happening now or happened earlier in the window.

It’s a host-level signal, not a per-URL one. That’s what makes it different from Page Indexing errors, which are about individual pages — host status is about whether Google can connect to the host at all. When it can’t — especially when robots.txt is unreachable — Google slows or stops crawling the entire site, which is why a red host status is closer to an emergency than a notice.

Host status is one section of the broader Crawl Stats report (Settings → Crawl stats), available only at the domain/root-property level. The rest of that report covers crawl volume, response codes, and timing; host status isolates the availability question.

Related: Crawl Stats report, Robots.txt, Crawling

Changelog