Free, no signup. Server logs answer a question your analytics can't: what are crawlers actually doing on your site. Drop one in and see crawl budget by bot and by section, how much is wasted on 404s and
redirects, how AI crawlers compare to search engines, and which requests are impostors
faking a crawler user-agent.
Multiple files allowed. Big logs stream in chunks.
Runs entirely in your browser — nothing you paste is uploaded or stored. Spot-checking a suspicious IP against the Googlebot Verifier sends only that IP to our server. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Bot data last synced Jul 27, 2026.
Feedback
Report a bug
Found something broken in Log File Analyzer? Let us know what happened — this goes straight to a private triage queue, not a public list.
Reading log…
Sample report
Say your log contains two lines — a genuine Googlebot hit and a Googlebot impostor:
1 IP sent a search/AI-crawler User-Agent from outside that operator's published ranges — robots.txt won't stop them (impostors ignore it). Block or rate-limit these at the firewall/WAF by IP or ASN. Top offenders: 193.42.33.7 (as Googlebot, ×1). Run each through the Googlebot Verifier for the owning network (ASN) before blocking.
Both lines roll up under one Googlebot row — the table groups by crawler
identity, not by verification status, so the genuine 200 and the spoofed 404 both count toward
the same bot's totals. That's why Waste is 50% (1 of 2 requests was non-200), not
0% — see what waste means ↓.
Verified vs. spoofed is per-request, shown per-IP in the spoofer table below the
bot row: 193.42.33.7 isn't in Google's published ranges, so its Googlebot user-agent
is flagged rather than trusted.
The recommendation is severity-ranked — a spoofed crawler hitting
/wp-login.php is high severity because robots.txt can't stop an impostor
that ignores it; only a firewall/WAF IP or ASN block does. Confirm the real owner first with the
Googlebot Verifier.
How to use it
Export an access log from your server, CDN, or Cloudflare Logpush. Common formats
(nginx/Apache, IIS/W3C, JSON) all work — the format is auto-detected.
Drop the file onto the upload box, or browse to it. You can
drop several files at once; large logs stream in chunks. Nothing is uploaded — parsing runs in
your browser. No log handy? Click load demo data.
Read the summary cards first (requests parsed, crawler share, verified vs.
spoofed), then check the llms.txt fetch evidence before working down through
crawl budget by bot, the spoofer report, section budget, and the most-wasted URLs.
Check the Recommendations panel for ranked fixes and copy-paste robots.txt
lines. In the spoofer report, hit Verify IP to run a live reverse-DNS + ASN
spot-check on any suspicious address.
Use Export bots CSV or Export wasted URLs CSV to pull the
findings into a spreadsheet or ticket.
Saved targets, named lists, and recent check summaries remain only in this browser.
Who's crawling you
Requests grouped by crawler type. AI crawlers and search engines want different things from your site. "Other / unclassified" is everything with no known crawler user-agent — human browsers and untracked tools.
Crawl over time
Daily crawl requests by crawler type across the log's date range.
Crawl budget by bot
Each crawler seen in the log: how many requests, how it was verified, and how much of its crawl hit redirects or errors (excluding 304 revalidation).
Spoofer report
Requests that sent a crawler's user-agent from an IP outside that operator's published ranges — likely impostors.
robots.txt won't stop these (impostors ignore it); block by IP or ASN. Spot-check any IP for the real network owner via
forward-confirmed reverse DNS — this is the one action that leaves your browser (a single lookup to our verifier API).
Crawl budget by section
Where crawlers spend their time on your site (bot traffic only). High waste in a section means crawlers are burning budget on broken or redirecting URLs there.
Most-wasted URLs
Individual URLs where crawlers hit redirects or errors most often (excluding 304 revalidation) — the highest-leverage fixes for crawl budget.
AI-invented URL candidates
Observed 404/410 requests with an AI crawler or known AI-assistant referrer. This is the primary evidence mode: a row is a candidate, not proof that a model invented it — it may be a deleted or migrated URL. Suggested destinations compare only against the sitemap/URL list you supplied above.
Ask the site’s open-model simulator for plausible paths, then check their live status. This is explicitly a simulation, never observed assistant output; log evidence above is stronger.
Expected URLs vs crawler requests
Recommendations
Rate this tool
A server access log is the ground truth for how crawlers actually behave on your site — more
complete than any report, because it records every single request. This tool turns a raw log into
a crawl-budget picture: who's crawling, what they're wasting budget on, and who's faking a crawler
identity. For the reasoning behind each metric, see the
AI crawler log analysis guide and the
crawl budget explainer.
What the results mean
The verification badges on each bot and IP tell you how much to trust a request's identity:
Verified — the source IP falls inside the crawler
operator's published ranges. This is a genuine crawler.
Spoofed — a crawler user-agent from an IP
outside that operator's ranges. A likely impostor; block by IP or ASN.
Unverifiable — a known crawler whose operator
publishes no IP ranges to check against (e.g. Bytespider), so genuine and fake can't be
separated by list alone.
Crawlers are grouped by type, and each group is colour-coded:
Search — traditional search-index crawlers (Googlebot, bingbot).
AI training — bots that fetch pages to train models (e.g. GPTBot).
AI search — bots that fetch pages to answer live AI queries (e.g. OAI-SearchBot).
User-triggered — fetches a person kicked off, such as a citation preview.
Other / unclassified — no known crawler user-agent: human browsers and untracked tools.
Waste is the share of a bot's, section's, or URL's requests that returned a
redirect (excluding 304), 4xx, or 5xx. The status mix bar breaks all responses down: green 2xx, amber 3xx, red 4xx, dark-red 5xx.
Recommendations carry a severity — high (red), medium (amber),
or info (blue) — reflecting how much budget the fix reclaims.
How it works
Parsed in your browser. A Web Worker streams the file chunk by chunk and
parses each line — nginx/Apache combined & common, IIS/W3C Extended, and JSON (including
Cloudflare Logpush). The file is never uploaded.
Every request classified. The user-agent is matched to a known crawler, then
the source IP is checked against that operator's published ranges — the same weekly-refreshed
lists the Googlebot Verifier uses. A crawler UA from an IP outside
its ranges is flagged as a likely spoofer.
Aggregated, not stored. Requests roll up into crawl budget, status-code
waste, section and AI-vs-search breakdowns, and a spoofer report. Close the tab and it's gone.
Optional spot-checks. For a suspicious IP you can run a full
forward-confirmed reverse-DNS check and ASN lookup — the only step that contacts a server,
rate-limited, one IP at a time.
Crawler IP ranges were last refreshed
today.
Features
Auto-detects nginx/Apache combined & common, IIS/W3C Extended, and JSON (Cloudflare Logpush) logs; multiple files at once.
Crawl budget by bot with verified/spoofed/unverifiable counts, unique IPs, status mix, and waste percentage.
AI-vs-search breakdown so you can see how much of your budget AI crawlers now consume versus traditional search.
Spoofer report listing crawler-UA requests from out-of-range IPs, with a live per-IP reverse-DNS + ASN verify button.
Crawl budget by section and a most-wasted-URLs table pinpointing the highest-leverage fixes.
Daily crawl-over-time trend chart when the log carries timestamps.
Ranked recommendations with copy-paste robots.txt lines, plus CSV export of bots and wasted URLs.
Limitations
It analyses what's in the log — it can't see requests your server never logged, and it can't tell
you why a crawler chose a URL. Verification is only as good as the operators' published
ranges: crawlers with no published ranges are marked unverifiable rather than judged, and a
determined impostor using a genuine-crawler IP won't be caught by list matching alone (use the
reverse-DNS spot-check for those). Sections are inferred from URL paths, so your site's structure
determines how clean the grouping is. It reports crawl behaviour, not indexing outcomes — pair it
with Search Console to connect a crawled URL to whether it actually ranks.
Frequently asked questions
Are my log files uploaded anywhere?
No. Parsing and analysis run entirely in your browser through a Web Worker — the file is read from disk and streamed in chunks locally, and nothing is sent to a server. The one exception is the optional per-IP spot-check in the spoofer report, which sends a single IP address (not your log) to the verifier API to run a forward-confirmed reverse-DNS and ASN lookup. Close the tab and all parsed data is gone.
What log formats does it support?
The format is auto-detected. It reads nginx and Apache combined and common formats, IIS / W3C Extended, and JSON logs including Cloudflare Logpush. You can drop multiple files at once, and large logs stream in chunks so a multi-gigabyte access log will not lock up the tab.
How does it tell a real crawler from a spoofed one?
Each request is matched to a known crawler by user-agent, then the source IP is checked against that operator’s published IP ranges — the same weekly-refreshed lists the GooglebotGooglebot is Google's web crawler — the software that fetches pages so Google can index and rank them. It comes in two variants, Googlebot Smartphone (primary, under mobile-first indexing) and Googlebot Desktop, and runs an evergreen Chromium renderer. Verifier uses. A request that sends, say, a Googlebot user-agent from an IP outside Google’s ranges is flagged as a likely spoofer. Some operators (like ByteDance’s Bytespider) publish no ranges, so those hits are marked unverifiable rather than spoofed.
What counts as wasted crawl budget?
Any crawler requestA crawler — also called a spider or bot — is an automated program that fetches web pages, extracts their links, and queues new URLs to visit. Search engines use crawlers to discover and download content for their index. that gets a redirect (except a useful 304 Not ModifiedHTTP 304 Not Modified is the response to a conditional GET/HEAD whose condition evaluates false, in the 3xx class but not a redirect: it has no Location header and no body. It tells a client its cached copy is still valid. For SEO it has no direct ranking effect and no indexing effect beyond a possible signal recalculation, though it can help crawl efficiency indirectly on large sites. revalidation), a 4xx error such as 404 or 410, or a 5xx server error. The tool rolls waste up per bot, per section, and per individual URL so you can see where the highest-leverage fixes are.
Can I block spoofers with robots.txt?
No. Impostors send a crawler user-agent precisely so they can ignore robots.txtA plain-text file at the root of a host that tells crawlers which URLs they may and may not request. It controls crawling, not indexing — a blocked URL can still be indexed if it's linked from elsewhere., so a Disallow line does nothing to them — block them by IP or ASN at the edge or firewall instead. robots.txt is the right tool for genuine crawlers wasting budget on sections you don’t want crawled, and the Recommendations panel generates copy-paste Disallow lines for exactly those cases.
Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Common issues & how to fix them
Log File Analyzer · one finding
Crawler spends requests on non-useful URLs
Recommended action: Disallow or canonicalize low-value parameter, filter, search, and duplicate URLs, then strengthen links to the URLs crawlers should prioritize.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Crawler repeatedly receives error responses
Recommended action: Repair or redirect the most-requested crawler URLs returning 4xx/5xx responses and remove internal links to URLs that should stay gone.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Crawler encounters excessive redirects
Recommended action: Update internal links and sitemap entries to point directly to final URLs, and collapse redirect chains to one hop.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Crawler visits URL outside the expected URL set
Recommended action: Either add the crawler-discovered URL to the canonical internal-link and sitemap set or redirect/remove it if it is unintended.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Expected important URL has no crawler hits
Recommended action: Add crawlable internal links and a canonical sitemap entry for the expected URL, then check robots and authentication barriers.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Crawler activity drops materially
Recommended action: Compare the drop date with robots, status, sitemap, deployment, and server-capacity changes, then restore the affected crawl path.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Crawler activity spikes materially
Recommended action: Identify the URL pattern causing the spike and contain newly exposed parameters, traps, redirects, or duplicate crawl paths.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
Log File Analyzer · one finding
Crawler response times are slow
Recommended action: Cache or optimize the slowest crawler endpoints and reduce origin latency before requesting a fresh crawl.
Fix principle: Crawl-budget remediation starts with verified crawler requests and URL-pattern evidence. Fix the generating source of repeated errors, redirects, duplicates, traps, and slow responses; do not block symptoms until you understand whether internal links, sitemaps, external references, or stale crawler memory keep producing them.
Implementation for your setup
Confirm first: Verified search crawlers spend material request volume on errors, redirects, duplicate parameters, traps, or other non-useful URL families. Important URLs receive little or no verified crawl activity while low-value families dominate requests. Crawl volume changes sharply or response latency/error rates rise for verified crawlers.
Do not apply when: The observation is based only on an unverified user-agent string or an unrepresentative log window. The site is small/stable enough that the observed requests do not delay discovery or recrawling of important content. The apparent waste is a bounded migration recrawl or another expected temporary event.
Likely control points: server · cms-app · edge-cdn · content-data · managed-vendor
Verify crawler identity where possible and segment requests by normalized URL family, status, redirect count, response time, bytes, and time window.
Separate persistent waste from expected events such as a migration, new sitemap, or short crawl spike.
Trace each high-volume low-value family back to its discovery source: internal links, sitemap entries, parameters/facets, redirecting links, external references, or previously known URLs.
Fix the source: update internal links and sitemaps, collapse duplicate URL generation, flatten redirects, return correct 404/410 responses, improve caching, or repair slow/erroring templates.
Use robots.txt only for URL families that should not be crawled at all; do not block a URL whose canonical/noindex or error evidence the crawler still needs to observe.
Verify on this platform
Compare equivalent before/after log windows by verified crawler, URL family, response class, latency, and useful-page crawl share.
Confirm important URLs continue to be discovered and served successfully.
Use Search Console crawl and indexing evidence as a separate corroborating source, not as a substitute for request logs.
Rollback
Restore a removed internal discovery path, sitemap source, or crawl rule if important URLs lose their only reliable discovery route.
Revert caching or routing changes if they cause stale content or incorrect responses.
Caveats
Google's crawl-budget guidance primarily targets very large or rapidly changing sites; small sites usually do not need a crawl-budget project.
User-agent strings are spoofable. Treat unverified bot identity as uncertain evidence.
Lower crawl volume is not automatically better; the goal is efficient coverage of useful content.
Use verified crawler identity and comparable time windows.
Measure request share, errors, redirects, latency, and useful-page coverage by URL family.
Confirm the generating source—not only the observed URLs—was corrected.
Preserve discovery and successful responses for important inventory.
Escalate when
Escalate when identity cannot be verified, the site is large enough that changes can materially alter discovery, or URL generation spans multiple teams and no owner can stop the source.
WarningsCrawler claim is outside the operator’s published rangeFix: Treat the crawler as unverified and block or investigate it until its IP matches the operator’s published range.
