Log File Analyzer
Free, no signup. Server logs answer a question your analytics can't: what are crawlers actually doing on your site. Drop one in and see crawl budget by bot and by section, how much is wasted on 404s and redirects, how AI crawlers compare to search engines, and which requests are impostors faking a crawler user-agent.
Handles nginx / Apache (combined & common), IIS / W3C Extended, and JSON logs (incl. Cloudflare Logpush) — format auto-detected. Crawler ranges updated 34 days ago.
Runs entirely in your browser — nothing you paste is uploaded or stored. Spot-checking a suspicious IP against the Googlebot Verifier sends only that IP to our server. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Sample report
Say your log contains two lines — a genuine Googlebot hit and a Googlebot impostor:
66.249.66.1 - - [08/Jul/2026:09:14:22 +0000] "GET /blog/technical-seo-guide HTTP/1.1" 200 8213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
193.42.33.7 - - [08/Jul/2026:09:14:41 +0000] "GET /wp-login.php HTTP/1.1" 404 402 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"…and the analyzer returns:
Example data — captured by running this page's own analyzer against the two lines above
| Bot | Type | Requests | Verified | Spoofed | Waste |
|---|---|---|---|---|---|
| Googlebot Google | Search | 2 | 1 | 1 | 50% |
| IP | Claimed to be | Requests | User-agent |
|---|---|---|---|
193.42.33.7 | Googlebot | 1 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
1 requests spoofed a verified crawler
1 IP sent a search/AI-crawler User-Agent from outside that operator's published ranges — robots.txt won't stop them (impostors ignore it). Block or rate-limit these at the firewall/WAF by IP or ASN. Top offenders: 193.42.33.7 (as Googlebot, ×1). Run each through the Googlebot Verifier for the owning network (ASN) before blocking.
- Both lines roll up under one Googlebot row — the table groups by crawler identity, not by verification status, so the genuine 200 and the spoofed 404 both count toward the same bot's totals. That's why Waste is 50% (1 of 2 requests was non-200), not 0% — see what waste means ↓.
- Verified vs. spoofed is per-request, shown per-IP in the spoofer table below the
bot row:
193.42.33.7isn't in Google's published ranges, so its Googlebot user-agent is flagged rather than trusted. - The recommendation is severity-ranked — a spoofed crawler hitting
/wp-login.phpis high severity becauserobots.txtcan't stop an impostor that ignores it; only a firewall/WAF IP or ASN block does. Confirm the real owner first with the Googlebot Verifier.
How to use it
- Export an access log from your server, CDN, or Cloudflare Logpush. Common formats (nginx/Apache, IIS/W3C, JSON) all work — the format is auto-detected.
- Drop the file onto the upload box, or browse to it. You can drop several files at once; large logs stream in chunks. Nothing is uploaded — parsing runs in your browser. No log handy? Click load demo data.
- Read the summary cards first (requests parsed, crawler share, verified vs. spoofed), then check the llms.txt fetch evidence before working down through crawl budget by bot, the spoofer report, section budget, and the most-wasted URLs.
- Check the Recommendations panel for ranked fixes and copy-paste robots.txt lines. In the spoofer report, hit Verify IP to run a live reverse-DNS + ASN spot-check on any suspicious address.
- Use Export bots CSV or Export wasted URLs CSV to pull the findings into a spreadsheet or ticket.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Who's crawling you
Requests grouped by crawler type. AI crawlers and search engines want different things from your site. "Other / unclassified" is everything with no known crawler user-agent — human browsers and untracked tools.
Crawl over time
Daily crawl requests by crawler type across the log's date range.
Crawl budget by bot
Each crawler seen in the log: how many requests, how it was verified, and how much of its crawl hit redirects or errors (excluding 304 revalidation).
Spoofer report
Requests that sent a crawler's user-agent from an IP outside that operator's published ranges — likely impostors. robots.txt won't stop these (impostors ignore it); block by IP or ASN. Spot-check any IP for the real network owner via forward-confirmed reverse DNS — this is the one action that leaves your browser (a single lookup to our verifier API).
Crawler × page-type matrix
Where each recognized crawler concentrates requests across inferred page cohorts. Cell intensity represents request count; the label also shows wasted requests.
Crawl budget by section
Where crawlers spend their time on your site (bot traffic only). High waste in a section means crawlers are burning budget on broken or redirecting URLs there.
Most-wasted URLs
Individual URLs where crawlers hit redirects or errors most often (excluding 304 revalidation) — the highest-leverage fixes for crawl budget.
AI-invented URL candidates
Observed 404/410 requests with an AI crawler or known AI-assistant referrer. This is the primary evidence mode: a row is a candidate, not proof that a model invented it — it may be a deleted or migrated URL. Suggested destinations compare only against the sitemap/URL list you supplied above.
Optional URL probe simulation
Ask the site’s open-model simulator for plausible paths, then check their live status. This is explicitly a simulation, never observed assistant output; log evidence above is stronger.
Expected URLs vs crawler requests
Recommendations
Rate this tool
A server access log is the ground truth for how crawlers actually behave on your site — more complete than any report, because it records every single request. This tool turns a raw log into a crawl-budget picture: who's crawling, what they're wasting budget on, and who's faking a crawler identity. For the reasoning behind each metric, see the AI crawler log analysis guide and the crawl budget explainer.
What the results mean
The verification badges on each bot and IP tell you how much to trust a request's identity:
- Verified — the source IP falls inside the crawler operator's published ranges. This is a genuine crawler.
- Spoofed — a crawler user-agent from an IP outside that operator's ranges. A likely impostor; block by IP or ASN.
- Unverifiable — a known crawler whose operator publishes no IP ranges to check against (e.g. Bytespider), so genuine and fake can't be separated by list alone.
Crawlers are grouped by type, and each group is colour-coded:
- Search — traditional search-index crawlers (Googlebot, bingbot).
- AI training — bots that fetch pages to train models (e.g. GPTBot).
- AI search — bots that fetch pages to answer live AI queries (e.g. OAI-SearchBot).
- User-triggered — fetches a person kicked off, such as a citation preview.
- Other / unclassified — no known crawler user-agent: human browsers and untracked tools.
Waste is the share of a bot's, section's, or URL's requests that returned a redirect (excluding 304), 4xx, or 5xx. The status mix bar breaks all responses down: green 2xx, amber 3xx, red 4xx, dark-red 5xx. Recommendations carry a severity — high (red), medium (amber), or info (blue) — reflecting how much budget the fix reclaims.
How it works
- Parsed in your browser. A Web Worker streams the file chunk by chunk and parses each line — nginx/Apache combined & common, IIS/W3C Extended, and JSON (including Cloudflare Logpush). The file is never uploaded.
- Every request classified. The user-agent is matched to a known crawler, then the source IP is checked against that operator's published ranges — the same weekly-refreshed lists the Googlebot Verifier uses. A crawler UA from an IP outside its ranges is flagged as a likely spoofer.
- Aggregated, not stored. Requests roll up into crawl budget, status-code waste, section and AI-vs-search breakdowns, and a spoofer report. Close the tab and it's gone.
- Optional spot-checks. For a suspicious IP you can run a full forward-confirmed reverse-DNS check and ASN lookup — the only step that contacts a server, rate-limited, one IP at a time.
Crawler IP ranges were last refreshed 34 days ago.
Features
- Auto-detects nginx/Apache combined & common, IIS/W3C Extended, and JSON (Cloudflare Logpush) logs; multiple files at once.
- Crawl budget by bot with verified/spoofed/unverifiable counts, unique IPs, status mix, and waste percentage.
- AI-vs-search breakdown so you can see how much of your budget AI crawlers now consume versus traditional search.
- Spoofer report listing crawler-UA requests from out-of-range IPs, with a live per-IP reverse-DNS + ASN verify button.
- Crawl budget by section and a most-wasted-URLs table pinpointing the highest-leverage fixes.
- Daily crawl-over-time trend chart when the log carries timestamps.
- Ranked recommendations with copy-paste robots.txt lines, plus CSV export of bots and wasted URLs.
Limitations
It analyses what's in the log — it can't see requests your server never logged, and it can't tell you why a crawler chose a URL. Verification is only as good as the operators' published ranges: crawlers with no published ranges are marked unverifiable rather than judged, and a determined impostor using a genuine-crawler IP won't be caught by list matching alone (use the reverse-DNS spot-check for those). Sections are inferred from URL paths, so your site's structure determines how clean the grouping is. It reports crawl behaviour, not indexing outcomes — pair it with Search Console to connect a crawled URL to whether it actually ranks.
Frequently asked questions
Are my log files uploaded anywhere?
No. Parsing and analysis run entirely in your browser through a Web Worker — the file is read from disk and streamed in chunks locally, and nothing is sent to a server. The one exception is the optional per-IP spot-check in the spoofer report, which sends a single IP address (not your log) to the verifier API to run a forward-confirmed reverse-DNS and ASN lookup. Close the tab and all parsed data is gone.
What log formats does it support?
The format is auto-detected. It reads nginx and Apache combined and common formats, IIS / W3C Extended, and JSON logs including Cloudflare Logpush. You can drop multiple files at once, and large logs stream in chunks so a multi-gigabyte access log will not lock up the tab.
How does it tell a real crawler from a spoofed one?
Each request is matched to a known crawler by user-agent, then the source IP is checked against that operator’s published IP ranges — the same weekly-refreshed lists the Googlebot Verifier uses. A request that sends, say, a Googlebot user-agent from an IP outside Google’s ranges is flagged as a likely spoofer. Some operators (like ByteDance’s Bytespider) publish no ranges, so those hits are marked unverifiable rather than spoofed.
What counts as wasted crawl budget?
Any crawler request that gets a redirect (except a useful 304 Not Modified revalidation), a 4xx error such as 404 or 410, or a 5xx server error. The tool rolls waste up per bot, per section, and per individual URL so you can see where the highest-leverage fixes are.
Can I block spoofers with robots.txt?
No. Impostors send a crawler user-agent precisely so they can ignore robots.txt, so a Disallow line does nothing to them — block them by IP or ASN at the edge or firewall instead. robots.txt is the right tool for genuine crawlers wasting budget on sections you don’t want crawled, and the Recommendations panel generates copy-paste Disallow lines for exactly those cases.
Feature requests for Log File Analyzer
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
Sobre a ferramenta
Grátis, sem cadastro. Servidor logs answer um question seu analytics can't: o que são rastreadores actually doing em seu site. Drop um em e see rastreamento budget por bot e por seção, como much é wasted em 404s e
um crawl-budget picture: who's rastreamento, o que they're wasting budget em, e who's faking um rastreador
Recursos
- Rastreamento budget por bot com verificado/falsificada/unverifiable counts, unique IPs, status mix, e waste percentage.
- AI-vs-search breakdown so você pode see como much de seu budget AI rastreadores now consume versus traditional pesquisa.
- Exportar um access log de seu servidor, CDN, ou Cloudflare Logpush. Comum formats
- Rastreadores são grouped por tipo , e cada group é colour-coded:
Como funciona
determines como clean o grouping é. Isso relatórios rastreamento behaviour, não indexação outcomes — pair isso
Limitações
- A análise mostra o que está no log — não enxerga solicitações que o servidor nunca registrou e não informa se uma URL foi indexada. Use o painel de recomendações e o Search Console para complementar os resultados.
- publishes Não IP intervalos para verificação against (e.g. Bytespider), so genuine e fake can't ser
Perguntas frequentes
Meus arquivos de log ão enviados para algum lugar?
não. análise e analysis executar inteiramente em seu navegador por meio de a web Worker — o arquivo é leitura de disk e streamdo em blocos localmente, e nada é enviado para a servidor. o um excepção é o opcional per-IP spot-check em o falsificador relatório, qual sends a único IP address (não seu log) para o verifier API para executar a forward-confirmdo reverse-DNS e ASN lookup. fechar o aba e todo analisado dados é desaparece.
What log formatos faz ele suporte?
o formato é auto-detectdo. ele lê nginx e Apache combinado e comum formatos, IIS / W3C Extenddo, e JSON logs incluindo Cloudflare Logpush. você pode drop múltiplos arquivos em once, e grande logs transmitir em blocos so a multi-gigabyte acesso log vai não bloquear up o aba.
Como ele tell a real crawler de a falsificado um?
cada solicitação é correspondido para a conhecido crawler por user-agent, então o fonte IP é checkdo contra que operador’ publicado IP intervalos — o mesmo weekly-refreshdo lists o Googlebot Verifier usa. A solicitação que sends, say, a Googlebot user-agent de um IP fora Google’ intervalos é flaggdo como a provável falsificador. Some operadores (like ByteDance’ Bytespider) publicar não intervalos, so esses hits are marcado não verificável em vez do que falsificado.
O que conta como ouçamento de rastreamento desperdiçado?
qualquer crawler solicitação que gets a redirecionamento (except a útil 304 não Modifido revalidação), a 4xx erro such como 404 ou 410, ou a 5xx servidor erro. o ferramenta rolls waste up por bot, por seção, e por individual URL so você pode ver onde o highest-leverage fixes are.
pode I block spoofers com robots.txt?
não. Impostors send a crawler user-agent precisemente so eles pode ignore robots.txt, so a Disallow linha faz nada para eles — block eles por IP ou ASN em o edge ou firewall em vez disso. robots.txt é o right ferramenta para genuine rastreadores wastndo orçamento em sections você don’t want crawldo, e o Recommendations panel generates copy-paste Disallow lines para exatamente esses cases.