Log File Analyzer
Free, no signup. Server logs answer a question your analytics can't: what are crawlers actually doing on your site. Drop one in and see crawl budget by bot and by section, how much is wasted on 404s and redirects, how AI crawlers compare to search engines, and which requests are impostors faking a crawler user-agent.
Handles nginx / Apache (combined & common), IIS / W3C Extended, and JSON logs (incl. Cloudflare Logpush) — format auto-detected. Crawler ranges updated 34 days ago.
Runs entirely in your browser — nothing you paste is uploaded or stored. Spot-checking a suspicious IP against the Googlebot Verifier sends only that IP to our server. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.
Sample report
Say your log contains two lines — a genuine Googlebot hit and a Googlebot impostor:
66.249.66.1 - - [08/Jul/2026:09:14:22 +0000] "GET /blog/technical-seo-guide HTTP/1.1" 200 8213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
193.42.33.7 - - [08/Jul/2026:09:14:41 +0000] "GET /wp-login.php HTTP/1.1" 404 402 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"…and the analyzer returns:
Example data — captured by running this page's own analyzer against the two lines above
| Bot | Type | Requests | Verified | Spoofed | Waste |
|---|---|---|---|---|---|
| Googlebot Google | Search | 2 | 1 | 1 | 50% |
| IP | Claimed to be | Requests | User-agent |
|---|---|---|---|
193.42.33.7 | Googlebot | 1 | Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) |
1 requests spoofed a verified crawler
1 IP sent a search/AI-crawler User-Agent from outside that operator's published ranges — robots.txt won't stop them (impostors ignore it). Block or rate-limit these at the firewall/WAF by IP or ASN. Top offenders: 193.42.33.7 (as Googlebot, ×1). Run each through the Googlebot Verifier for the owning network (ASN) before blocking.
- Both lines roll up under one Googlebot row — the table groups by crawler identity, not by verification status, so the genuine 200 and the spoofed 404 both count toward the same bot's totals. That's why Waste is 50% (1 of 2 requests was non-200), not 0% — see what waste means ↓.
- Verified vs. spoofed is per-request, shown per-IP in the spoofer table below the
bot row:
193.42.33.7isn't in Google's published ranges, so its Googlebot user-agent is flagged rather than trusted. - The recommendation is severity-ranked — a spoofed crawler hitting
/wp-login.phpis high severity becauserobots.txtcan't stop an impostor that ignores it; only a firewall/WAF IP or ASN block does. Confirm the real owner first with the Googlebot Verifier.
How to use it
- Export an access log from your server, CDN, or Cloudflare Logpush. Common formats (nginx/Apache, IIS/W3C, JSON) all work — the format is auto-detected.
- Drop the file onto the upload box, or browse to it. You can drop several files at once; large logs stream in chunks. Nothing is uploaded — parsing runs in your browser. No log handy? Click load demo data.
- Read the summary cards first (requests parsed, crawler share, verified vs. spoofed), then check the llms.txt fetch evidence before working down through crawl budget by bot, the spoofer report, section budget, and the most-wasted URLs.
- Check the Recommendations panel for ranked fixes and copy-paste robots.txt lines. In the spoofer report, hit Verify IP to run a live reverse-DNS + ASN spot-check on any suspicious address.
- Use Export bots CSV or Export wasted URLs CSV to pull the findings into a spreadsheet or ticket.
Site passport Local context for this saved site
Local data
Saved targets, named lists, and recent check summaries remain only in this browser.
Who's crawling you
Requests grouped by crawler type. AI crawlers and search engines want different things from your site. "Other / unclassified" is everything with no known crawler user-agent — human browsers and untracked tools.
Crawl over time
Daily crawl requests by crawler type across the log's date range.
Crawl budget by bot
Each crawler seen in the log: how many requests, how it was verified, and how much of its crawl hit redirects or errors (excluding 304 revalidation).
Spoofer report
Requests that sent a crawler's user-agent from an IP outside that operator's published ranges — likely impostors. robots.txt won't stop these (impostors ignore it); block by IP or ASN. Spot-check any IP for the real network owner via forward-confirmed reverse DNS — this is the one action that leaves your browser (a single lookup to our verifier API).
Crawler × page-type matrix
Where each recognized crawler concentrates requests across inferred page cohorts. Cell intensity represents request count; the label also shows wasted requests.
Crawl budget by section
Where crawlers spend their time on your site (bot traffic only). High waste in a section means crawlers are burning budget on broken or redirecting URLs there.
Most-wasted URLs
Individual URLs where crawlers hit redirects or errors most often (excluding 304 revalidation) — the highest-leverage fixes for crawl budget.
AI-invented URL candidates
Observed 404/410 requests with an AI crawler or known AI-assistant referrer. This is the primary evidence mode: a row is a candidate, not proof that a model invented it — it may be a deleted or migrated URL. Suggested destinations compare only against the sitemap/URL list you supplied above.
Optional URL probe simulation
Ask the site’s open-model simulator for plausible paths, then check their live status. This is explicitly a simulation, never observed assistant output; log evidence above is stronger.
Expected URLs vs crawler requests
Recommendations
Rate this tool
A server access log is the ground truth for how crawlers actually behave on your site — more complete than any report, because it records every single request. This tool turns a raw log into a crawl-budget picture: who's crawling, what they're wasting budget on, and who's faking a crawler identity. For the reasoning behind each metric, see the AI crawler log analysis guide and the crawl budget explainer.
What the results mean
The verification badges on each bot and IP tell you how much to trust a request's identity:
- Verified — the source IP falls inside the crawler operator's published ranges. This is a genuine crawler.
- Spoofed — a crawler user-agent from an IP outside that operator's ranges. A likely impostor; block by IP or ASN.
- Unverifiable — a known crawler whose operator publishes no IP ranges to check against (e.g. Bytespider), so genuine and fake can't be separated by list alone.
Crawlers are grouped by type, and each group is colour-coded:
- Search — traditional search-index crawlers (Googlebot, bingbot).
- AI training — bots that fetch pages to train models (e.g. GPTBot).
- AI search — bots that fetch pages to answer live AI queries (e.g. OAI-SearchBot).
- User-triggered — fetches a person kicked off, such as a citation preview.
- Other / unclassified — no known crawler user-agent: human browsers and untracked tools.
Waste is the share of a bot's, section's, or URL's requests that returned a redirect (excluding 304), 4xx, or 5xx. The status mix bar breaks all responses down: green 2xx, amber 3xx, red 4xx, dark-red 5xx. Recommendations carry a severity — high (red), medium (amber), or info (blue) — reflecting how much budget the fix reclaims.
How it works
- Parsed in your browser. A Web Worker streams the file chunk by chunk and parses each line — nginx/Apache combined & common, IIS/W3C Extended, and JSON (including Cloudflare Logpush). The file is never uploaded.
- Every request classified. The user-agent is matched to a known crawler, then the source IP is checked against that operator's published ranges — the same weekly-refreshed lists the Googlebot Verifier uses. A crawler UA from an IP outside its ranges is flagged as a likely spoofer.
- Aggregated, not stored. Requests roll up into crawl budget, status-code waste, section and AI-vs-search breakdowns, and a spoofer report. Close the tab and it's gone.
- Optional spot-checks. For a suspicious IP you can run a full forward-confirmed reverse-DNS check and ASN lookup — the only step that contacts a server, rate-limited, one IP at a time.
Crawler IP ranges were last refreshed 34 days ago.
Features
- Auto-detects nginx/Apache combined & common, IIS/W3C Extended, and JSON (Cloudflare Logpush) logs; multiple files at once.
- Crawl budget by bot with verified/spoofed/unverifiable counts, unique IPs, status mix, and waste percentage.
- AI-vs-search breakdown so you can see how much of your budget AI crawlers now consume versus traditional search.
- Spoofer report listing crawler-UA requests from out-of-range IPs, with a live per-IP reverse-DNS + ASN verify button.
- Crawl budget by section and a most-wasted-URLs table pinpointing the highest-leverage fixes.
- Daily crawl-over-time trend chart when the log carries timestamps.
- Ranked recommendations with copy-paste robots.txt lines, plus CSV export of bots and wasted URLs.
Limitations
It analyses what's in the log — it can't see requests your server never logged, and it can't tell you why a crawler chose a URL. Verification is only as good as the operators' published ranges: crawlers with no published ranges are marked unverifiable rather than judged, and a determined impostor using a genuine-crawler IP won't be caught by list matching alone (use the reverse-DNS spot-check for those). Sections are inferred from URL paths, so your site's structure determines how clean the grouping is. It reports crawl behaviour, not indexing outcomes — pair it with Search Console to connect a crawled URL to whether it actually ranks.
Frequently asked questions
Are my log files uploaded anywhere?
No. Parsing and analysis run entirely in your browser through a Web Worker — the file is read from disk and streamed in chunks locally, and nothing is sent to a server. The one exception is the optional per-IP spot-check in the spoofer report, which sends a single IP address (not your log) to the verifier API to run a forward-confirmed reverse-DNS and ASN lookup. Close the tab and all parsed data is gone.
What log formats does it support?
The format is auto-detected. It reads nginx and Apache combined and common formats, IIS / W3C Extended, and JSON logs including Cloudflare Logpush. You can drop multiple files at once, and large logs stream in chunks so a multi-gigabyte access log will not lock up the tab.
How does it tell a real crawler from a spoofed one?
Each request is matched to a known crawler by user-agent, then the source IP is checked against that operator’s published IP ranges — the same weekly-refreshed lists the Googlebot Verifier uses. A request that sends, say, a Googlebot user-agent from an IP outside Google’s ranges is flagged as a likely spoofer. Some operators (like ByteDance’s Bytespider) publish no ranges, so those hits are marked unverifiable rather than spoofed.
What counts as wasted crawl budget?
Any crawler request that gets a redirect (except a useful 304 Not Modified revalidation), a 4xx error such as 404 or 410, or a 5xx server error. The tool rolls waste up per bot, per section, and per individual URL so you can see where the highest-leverage fixes are.
Can I block spoofers with robots.txt?
No. Impostors send a crawler user-agent precisely so they can ignore robots.txt, so a Disallow line does nothing to them — block them by IP or ASN at the edge or firewall instead. robots.txt is the right tool for genuine crawlers wasting budget on sections you don’t want crawled, and the Recommendations panel generates copy-paste Disallow lines for exactly those cases.
Feature requests for Log File Analyzer
Upvote what you want most. New ideas can be submitted from the floating Feedback menu; requests appear here once approved, and the most-wanted rise to the top.
You won't be emailed about that request anymore.
Loading…
➕ Request a feature
New requests are reviewed before they appear here.
À propos de l’outil
Importez un journal d’accès serveur pour voir comment les robots explorent réellement votre site : URLs demandées, codes d’état, groupes de robots, fréquence, octets et indices de vérification.
Fonctionnalités
- Analyse des formats de journaux courants avec regroupement par robot, URL, statut et période.
- Vérification prudente des user-agents et des plages publiées lorsque les preuves le permettent.
- Repérage des chemins coûteux, erreurs, redirections, llms.txt et comportements d’exploration atypiques.
- Résultats exportables avec séparation nette entre comportement de crawl et résultat d’indexation.
Fonctionnement
L’outil parse les lignes, normalise horodatage, méthode, URL, code et user-agent, puis regroupe les requêtes par robot et chemin. Il rapproche les preuves de vérification connues sans transformer un nom de robot en identité authentifiée.
Limites
- Un journal montre ce que votre serveur a reçu ; il ne prouve pas à lui seul l’indexation, le classement ou l’intention d’un opérateur.
- Un user-agent peut être usurpé et une plage publiée peut évoluer ; les constats restent bornés par la preuve disponible.
- Les formats incomplets, les proxys et l’échantillonnage peuvent masquer une partie du trafic réel.
Questions fréquentes
Un journal prouve-t-il que Googlebot est authentique ?
Non. Le user-agent seul est falsifiable. Lorsque la preuve est disponible, vérifiez aussi les plages publiées et le DNS inverse avant de conclure.
Le contrôle mesure-t-il l’indexation ?
Non. Il décrit le comportement observé par le serveur. Associez-le à des données Search Console et à des contrôles d’indexation pour étudier les résultats.
Pourquoi regrouper les robots ?
Le regroupement rend visibles fréquence, statuts et chemins par famille de robots. Vous pouvez alors distinguer exploration utile, erreurs répétées et trafic à examiner.
Les adresses IP sont-elles conservées ?
Le fichier sert à cette analyse et les garde-fous de confidentialité de l’outil limitent les données stockées et les compteurs agrégés.