言語

Log File Analyzer

Free, no signup. Server logs answer a question your analytics can't: what are crawlers actually doing on your site. Drop one in and see crawl budget by bot and by section, how much is wasted on 404s and redirects, how AI crawlers compare to search engines, and which requests are impostors faking a crawler user-agent.

Built by Patrick Stox

Handles nginx / Apache (combined & common), IIS / W3C Extended, and JSON logs (incl. Cloudflare Logpush) — format auto-detected. Crawler ranges updated 34 days ago.

Drop your access log here
or ·
Multiple files allowed. Big logs stream in chunks.

Runs entirely in your browser — nothing you paste is uploaded or stored. Spot-checking a suspicious IP against the Googlebot Verifier sends only that IP to our server. Anonymous run-level outcome counters may be used for aggregate research; URLs, domains, IPs, and identifiers are never included, and no statistic is released below 100 runs.

Feedback
Report a bug

Found something broken in Log File Analyzer? Let us know what happened — this goes straight to a private triage queue, not a public list.

What will be sent 
 No tool inputs, uploads, pasted source, complete results, query parameters, or URL fragments are attached automatically. You can edit or remove the selected passage above. Browser and anti-abuse metadata is processed for spam prevention. 
  
 
  
 
 
  
 
 
      
 
Sample report
 
Say your log contains two lines — a genuine Googlebot hit and a Googlebot impostor:
 
 
66.249.66.1 - - [08/Jul/2026:09:14:22 +0000] "GET /blog/technical-seo-guide HTTP/1.1" 200 8213 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
193.42.33.7 - - [08/Jul/2026:09:14:41 +0000] "GET /wp-login.php HTTP/1.1" 404 402 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)"
 
 
…and the analyzer returns:
 
Example data — captured by running this page's own analyzer against the two lines above
 
 
 
2
requests parsed
all lines parsed
 
2
crawler requests
100% of traffic
 
1
verified
IP in published ranges
 
1
likely spoofed
crawler UA, wrong network
 
 
 
BotTypeRequestsVerifiedSpoofedWaste
  
 Googlebot
Google
 Search 2 1 1 50% 
  
 
 
IPClaimed to beRequestsUser-agent
  
 193.42.33.7 Googlebot 1 Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) 
  
 
 
1 requests spoofed a verified crawler
 
1 IP sent a search/AI-crawler User-Agent from outside that operator's published ranges — robots.txt won't stop them (impostors ignore it). Block or rate-limit these at the firewall/WAF by IP or ASN. Top offenders: 193.42.33.7 (as Googlebot, ×1). Run each through the Googlebot Verifier for the owning network (ASN) before blocking.
 
 
 
     
  • Both lines roll up under one Googlebot row — the table groups by crawler
            identity, not by verification status, so the genuine 200 and the spoofed 404 both count toward
            the same bot's totals. That's why Waste is 50% (1 of 2 requests was non-200), not
            0% — see what waste means ↓.
    •  
  • Verified vs. spoofed is per-request, shown per-IP in the spoofer table below the
            bot row: 193.42.33.7 isn't in Google's published ranges, so its Googlebot user-agent
            is flagged rather than trusted.
    •  
  • The recommendation is severity-ranked — a spoofed crawler hitting
/wp-login.php is high severity because robots.txt can't stop an impostor
            that ignores it; only a firewall/WAF IP or ASN block does. Confirm the real owner first with the
Googlebot Verifier.
    •  
 
 
 
How to use it
 
     
  1. Export an access log from your server, CDN, or Cloudflare Logpush. Common formats
            (nginx/Apache, IIS/W3C, JSON) all work — the format is auto-detected.
    2.  
  2. Drop the file onto the upload box, or browse to it. You can
            drop several files at once; large logs stream in chunks. Nothing is uploaded — parsing runs in
            your browser. No log handy? Click load demo data.
    3.  
  3. Read the summary cards first (requests parsed, crawler share, verified vs.
            spoofed), then check the llms.txt fetch evidence before working down through
            crawl budget by bot, the spoofer report, section budget, and the most-wasted URLs.
    4.  
  4. Check the Recommendations panel for ranked fixes and copy-paste robots.txt
            lines. In the spoofer report, hit Verify IP to run a live reverse-DNS + ASN
            spot-check on any suspicious address.
    5.  
  5. Use Export bots CSV or Export wasted URLs CSV to pull the
            findings into a spreadsheet or ticket.
    6.  
 
 
      
      
 

A server access log is the ground truth for how crawlers actually behave on your site — more
          complete than any report, because it records every single request. This tool turns a raw log into
          a crawl-budget picture: who's crawling, what they're wasting budget on, and who's faking a crawler
          identity. For the reasoning behind each metric, see the
AI crawler log analysis guide and the
crawl budget explainer.

 
What the results mean
 
The verification badges on each bot and IP tell you how much to trust a request's identity:
 
     
  • Verified — the source IP falls inside the crawler
            operator's published ranges. This is a genuine crawler.
    •  
  • Spoofed — a crawler user-agent from an IP
outside that operator's ranges. A likely impostor; block by IP or ASN.
    •  
  • Unverifiable — a known crawler whose operator
            publishes no IP ranges to check against (e.g. Bytespider), so genuine and fake can't be
            separated by list alone.
    •  
 
Crawlers are grouped by type, and each group is colour-coded:
 
     
  • Search — traditional search-index crawlers (Googlebot, bingbot).
    •  
  • AI training — bots that fetch pages to train models (e.g. GPTBot).
    •  
  • AI search — bots that fetch pages to answer live AI queries (e.g. OAI-SearchBot).
    •  
  • User-triggered — fetches a person kicked off, such as a citation preview.
    •  
  • Other / unclassified — no known crawler user-agent: human browsers and untracked tools.
    •  
 
 Waste is the share of a bot's, section's, or URL's requests that returned a
          redirect (excluding 304), 4xx, or 5xx. The status mix bar breaks all responses down: green 2xx, amber 3xx, red 4xx, dark-red 5xx.
          Recommendations carry a severity — high (red), medium (amber),
          or info (blue) — reflecting how much budget the fix reclaims.

 
How it works
 
     
  1. Parsed in your browser. A Web Worker streams the file chunk by chunk and
            parses each line — nginx/Apache combined & common, IIS/W3C Extended, and JSON (including
            Cloudflare Logpush). The file is never uploaded.
    2.  
  2. Every request classified. The user-agent is matched to a known crawler, then
            the source IP is checked against that operator's published ranges — the same weekly-refreshed
            lists the Googlebot Verifier uses. A crawler UA from an IP outside
            its ranges is flagged as a likely spoofer.
    3.  
  3. Aggregated, not stored. Requests roll up into crawl budget, status-code
            waste, section and AI-vs-search breakdowns, and a spoofer report. Close the tab and it's gone.
    4.  
  4. Optional spot-checks. For a suspicious IP you can run a full
            forward-confirmed reverse-DNS check and ASN lookup — the only step that contacts a server,
            rate-limited, one IP at a time.
    5.  
 

Crawler IP ranges were last refreshed
34 days ago.

 
Features
 
     
  • Auto-detects nginx/Apache combined & common, IIS/W3C Extended, and JSON (Cloudflare Logpush) logs; multiple files at once.
    •  
  • Crawl budget by bot with verified/spoofed/unverifiable counts, unique IPs, status mix, and waste percentage.
    •  
  • AI-vs-search breakdown so you can see how much of your budget AI crawlers now consume versus traditional search.
    •  
  • Spoofer report listing crawler-UA requests from out-of-range IPs, with a live per-IP reverse-DNS + ASN verify button.
    •  
  • Crawl budget by section and a most-wasted-URLs table pinpointing the highest-leverage fixes.
    •  
  • Daily crawl-over-time trend chart when the log carries timestamps.
    •  
  • Ranked recommendations with copy-paste robots.txt lines, plus CSV export of bots and wasted URLs.
    •  
 
Limitations
 

It analyses what's in the log — it can't see requests your server never logged, and it can't tell
          you why a crawler chose a URL. Verification is only as good as the operators' published
          ranges: crawlers with no published ranges are marked unverifiable rather than judged, and a
          determined impostor using a genuine-crawler IP won't be caught by list matching alone (use the
          reverse-DNS spot-check for those). Sections are inferred from URL paths, so your site's structure
          determines how clean the grouping is. It reports crawl behaviour, not indexing outcomes — pair it
          with Search Console to connect a crawled URL to whether it actually ranks.

 
Frequently asked questions
 
Are my log files uploaded anywhere?
No. Parsing and analysis run entirely in your browser through a Web Worker — the file is read from disk and streamed in chunks locally, and nothing is sent to a server. The one exception is the optional per-IP spot-check in the spoofer report, which sends a single IP address (not your log) to the verifier API to run a forward-confirmed reverse-DNS and ASN lookup. Close the tab and all parsed data is gone.
What log formats does it support?
The format is auto-detected. It reads nginx and Apache combined and common formats, IIS / W3C Extended, and JSON logs including Cloudflare Logpush. You can drop multiple files at once, and large logs stream in chunks so a multi-gigabyte access log will not lock up the tab.
How does it tell a real crawler from a spoofed one?
Each request is matched to a known crawler by user-agent, then the source IP is checked against that operator’s published IP ranges — the same weekly-refreshed lists the Googlebot Verifier uses. A request that sends, say, a Googlebot user-agent from an IP outside Google’s ranges is flagged as a likely spoofer. Some operators (like ByteDance’s Bytespider) publish no ranges, so those hits are marked unverifiable rather than spoofed.
What counts as wasted crawl budget?
Any crawler request that gets a redirect (except a useful 304 Not Modified revalidation), a 4xx error such as 404 or 410, or a 5xx server error. The tool rolls waste up per bot, per section, and per individual URL so you can see where the highest-leverage fixes are.
Can I block spoofers with robots.txt?
No. Impostors send a crawler user-agent precisely so they can ignore robots.txt, so a Disallow line does nothing to them — block them by IP or ASN at the edge or firewall instead. robots.txt is the right tool for genuine crawlers wasting budget on sections you don’t want crawled, and the Recommendations panel generates copy-paste Disallow lines for exactly those cases.
 
 
 
       
Next steprobots.txt Generator — generate the corrected version.

People also explore
ToolGooglebot VerifierToolrobots.txt TesterToolAI-Crawler Access CheckerArticleUser AgentArticleCrawlingArticleCrawl Budget
 
 
 
Feature requests for Log File Analyzer
 

Upvote what you want most. New ideas can be submitted from the floating Feedback menu;
      requests appear here once approved, and the most-wanted rise to the top.

   
Loading…
  
  ➕ Request a feature   
New requests are reviewed before they appear here.
 
     
  
  
 
 
 
 
     
ツールについて
サーバーアクセスログから、ボットとセクション別のクロールバジェット、ステータスコードの無駄、AI対検索の内訳を確認します。
クローラーのユーザーエージェントを偽装するスプーファーを報告し、nginx、Apache、IIS／W3C、JSONログをブラウザー内で解析します。アップロードは行いません。
機能
  • ボット・セクション別のクロールバジェット
  • ステータスコードの無駄と応答内訳
  • AIクローラーと検索クローラーの比較
  • クローラー偽装のスプーファーレポート
仕組み
ログ形式を判定し、タイムスタンプ、URL、ステータス、ユーザーエージェント、任意のIPを正規化します。クローラー種別とセクションで集計し、無駄なステータス、AI対検索の比率、検証が必要な偽装候補を分けて表示します。
制限事項
  • 結果は入力ログの形式、期間、サンプリング、ユーザーエージェントに依存します。IPや逆引きの証拠がない場合、ボットの身元を保証できません。
  • 解析はログの観測事実を要約するもので、検索順位、クロールの将来、サイト全体のセキュリティを保証しません。
よくある質問
このツールはGoogleの実際のクロールを記録しますか？
いいえ。提供されたアクセスログだけを解析します。ログの欠落、プロキシ、サンプリングにより観測できないリクエストがあります。
ユーザーエージェントだけでボットを検証できますか？
いいえ。ユーザーエージェントは偽装できます。可能な場合はIPレンジや正引き確認など、別の証拠で検証してください。
ログはアップロードされますか？
ブラウザー内での解析を前提にしています。共有・保存する場合はIP、Cookie、URL、識別子などの機密情報を確認してください。