Main area
Agency
technisches-seo

Log File Analysis for SEO

Redaktion ·

Most SEO tools tell you what Google should crawl. Server logs tell you what Google actually crawled. That’s a fundamental difference. Every request to your server leaves a line in the access log: which IP, which URL, which status code, which user agent, which timestamp. Filter out the Googlebot hits and you hold the only unaltered record of real crawl behavior — unsampled, unaggregated, with no interpretation layer.

What logs reveal that Search Console doesn’t

Google Search Console is useful, but it aggregates and samples. The crawl-stats report shows trends and totals, not individual URLs in full depth. Logs close exactly that gap and answer questions GSC looks past:

  • Which URLs does Googlebot visit, and how often? You see the exact crawl frequency per URL and per directory. Suddenly it’s visible that /blog/ is visited daily but /services/ only every three weeks.
  • Which status codes does the bot get? 200, 301, 404, 500 — per URL. A cluster of 404s or 5xx errors in bot traffic is a direct sign of wasted crawl budget and crawl friction.
  • Where does the budget flow? If Googlebot crawls thousands of parameterized filter URLs (?sort=…&color=…) while important pages rarely come up, you’re burning budget on junk. You only see that in the log.
  • Orphan pages. URLs that appear in the log but aren’t linked internally anywhere — or, conversely, important pages the bot never visits.

In short: logs show reality, GSC shows the summary.

Verify first: is this even Googlebot?

Before you analyze anything, comes the most important and most overlooked step: verify that the hits really come from Googlebot. The user-agent string is worthless as proof — anyone can pose as Googlebot, and scrapers do so en masse. If you analyze unfiltered, you mix fake bots into your crawl statistics and draw false conclusions.

Google states the correct procedure (per Search Central, June 2026): reverse DNS plus forward DNS. In two steps:

  1. Reverse DNS on the request IP: host <IP>. The result must end in googlebot.com, google.com, or googleusercontent.com — e.g. crawl-66-249-66-1.googlebot.com.
  2. Forward DNS on the resolved hostname: host <hostname>. The returned IP must again be the original IP.

If both directions match, it’s genuine Googlebot. If one doesn’t, it’s an impostor and drops out of the analysis. Alternatively, Google publishes machine-readable IP range lists (JSON) against which you can match the IPs directly — the faster route for automated analysis.

What you analyze logs with

Three paths, by size and ambition:

  • Screaming Frog Log File Analyser: the standard tool for most sites. Imports the raw logs, verifies Googlebot, and outputs crawl frequency, status codes, and directory breakdown directly. Low barrier to entry.
  • ELK stack (Elasticsearch, Logstash, Kibana) or comparable pipelines: for large sites with millions of lines per day. More setup effort, but real-time dashboards and arbitrary queries.
  • Custom analysis with scripts (Python/awk): when you have targeted questions and don’t want a tool license. Flexible, but you build the verification and aggregation yourself.

Which tool — secondary. What matters is that you only analyze verified Googlebot traffic.

What to look for specifically

A log analysis without a question is data-staring. The productive questions are:

  • Budget waste: What share of bot requests goes to parameter URLs, filter combinations, session IDs, or pagination depths that shouldn’t rank? That’s the first lever — remove those URLs from the crawl via robots.txt or parameter handling.
  • Undercrawled important pages: Are there money pages or fresh content Googlebot visits rarely or never? Then internal linking is missing or the page sits too deep in the click hierarchy.
  • Error clusters: Where do 404s and 5xx pile up in bot traffic? Every error request is burned budget and a signal to Google that the site is unstable.
  • Crawl frequency vs. freshness: Are frequently updated pages also crawled frequently? If your news section is only visited weekly, fresh content reaches the index late.

Crawl frequency itself hangs on the crawl rate, which Google adjusts dynamically to server load and site value — logs show you where that rate lands. And whether a crawled page ends up in the index is a separate question: crawling is the prerequisite for indexing, but no guarantee. How Googlebot works in general is in the glossary under crawler.

FAQ

What do server logs show that Search Console doesn’t? Logs show every single Googlebot request unsampled: exact URL, status code, timestamp, frequency per directory. GSC aggregates and samples this data. Questions like “which exact URL is crawled how often” or “where does my budget flow” are only answered by the raw logs.

Why isn’t the user agent enough to identify Googlebot? Because the user-agent string is freely spoofable — any scraper can pose as Googlebot. If you analyze unfiltered, fake bots distort your statistics. Verify genuine Googlebot via reverse DNS plus forward DNS or against Google’s published IP ranges.

How do I verify genuine Googlebot? Reverse DNS on the request IP (host <IP>); the hostname must end in googlebot.com, google.com, or googleusercontent.com. Then forward DNS on that hostname — the IP must again be the original one. If both directions match, it’s genuine. Alternatively, match against Google’s IP-range JSONs.

Which tool do I need for log analysis? For most sites the Screaming Frog Log File Analyser is enough. Large sites with millions of lines use ELK stack or similar pipelines. For targeted questions your own Python/awk scripts suffice. More important than the tool is analyzing only verified Googlebot traffic.

Which problems does a log analysis typically uncover? Wasted crawl budget on parameter and filter URLs, important pages Googlebot visits rarely or never, clusters of 404 and 5xx errors in bot traffic, and discrepancies between update and crawl frequency. All findings that GSC and classic crawlers don’t deliver this way.

Themenuebersicht

Technisches SEOCrawling & Indexierung