Term
Log File Analysis
Log file analysis evaluates server logs to see which URLs Googlebot actually crawls, how often, and with which status code. Reveals real crawl behavior — unlike Search Console (aggregated) or synthetic crawlers.
Log File Analysis — explained in more detail
Server access logs record every single HTTP request to the site — including user agent, URL, status code, timestamp, and bytes transferred. In log analysis, you filter on bot user agents (Googlebot, Bingbot, AI crawlers) with reverse-DNS-verified IPs to see where crawl budget actually goes.
Typical findings: a large share of crawls hits filter/sort URLs (faceted navigation), redirect chains, or 4xx/5xx pages. From there, the actions are concrete: robots.txt rules, redirect cleanup, canonical fixes. Tools: Screaming Frog Log Analyzer, OnCrawl, JetOctopus, Botify, or a simple ELK-stack setup.
Example / In practice
A shop sees “1.2 M crawled URLs” in Search Console, but only 80,000 are indexed. Logs reveal: 70% of Googlebot hits go to sort and pagination URLs that are canonicalized away. Action: robots.txt disallow for sort parameters — crawl budget shifts to indexable product pages, and new products get picked up faster.
Distinction from similar terms
Search Console crawl stats shows aggregates from Google itself, without URL granularity. Site-audit crawlers (Screaming Frog desktop mode) are synthetic — they show what’s crawlable, not what Google actually crawls. Logs are the only source of actual crawl behavior.
Entdecke mehr
301 Redirect
A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.
LexikonTechnical SEO — what Google actually has to crawl, render and index
How crawl budget, robots, sitemap, JS rendering, indexing, canonical and Core Web Vitals fit together — the full arc for production sites.
NewsGoogle Search Central Live Toronto 2026 — what Google officially said about the future of search
Information Gain as the new guideline, Google-Extended clarified, AI Mode with 93% zero-click. The key statements from Toronto.