Term
Googlebot
Googlebot is Google's web crawler that discovers, downloads, and processes pages for the search index. Operates in two main variants — Smartphone and Desktop.
Googlebot — explained in more detail
Googlebot is the umbrella name for Google’s crawler family. The two production variants are Googlebot Smartphone (the default for mobile-first indexing, used for most sites) and Googlebot Desktop (for sites not yet on mobile-first, plus cross-checks). Specialized crawlers such as Googlebot-Image, Googlebot-Video, and Googlebot-News run alongside.
Identification works via the User-Agent string and — more reliably — via reverse-DNS lookup against *.googlebot.com or *.google.com. Control happens through robots.txt, the meta name="robots" tag, and the X-Robots-Tag HTTP header.
Example / In practice
In server logs, Googlebot looks like this: Mozilla/5.0 (Linux; Android …) … Googlebot/2.1; +http://www.google.com/bot.html. To block the crawler, use robots.txt with User-agent: Googlebot + Disallow:. Note: robots.txt blocks crawling, not indexing — noindex belongs in the meta tag.
Distinction from similar terms
Google-Extended is a separate User-Agent for Google’s AI training and Vertex AI data — it does not affect regular Google Search. To block AI training but stay in the index, block only Google-Extended, not Googlebot.
Entdecke mehr
301 Redirect
A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.
LexikonTechnical SEO — what Google actually has to crawl, render and index
How crawl budget, robots, sitemap, JS rendering, indexing, canonical and Core Web Vitals fit together — the full arc for production sites.
NewsGoogle Search Central Live Toronto 2026 — what Google officially said about the future of search
Information Gain as the new guideline, Google-Extended clarified, AI Mode with 93% zero-click. The key statements from Toronto.