Term
Crawler (general)
An automated program that systematically visits web pages, reads their content, and follows links — the foundation for search-engine indices, monitoring tools, and AI training data.
Crawler — explained in more detail
A crawler (also spider or bot) starts from a list of known URLs (seed set), loads the HTML response, extracts links from it, and places them into a crawl queue. Robots protocols (robots.txt, <meta robots>, X-Robots-Tag) and user-agent identification let site operators control the behavior.
Crawlers fall roughly into: search-engine crawlers (Googlebot, Bingbot — build indices), AI crawlers (GPTBot, ClaudeBot, PerplexityBot — collect training and answer data), SEO tools (Screaming Frog, AhrefsBot — audit and competitor analysis), and monitoring/scrapers (uptime checks, commercial data extraction).
Example / practical use
When putting a new site live, you can nudge Googlebot via a Sitemap entry in robots.txt and URL submission through Search Console — otherwise organic discovery often takes days to weeks.
Distinction from related terms
A crawler fetches URLs and follows links; a scraper aims at structured data extraction. Indexer is the downstream stage: the crawler supplies raw data; the indexer turns it into a searchable index.
Entdecke mehr
301 Redirect
A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.
LexikonTechnical SEO — what Google actually has to crawl, render and index
How crawl budget, robots, sitemap, JS rendering, indexing, canonical and Core Web Vitals fit together — the full arc for production sites.
NewsGoogle Search Central Live Toronto 2026 — what Google officially said about the future of search
Information Gain as the new guideline, Google-Extended clarified, AI Mode with 93% zero-click. The key statements from Toronto.