Crawling & Indexierung

B2B shops with login have SEO-specific challenges — most of the catalog and prices sit behind auth, classic crawling logic doesn't apply. SEO concentrates on public funnel pages.

Microsoft's search engine with its own index and crawler (Bingbot) — also powers DuckDuckGo, Yahoo, Ecosia, and ChatGPT Search, making it worth more market share than Bing's direct usage suggests.

Microsoft's free tool for monitoring Bing visibility — index status, crawl errors, keyword reports, and URL submission. The Bing-index counterpart to Google Search Console.

Microsoft Bing's official web crawler, which captures pages for the Bing index and is therefore also indirectly relevant for DuckDuckGo, Yahoo, and Bing-based AI answers.

Search engine from the Brave browser company with its own index — privacy-focused and one of the few fully independent indexes alongside Google, Bing, and Yandex.

Crawl budget is the number of URLs Googlebot crawls on a website within a given time frame — determined by server load and the perceived popularity of the content.

The speed at which a search-engine crawler issues requests to a site — measured in requests per second or per day and automatically adjusted to server response time.

An automated program that systematically visits web pages, reads their content, and follows links — the foundation for search-engine indices, monitoring tools, and AI training data.

Privacy-focused search engine that draws results primarily from the Bing index and enriches them with its own sources (Wikipedia, Apple Maps, the DuckDuckBot crawler).

Faceted navigation refers to filter and sort URLs in shops and listings (color, size, price). It quickly generates millions of crawl URLs — a crawl-budget killer without robots.txt, noindex, or parameter handling.

Free Google tool for monitoring a site's visibility in the Google index — performance reports, index coverage, Core Web Vitals data, and URL inspection. A mandatory SEO tool.

Googlebot is Google's web crawler that discovers, downloads, and processes pages for the search index. Operates in two main variants — Smartphone and Desktop.

Hreflang is an HTML/header attribute telling search engines which language and regional versions of a page exist — crucial for multilingual sites.

Indexing is the process by which a search engine analyzes a crawled URL and adds it to its search index. Crawling ≠ indexing — they are separate steps.

International SEO controls which language and country version of a site ranks in which market. hreflang is the central signal — bidirectional, with an x-default fallback and consistent canonicals per locale.

Paid, ad-free search engine that aggregates results from its own crawlers (Teclis, TinyGem) and API sources — targeted at users willing to pay for ad-free, quality-filtered search.

Crawlers from AI providers that gather web content for training or live answers. Examples — GPTBot, ClaudeBot, PerplexityBot, Google-Extended.

Log file analysis evaluates server logs to see which URLs Googlebot actually crawls, how often, and with which status code. Reveals real crawl behavior — unlike Search Console (aggregated) or synthetic crawlers.

An indexing mode where Google primarily uses the mobile version of a page for crawling, indexing, and ranking. The default for all sites since July 2024.

News SEO is the SEO discipline for news-oriented sites — speed, freshness, news sitemap, NewsArticle schema, and inclusion in Google News and Top Stories are the levers.

Robots directives that control whether a page gets indexed or its links followed. Set via the meta robots tag or the X-Robots-Tag HTTP header.

robots.txt is a text file at a domain's root telling search-engine crawlers which URLs may be crawled and which must not.

SERP stands for Search Engine Results Page — the result page Google, Bing and other search engines return for a given query.

A sitemap is an XML file listing all relevant URLs of a website — it helps search engines crawl and index large sites systematically.

A soft 404 is a page with no real content (out-of-stock product, empty search) that incorrectly returns HTTP 200 instead of 404/410. Google usually detects it and reports it in Search Console.

Spam updates are regular Google algorithm adjustments that detect manipulative SEO tactics and remove or sharply demote affected pages in search results.

A 301 redirect is an HTTP status code permanently redirecting one URL to another — the standard for URL changes, domain moves and site migrations.

HTTP header that sets robots directives (e.g. noindex, nofollow) at the server level — also works for non-HTML resources like PDFs, images, or JSON responses.