Main area
Agency
technisches-seo

Technical SEO — what Google actually has to crawl, render and index

Redaktion ·

What this is about — and why technical SEO often fails invisibly

Technical SEO is the layer nobody sees when it works — and the one that breaks everything when it goes silent. A site can have outstanding content, solid backlinks and clean UX — if Googlebot gets stuck three clicks deep, blows its rendering budget on a single-page app, or the mobile layout shifts on first scroll, the pages still won’t make it to the top. Or into the index at all.

The topic is broad, but it has a clear order: find → crawl → render → index → evaluate. Each step has its own pitfalls, each has its own budget, and at every transition, pages that could have ranked drop out. This article walks you through the five layers — as a map you can use to locate the levers on your own site.

Layer 1 — Find: sitemap, internal links, discovery

Before anything can be indexed, Google has to know the URL exists. Three paths get it there: internal linking, external backlinks, and the XML sitemap. Sitemap and internal links are the two levers you actually control.

Sitemap reality. A sitemap is not an indexing command, it’s a suggestion — “here are all the URLs I consider worth indexing.” Google reads it, compares it to its own discovery model and decides for itself. Meaning: URLs that are only reachable via the sitemap, with no internal links anywhere, usually rank weakly. A sitemap doesn’t replace a link structure, it complements one.

Internal links — the underrated currency. A page two clicks from the homepage is crawled more often and weighted more strongly than one five clicks deep. Click depth is not an official ranking signal, but it correlates very strongly with crawl frequency and indexing probability. Rule of thumb: important pages ≤ 3 clicks from the homepage, never deeper than 5.

Orphan pages. Pages that show up in the sitemap but receive zero internal links are outsiders from Google’s perspective. Some get indexed, many don’t. Anyone regularly seeing “Discovered, currently not indexed” in Search Console often has exactly this problem.

Layer 2 — Crawl: robots, crawl budget, crawl rate

Once Google knows a URL, it sends a crawler. On large sites this quickly becomes a volume problem — and that’s where the crawl budget kicks in.

Understanding crawl budget. Google assigns every site an implicit budget — how many URLs the bot fetches per day. The budget comes from two factors: crawl rate limit (what the server can handle without choking) and crawl demand (how valuable Google considers the site). Small sites under 1,000 URLs never see this — Google just crawls everything. From 10,000+ URLs, especially with dynamically generated filter/sort pages, budget gets tight.

What burns budget.

  • Endless filter URLs. /products?color=blue&size=m&sort=price — if every combination produces its own URL, Google suddenly has millions of URLs to crawl that all show the same inventory.
  • Pagination without end. Pages 1 to 4,000 with ten products each — the bot dutifully crawls them and never gets to the actually important detail pages.
  • Soft 404s and redirect chains. Every redirect costs a crawl. Three chained redirects are three wasted requests.
  • Slow response times. When the server stalls under load, Google throttles the crawl rate — good for the server, bad for indexing freshness.

Robots.txt — what actually happens. robots.txt is a crawling directive, not an indexing directive. A URL blocked in robots.txt is not crawled — but it can still be indexed if it’s known via external links. Google then shows the “No information is available for this page” notice in the search results. If you want a page out of the index, you need a noindex meta tag or an X-Robots-Tag header — and the page must be crawlable, otherwise the bot can’t read the noindex. Classic beginner mistake: blocking a page in robots.txt and setting noindex — the noindex never fires.

X-Robots-Tag for non-HTML. PDFs, images, feeds — anything that can’t carry an HTML meta tag is controlled via the X-Robots-Tag HTTP header. Classic use case: PDF whitepapers that should remain reachable (for direct links) but shouldn’t show up in Google search — set X-Robots-Tag: noindex, done.

Layer 3 — Render: what Googlebot does with your JavaScript

This is where it gets tricky — and for JavaScript-heavy sites it’s often the most expensive layer. Googlebot crawls in two waves: first the initial HTML, then — if needed — a second wave in which a headless Chrome instance executes the JavaScript and extracts the rendered DOM.

The two-wave reality.

  1. First wave: HTML crawl, immediately. If the content is already in the HTML, it gets indexed straight away.
  2. Second wave: JavaScript rendering. Can take hours to days before Google sends the renderer. In the meantime the page is indexed with the empty HTML skeleton — practically without content.

In concrete terms: a pure client-side rendering site (classic React SPA without SSR) is slower to index than an SSR site, and invisible in the meantime. Google renders it eventually — but not immediately, and not reliably on every crawl.

Three rendering strategies compared:

| Strategy | What Googlebot sees in wave 1 | Indexing speed | Pitfall | |---|---|---|---| | SSR (server-side rendering) | Full HTML | Immediate | Server load, complicated caching | | SSG (static site generation) | Full HTML | Immediate | Build time on large sites | | CSR (client-side rendering) | Empty <div id="app"> | Delayed (wave 2) | Content invisible until render | | Hybrid / hydration | HTML + JS hydration | Immediate (HTML) | Hydration mismatch risk |

What makes the renderer fail. Lazy loading that only triggers on mouse movement. Content that only appears after clicking a tab. Routing that uses pushState but no real server URLs. Cookie banners that hide everything behind them. Googlebot copes with these cases sometimes better, sometimes worse — you can’t rely on it.

Layer 4 — Index: canonical, duplicate content, noindex

The crawler has the page, the renderer has the content — now Google decides whether the URL goes into the index at all.

Canonical — the most important invisible tag. <link rel="canonical"> tells Google: “If you find this page under multiple URLs, this is the correct one.” Classic cases: product pages with filter parameters (?color=red points back to the original), print versions, AMP variants, multiple paths through categories.

What happens when canonical is missing or wrong. Google guesses on its own — and sometimes decides differently than you expected. The “Canonical chosen by Google ≠ user-declared canonical” phenomenon shows up in Search Console as “Duplicate, Google chose different canonical than user.” Common cause: weak content on the desired canonical, so Google prefers the variant with more backlinks.

noindex vs. canonical — don’t confuse them.

  • noindex: “This page does not belong in the index.” Search results: none.
  • canonical: “This page belongs in the index, but under this other URL.” Ranking signals are passed to the canonical.

Setting both at the same time is a contradiction. Google usually follows the noindex. The noindex tag belongs on search result pages, login areas, tag lists with thin content, filter permutations — anything that has to be technically reachable but adds no value to the index.

Duplicate content — usually not a “penalty” problem. Google does not penalize duplicate content — Google sorts it. Meaning: of two identical pages, one ends up in the index, the other drops out or gets merged. The problem starts when the wrong variant lands in the index. Clean canonicals and consistent internal linking solve 90 % of cases.

Layer 5 — Evaluate: mobile-first, Core Web Vitals, TTFB

Once the page is in the index, the evaluation phase begins. Google measures how the page behaves for real users (field data via the Chrome User Experience Report) and complements that with lab data (PageSpeed Insights). The central metric family is the Core Web Vitals.

Mobile-first indexing — the mobile HTML is the original

Since the switch to mobile-first indexing, Google evaluates only the mobile version of a site. The desktop version is ignored. This has consequences many underestimate:

  • Content visible on desktop but hidden behind “read more” buttons on mobile counts as hidden — and may rank weaker.
  • Structured data only emitted on desktop doesn’t count.
  • Internal links that only appear in the desktop navigation (because the mobile hamburger menu is reduced) don’t count toward the internal link structure.

Anyone running a separate m.example.com mobile site (only old setups still do this today) has to live with the fact that that one is the main version — every optimization that only applies to desktop is wasted.

Core Web Vitals — three metrics, three pitfalls

Since 2024, the Core Web Vitals have stabilized on three metrics designed to measure real user experience:

LCP — Largest Contentful Paint. When does the largest content element appear? Target: ≤ 2.5 seconds. Most common causes for poor LCP: slow TTFB (server too slow), render-blocking resources (CSS/JS in <head> without defer), unoptimized hero images (3 MB JPEG without width/height and no modern formats).

CLS — Cumulative Layout Shift. How much do elements jump around during loading? Target: ≤ 0.1. Classic CLS killers: images without width/height attributes, web fonts triggering FOIT/FOUT, dynamically injected banners, cookie consent layers that pop up after 800 ms and push everything down.

INP — Interaction to Next Paint. How quickly does the page react to interactions (click, tap, keyboard)? Target: ≤ 200 ms. INP replaced FID (First Input Delay) in 2024. Common causes for poor INP: heavy JavaScript main thread (long tasks > 50 ms), too many event listeners, hydration that locks up the main thread for a second.

The lever behind all of it — TTFB

Time To First Byte is your server’s response time to the first request. It feeds directly into LCP and determines how quickly the browser can even start rendering. Target: ≤ 600 ms, ideally under 300 ms. If your TTFB is 1.2 seconds, you can optimize the frontend all you want — LCP will never get below 2.5 seconds.

Classic TTFB killers: WordPress without cache, dynamically generated pages without a CDN, database queries on every pageview, missing Cache-Control headers. A well-tuned static site setup (or SSG with a CDN in front) typically sits at 50–150 ms — about 10× better than a typical LAMP setup without tuning.

Practice: three typical sites, three failure clusters

Site A — small B2B site, ~80 pages. Crawl budget irrelevant, JS rendering usually a non-issue (classic CMS). The real levers here: TTFB under 300 ms (caching, CDN), clean canonicals (especially for /de/, /en/ variants), mobile layout without CLS. Core Web Vitals dominate the ranking picture.

Site B — Shopware shop with 12,000 products and filters. Crawl budget becomes the central question. Filter URLs must be excluded via robots.txt or tamed with noindex plus canonical. Pagination needs a strategy (either consistently indexable or consistently canonicalized to page 1). The sitemap has to be current. Render performance is secondary because Shopware renders server-side.

Site C — Next.js SaaS landing page with dashboard. Clear separation of rendering strategies: marketing pages as SSG (immediately indexable), app area as CSR with noindex (doesn’t belong in the index). Hydration performance is the number-one INP killer — anyone not paying attention here ends up with beautiful LCP values and catastrophic INP. The mobile layout must not “jump” during hydration, otherwise CLS.

Pitfalls that show up again and again

  • noindex on URLs blocked in robots.txt. Never read, so never effective.
  • Canonical pointing to a noindex page. Contradictory — Google ignores the canonical.
  • Sitemap with URLs that 404 or 301. Burns crawl budget and counts as a quality signal in the wrong direction.
  • Hreflang loops between DE/EN versions without self-reference. Every language variant must also reference itself via hreflang.
  • CDN that ignores User-Agent and serves Googlebot a cookie consent banner. The bot then sees the banner instead of the content.
  • HTTP/2 server push for CSS — the browser often had the CSS in cache already. Push has been practically dead in Chrome since 2022, some setups still try.
  • Lazy-loaded images without the loading="lazy" attribute. Manual lazy loading via IntersectionObserver is easily missed by the crawler — native loading="lazy" is safe.

Diagnostic tools in the right order

  1. Search Console — coverage report. Shows what’s indexed, what was rejected, and why. The first stop for every “why doesn’t this rank” problem.
  2. URL inspection in Search Console. Live test of a single URL: rendered HTML, screenshot, JavaScript console. Clarifies 80 % of rendering questions.
  3. PageSpeed Insights / Lighthouse. Lab and field data on Core Web Vitals. Field data is what really counts — lab data shows why it’s going wrong.
  4. Logfile analysis. What does Googlebot actually crawl, how often, with what status code? Indispensable on large sites — usually overkill on small ones.
  5. Screaming Frog / Sitebulb / JetOctopus. Crawlers that walk your site the way the bot would. Find broken canonicals, redirect chains, missing hreflang.

Conclusion

Technical SEO is not a single lever, it’s a chain: when one link breaks, everything behind it is lost. The right order — find, crawl, render, index, evaluate — is also the order in which you should diagnose problems. There’s no point fiddling with Core Web Vitals if the page never lands in the index because the canonical points the wrong way. And there’s no point fixing the canonical if Googlebot never crawls the page because it’s six clicks deep.

For most production sites, this prioritization works: first close the indexing gaps in Search Console (layers 1–4), then optimize Core Web Vitals (layer 5). Performance optimization on a site that isn’t in the index is cosmetics in the waiting room.

Anyone using SSR or SSG has solved most of the JavaScript rendering problems automatically — and can focus on crawl budget, canonicals and Core Web Vitals. Anyone running a pure SPA without SSR should ask honestly whether the SEO-relevant areas don’t belong in an SSG layer. Migrating is work, but it’s the only clean solution — everything else stays a waiting game with Google’s second render wave.

Themenuebersicht

Technisches SEOCrawling & IndexierungCore Web VitalsJavaScript SEOPageSpeed & Performance