Crawling
How Googlebot discovers your pages by following links across the web.
10 articles in this series
How Web Crawlers Work: Seeds, URL Frontiers & Crawl Rate
A web crawler is a program that discovers pages on the web by fetching URLs, reading their HTML, extracting links, and adding those new links to a queue of pages to visit next. Tha...
Crawl Strategies Explained: Breadth-First, Depth-First, and Focused Crawling
A web crawler's strategy determines which pages it discovers and in what order. The three foundational approaches (breadth-first, depth-first, and focused crawling) produce radically different outcomes from the exact same seed URL....
URL Discovery Explained: How Googlebot Finds Pages Through Links, Sitemaps, and Search Console
There is no central registry of web pages. Googlebot must continuously search for new and updated URLs on its own, using a process Google calls "URL discovery." There are three pathways into Googlebot's crawl frontier: following...
Crawl Budget Explained: Rate Limit, Crawl Demand, and What Wastes It
Crawl budget is the number of URLs Googlebot can and wants to crawl on your site within a given period. It is not a fixed number, and it is not a setting you can directly configure. It emerges from the interaction of two forces: how...
The robots.txt Protocol Explained: History, Syntax, Logic, and Real-World Traps
robots.txt is a plain text file at the root of a domain that instructs web crawlers which paths they are permitted to fetch. Proposed by Dutch software engineer Martijn Koster in February 1994 and refined into an IETF standard 28...
XML Sitemaps Explained: Schema, What to Include, What to Exclude, and Submission
An XML sitemap is a structured file that lists the URLs you want search engines to consider for crawling and indexing. It is a declaration of intent, not a command. Google's own documentation is explicit on this point: submitting a...
Near-Duplicate Detection Explained: Hashing, Shingling, and Canonical Consolidation
Search engines cannot afford to store or rank multiple copies of the same content. By some estimates, as many as 40 percent of pages on the web are duplicates or near-duplicates of other pages. Crawlers solve this at scale using two...
JavaScript SEO Explained: Googlebot's Two-Phase Crawl, SSR, and Dynamic Rendering
Googlebot can execute JavaScript. That fact alone has misled more development teams than almost any other statement in SEO. The ability to render is not the same as reliable, timely indexing. Googlebot crawls and renders in two...
Internal Link Architecture Explained: Hub-and-Spoke, Link Depth, and PageRank Flow
Site architecture is the mechanism by which a website distributes two resources that are always finite: PageRank and crawl budget. Every internal link is both a crawl pathway and an authority transfer. The hub-and-spoke model...
Diagnosing Crawl Problems: A Complete Audit Workflow Using Search Console, Log Files, and Third-Party Tools
Crawl problems are invisible from the outside. A site can look functional to users while Googlebot is silently wasting budget on redirect chains, failing to reach important pages buried six clicks deep, or crawling JavaScript shells...
