What does the term "crawl budget" refer to in SEO?

Crawl budget is the set of URLs that a search crawler can and wants to crawl on your site during a given period. For Google specifically, it combines two things: your crawl capacity limit (how much Google can fetch without overloading your server) and crawl demand (how much Googlebot wants your URLs, based on factors like popularity and freshness) Google crawl budget guide.

How Google defines crawl budget

Google describes crawl budget as the overlap of two elements Google crawl budget guide:

• Crawl capacity limit (also called hostload): Google adapts how hard it crawls so it doesn’t overwhelm your servers. If your site is fast and stable, capacity can rise; if latency grows, 5xx or 429 errors occur, or robots.txt is unavailable, capacity falls.
• Crawl demand: How much Googlebot needs to recrawl or discover your URLs, influenced by perceived inventory (unique vs duplicate URLs), popularity, and staleness of content. Different Google crawlers (for example AdsBot or Shopping) have their own demand profiles.

In this context, a “site” is a unique hostname, so www.example.com and code.example.com are separate for crawl budget purposes Google crawl budget guide. If you’re new to this topic, you can get foundational context in What is a Googlebot?.

Do you actually need to optimize crawl budget?

Many sites don’t. Google’s documentation says you typically don’t need this level of tuning unless you have a very large or very rapidly changing site, or Search Console shows many URLs as Discovered - currently not indexed. If your pages are crawled promptly after publishing and your sitemaps are current, you probably don’t need further crawl-budget work Google crawl budget guide.

Where to see crawl activity (and what it is not)

Use the Crawl Stats report in Search Console to see Google’s crawling history: request counts, response types, average response time, and availability (robots.txt, DNS, connectivity) Google Crawl Stats report. This report lists requested URLs and example fetches; it does not show which pages are indexed or how often they appeared in search results. For index status, use the Page Indexing report noted in Google’s crawl-budget guidance Google crawl budget guide. For basic verification steps, see How do I check if Google is crawling my site?.

Crawl capacity vs crawl demand: what you can influence

Improving crawl capacity

• Keep the site fast and stable; avoid serving spikes and 5xx/429 responses so Google’s systems can safely increase capacity limits Google crawl budget guide.
• Ensure robots.txt is reliably available; prolonged outages can slow or pause crawling Google Crawl Stats report.
• Support HTTP caching and 304 Not Modified to reduce transfer costs Google crawl budget guide.

Increasing crawl demand (safely)

• Reduce low-value URL variants. Consolidate true duplicates and avoid generating endless sorted/filtered URLs. If you can’t consolidate, you may choose to block unimportant variants in robots.txt to focus crawling on primary URLs Google crawl budget guide. Don’t block important pages.
• Keep sitemaps up to date and, when pages change, include lastmod to signal freshness Google crawl budget guide.
• Avoid long redirect chains and eliminate soft 404s so crawling isn’t wasted Google crawl budget guide.

A worked diagnosis example

Imagine a mid-sized ecommerce site notices fewer new product pages being recrawled quickly. Crawl Stats shows:

• Availability warnings last week tied to spikes in 5xx.
• High shares of requests to parameterized category URLs and repeated redirects.

Plausible next steps grounded in Google’s guidance:

• Stabilize serving so 5xx/429 stop appearing; confirm robots.txt is consistently fetchable Google Crawl Stats report.
• Consolidate or de-duplicate excessive sorted/filter URL patterns; if consolidation isn’t possible, use robots.txt only for unimportant variants so primary categories and products remain crawlable Google crawl budget guide. Avoid blocking key pages.
• Fix redirect chains and keep the product sitemap current with accurate lastmod Google crawl budget guide.

This approach targets both capacity (healthier responses) and demand (cleaner inventory and signals), without assuming a specific page count, crawl rate, or a guaranteed outcome.

Practical, low-risk ways to make crawling more efficient

• Keep your sitemaps complete and fresh; include only canonical, useful URLs.
• Serve fast pages; reduce latency and large resource bottlenecks.
• Return 404/410 for truly removed URLs; don’t keep dead pages returning 200 Google crawl budget guide.
• Use 304 Not Modified when appropriate; embrace HTTP caching Google crawl budget guide.
• Avoid long redirect chains; keep migrations clean Google crawl budget guide.
• Treat robots.txt as a scalpel, not a hammer. Only block unimportant, duplicative URLs; don’t block pages you want processed or potentially indexed Google crawl budget guide. For context, see Can robots.txt block indexing?

Checklist: keep crawl budget working for you

• Confirm pages are crawled soon after publishing; if yes, no extra optimization is usually needed Google crawl budget guide.
• Review Crawl Stats for spikes in 5xx/429, robots.txt fetch failures, or DNS/connectivity issues Google Crawl Stats report.
• Audit URL patterns: consolidate duplicates; restrict only unimportant variants.
• Refresh sitemaps and lastmod for updated content.
• Eliminate soft 404s and long redirect chains.
• Improve page load and TTFB; support 304 responses.

FAQ

Can I ask Google for more crawl budget?

You can’t directly set a higher crawl rate. In practice, capacity rises as your site serves stable, fast responses, and demand rises when your content is unique, valuable and popular for the Google surface you target. If server capacity is the bottleneck, adding resources can help Google crawl budget guide Google Crawl Stats report.

Why did my crawling spike?

Spikes often follow unblocking or adding large sections, or creating many new Dynamic Search Ads targets; identify the crawler type in Crawl Stats to confirm. If overloading occurs, address server capacity and narrow ad targets if applicable Google Crawl Stats report.