What is a Googlebot?

Googlebot is the generic name for the two web crawlers used by Google Search: Googlebot Smartphone and Googlebot Desktop. These crawlers discover and fetch pages so Google can consider them for indexing and, potentially, show them in search results (Googlebot | Google Search Central; How Google Search works).

What Googlebot does

Google Search operates in three stages: crawling, indexing, and serving. Googlebot performs the crawling stage, discovering URLs and fetching content to be processed. Google then renders pages (including running JavaScript using a recent version of Chrome) and analyzes content for indexing. Not every discovered URL is crawled or indexed, and Google does not guarantee crawling, indexing, or serving (How Google Search works).

Smartphone vs Desktop crawlers

• Googlebot Smartphone simulates a user on a mobile device; Googlebot Desktop simulates a desktop user (Googlebot | Google Search Central).
• For most sites, Google primarily indexes the mobile version of content, so most crawl requests come from the mobile crawler, with a minority from the desktop crawler (Googlebot | Google Search Central).
• Both crawlers obey the same user-agent token in robots.txt, so you cannot selectively target Smartphone vs Desktop via robots.txt (Googlebot | Google Search Central).

Crawl behavior and limits

• Typical rate: for most sites, Googlebot should not access the site more than once every few seconds on average, though short bursts can appear higher due to delays. If your server struggles, you can reduce the crawl rate (Googlebot | Google Search Central).
• File size limits: when crawling for Google Search, Googlebot fetches the first 2 MB of a supported file type, and the first 64 MB of a PDF. Each linked resource (CSS, JS, etc.) is fetched separately and subject to the same per-file limit (except PDFs). Once the limit is reached, the fetch stops and only the downloaded portion is considered for indexing; limits apply to uncompressed data. Other Google crawlers (for example, Image or Video) may have different limits (Googlebot | Google Search Central).

Controlling crawling vs indexing

There is an important difference between crawling and indexing. Blocking Googlebot from crawling a page does not necessarily prevent the page’s URL from appearing in results. To manage behavior (Googlebot | Google Search Central):

• Prevent crawling: use robots.txt.
• Prevent indexing: use a noindex directive.
• Prevent all access: use methods like password protection.

robots.txt examples for Googlebot

Both Smartphone and Desktop obey the same token, so rules apply to both.

# Block Googlebot from all crawling
User-agent: Googlebot
Disallow: /

# Block Googlebot from a folder (example)
User-agent: Googlebot
Disallow: /private/

Blocking Googlebot affects Google Search (including Discover and Search features) and products such as Images, Video, and News (Googlebot | Google Search Central).

Verifying real Googlebot requests

A user-agent string alone is not verification. Google notes that Googlebot’s HTTP user-agent header is often spoofed by other crawlers. The recommended approaches are reverse DNS lookup on the source IP of the request or matching the source IP against Googlebot IP ranges (Googlebot | Google Search Central).

Verification examples

# Example: reverse DNS lookup of a source IP from your logs
# Replace 203.0.113.10 with the actual source IP you observed
$ dig -x 203.0.113.10 +short

# Example: compare the source IP to published Googlebot IP ranges
# (Obtain the official ranges from Google and check membership)
$ ipcalc or your firewall/WAF IP set tooling

Only proceed with blocking or throttling after verification.

How Googlebot differs from other agents

• Googlebot is specifically the crawler for Google Search (How Google Search works).
• Google also runs other crawlers, such as Googlebot Image and Googlebot Video, which may have different limits (Googlebot | Google Search Central).
• If you see non-Search agents in your logs (for example names you may encounter like “Google-Extended”), do not assume they are Googlebot for Search. The official Googlebot documentation cited here does not describe those agents. Treat them separately and verify access according to their own published guidance, if any.

Important limits and expectations

• Crawling by Googlebot does not guarantee indexing or appearance in search results (How Google Search works).
• These documents do not make any claim about AI citations. Crawling by Googlebot for Search should not be interpreted as a guarantee of inclusion in any AI features.

Checklist: set up, control and verify Googlebot

• Confirm that pages you want indexed are not blocked by robots.txt and do not rely on crawling blocks to prevent indexing.
• Serve complete content within the first 2 MB of HTML and ensure critical resources load, given per-file fetch limits (Googlebot | Google Search Central).
• Expect mostly Smartphone crawler requests; you cannot target Smartphone vs Desktop separately in robots.txt (Googlebot | Google Search Central).
• If server load spikes, consider reducing Googlebot’s crawl rate rather than blocking entirely (Googlebot | Google Search Central).
• Verify any suspicious traffic as real Googlebot via reverse DNS or IP range checks before taking action (Googlebot | Google Search Central).

Related reading on My AI Search Report

• How to verify search and AI crawler access without exposing private pages
• What is OAI-SearchBot?
• How accurate are AI citations?

FAQ

Does Google accept payment to crawl more often or rank higher?

No. Google states it doesn’t accept payment to crawl a site more frequently or to rank it higher (How Google Search works).

If I block Googlebot, will my pages disappear from Google?

Blocking crawling can prevent fetching, but the URL might still appear in results. Use noindex to prevent indexing, or authentication to prevent access entirely (Googlebot | Google Search Central).

Why do I still see Googlebot Desktop if indexing is mobile-first?

Google primarily crawls with the Smartphone crawler, but still makes a minority of requests with the Desktop crawler (Googlebot | Google Search Central).

Related guides

• What is Retrieval-Augmented Generation in the context of AI applications?
• Is AEO better than SEO?