Can robots.txt block indexing?

No. A robots.txt file tells crawlers what they may crawl, but it does not reliably prevent a URL from appearing in search results. Disallowed pages can still be indexed and shown as bare URLs if they’re linked elsewhere. To keep a URL out of Google results, use noindex (and allow Googlebot to fetch it) or require authentication for private content (Google robots.txt guide; Google noindex guide).

What robots.txt actually does

robots.txt is primarily for managing crawler traffic and specifying which URLs a compliant crawler may access. Google states it is not a mechanism for keeping a web page out of Google Search (robots.txt guide). Respectful crawlers like Googlebot follow it, but not all crawlers do, and different crawlers can interpret rules differently (robots.txt guide).

Why disallowed URLs can still show in search

If a page is disallowed in robots.txt, Google may still index the URL when it discovers links to it on the web. In that case, a search result may appear without a description snippet (robots.txt guide). For sensitive information, Google recommends other methods, such as password protection (robots.txt guide).

How to reliably prevent indexing

Use a noindex meta tag

Add a noindex directive to the HTML head to have Google drop the page from results after it’s crawled:

<meta name="robots" content="noindex">

To target only Google’s web crawler:

<meta name="googlebot" content="noindex">

Google notes some other search engines might interpret noindex differently, so results can vary outside Google (noindex guide).

Or send an X-Robots-Tag HTTP header

For non-HTML files (PDFs, images, video) or when modifying headers is easier, return:

HTTP/1.1 200 OK
X-Robots-Tag: noindex

Header and meta implementations have the same effect in Google (noindex guide).

Let crawlers fetch the page to see noindex

Google must crawl the page to detect the noindex tag or header. If robots.txt blocks the URL, Googlebot can’t see the noindex and the page may continue to appear. Remove the robots.txt block for that URL so Googlebot can recrawl and process the noindex. Recrawling can take time depending on page importance; you can request recrawl with Search Console’s URL Inspection tool (noindex guide).

Keep private content private

robots.txt is not a security or privacy control. Some crawlers ignore it, and links from elsewhere can still surface URLs. For confidential material, require authentication (for example, password-protect the resource) rather than relying on robots.txt (robots.txt guide). If you need to verify that a crawler can access a test page without exposing the rest of a site, see our workflow in How to verify search and AI crawler access without exposing private pages.

Practical examples

robots.txt that blocks crawling (but not indexing)

# This tells compliant crawlers not to crawl these paths
User-agent: *
Disallow: /admin/
Disallow: /search?q=

Effect: Google won’t crawl those paths, but their URLs could still be indexed if discovered via links, and may appear without a description (robots.txt guide).

noindex on a page that is allowed to be crawled

<head>
  <meta name="robots" content="noindex">
</head>

Effect: Once Googlebot crawls the page and reads the tag, Google removes it from search regardless of inbound links (noindex guide).

Verification and troubleshooting

• If a noindexed page still appears, it may not have been crawled yet. Google notes this can take months for some pages; request a recrawl in URL Inspection (noindex guide).
• If robots.txt blocks the page, Google can’t see your noindex. Unblock it in robots.txt so the tag/header can be read (noindex guide).
• Use the Page Indexing report in Search Console to confirm that Google detected a noindex rule (noindex guide).
• Learn what Googlebot is and how it behaves in general in What is a Googlebot?

Actionable checklist

• Decide intent: reduce crawl load vs. keep out of results. Use robots.txt for crawl management; use noindex or authentication for exclusion.
• Implement noindex via meta or X-Robots-Tag on each URL you want removed from results.
• Ensure the URL is not disallowed in robots.txt so crawlers can read the noindex.
• For confidential content, require login/password; do not rely on robots.txt.
• Request a recrawl with URL Inspection and monitor Page Indexing for noindex detection.

FAQ

Does Google support noindex in robots.txt?

No. Specifying noindex in robots.txt is not supported by Google. Use a meta tag or X-Robots-Tag header instead (noindex guide).

Can I stop images or videos from appearing with robots.txt?

Yes, you can use robots.txt to prevent image, video, and audio files from appearing in Google Search results, but this does not stop others from linking to those files on the web (robots.txt guide).

What does a disallowed page look like in results?

If Google finds links to a URL that’s disallowed from crawling, the URL can appear in results without a description snippet (robots.txt guide).

A useful next step

For a closer look at page-exclusion and link-following directives, continue with CheckMySchema. Noindex vs nofollow: a clear guide for developers.