What pages should not be indexed?

Pages that should not be indexed are those that either shouldn’t be publicly discoverable or add little value in search: private or confidential areas, internal utilities (like login or cart), staging/test copies, and low-value duplicates or internal search results. Use authentication for truly private content, the noindex rule for public-but-not-for-search pages, and consolidation for duplicates. Don’t rely on robots.txt to hide content from Google, because disallowed URLs can still appear in results if linked elsewhere (Google robots.txt guide).

Make purpose-led decisions

Private or confidential pages (accounts, staging, internal dashboards)

Protect with authentication. Password-protected content won’t be crawled and will be removed from search over time if it had been indexed (Control what you share with Search). Do not expose secrets and rely on robots.txt; it’s not a security control and disallowed URLs can still be indexed if discovered via links (robots.txt guide).

Public utility pages not meant for search (login, cart, checkout status, print views)

Keep them accessible to users but add a noindex directive so they don’t appear in search results. You can add <meta name="robots" content="noindex"> in the HTML head, or return an X-Robots-Tag: noindex header for non-HTML assets like PDFs (Block indexing with noindex). Ensure these pages are not blocked by robots.txt; Google must crawl a page to see the noindex rule (noindex doc).

Internal search results and thin/low-value listings

If your site generates many near-duplicate or low-value internal search pages, consider preventing them from appearing in Search (for example by using noindex on those result pages) so Google focuses on your important content (Control what you share). Avoid blanket noindex on core commercial category pages.

Duplicates and filter variants

When multiple URLs show substantially the same content (sorting, trivial filters, session parameters), prefer consolidation rather than mass noindex. Focus crawling on your primary versions and treat the rest as duplicates to keep Google’s attention on the main pages (Control what you share). Avoid harmful bulk changes that remove valuable landing pages from the index.

Images and videos you don’t want in Search

You can prevent media files from appearing in Google Search results by blocking them from crawling in robots.txt (robots.txt guide). For individual non-HTML files, you can also return X-Robots-Tag: noindex if you want to target specific assets (noindex doc).

Choosing the right control

Page or file type	Your goal	Recommended control	Why
Account areas, admin, staging	Keep private, not publicly discoverable	Authentication (password protection)	Prevents access and indexing; robots.txt is not a security control (share control, robots.txt guide).
Login, cart, internal search pages	Publicly accessible but not useful in search	noindex (meta or HTTP header)	Removes URLs from results; must be crawlable to take effect (noindex doc).
Duplicate/filter variants	Keep one primary in Search	Consolidate to primary; avoid blanket noindex on commercial landing pages	Helps Google focus on important content and reduces duplicate exposure (share control).
Images or videos to keep out of Search	Exclude media from appearing	robots.txt block for media; or X-Robots-Tag: noindex per file	Google only indexes media it can crawl; header-based noindex works per asset (robots.txt guide, noindex doc).

Implementation snippets

HTML page

<head>
  <meta name="robots" content="noindex">
  <title>Sign in</title>
</head>

Non-HTML file or server-wide rule

HTTP/1.1 200 OK
Content-Type: application/pdf
X-Robots-Tag: noindex

Note: Don’t block these URLs in robots.txt if you need Google to see the noindex; Google must crawl the URL to process the directive (noindex doc).

Verification workflow

1. Confirm access: ensure the URL is crawlable if using noindex; do not disallow it in robots.txt (robots.txt guide).
2. Deploy the tag or header and test the live URL.
3. Request recrawl in Search Console’s URL Inspection to speed up processing (noindex doc).
4. Monitor the Page Indexing report to verify that Google extracted the noindex (noindex doc).
5. For emergencies, use the Search Console removals tool while permanent controls are in place (noindex doc).

Need to check crawler access without exposing private pages? See How to verify search and AI crawler access without exposing private pages. For more on why robots.txt isn’t an indexing control, read Can robots.txt block indexing?

Pitfalls to avoid

• Relying on robots.txt to hide pages: disallowed URLs can still appear in Search if linked externally (robots.txt guide).
• Blocking crawl to a URL you just marked noindex: Google won’t see the tag if it can’t crawl (noindex doc).
• Blanket noindex on commercial category or every paginated URL: this can wipe out valuable landing pages. Make case-by-case decisions.

Quick checklist

• Is the content private? Protect it with authentication.
• Is it public but not useful in search? Add noindex (HTML or X-Robots-Tag).
• Is it a duplicate or trivial variant? Consolidate to a primary and avoid mass noindex.
• Is it a media file you want out of Search? Block crawling in robots.txt or set X-Robots-Tag: noindex per file.
• Verify: ensure not disallowed, request recrawl, confirm in Search Console, and use temporary removals if urgent.

FAQs

Can I use robots.txt to remove a page from Google?

No. robots.txt controls crawling, not indexing. Disallowed URLs may still appear if linked elsewhere. Use noindex or authentication instead (robots.txt guide).

Should I noindex my login page?

Yes, if it’s publicly accessible but not intended for searchers. Add a noindex directive and keep it crawlable so Google can see the tag (noindex doc).

What’s the fastest way to get a page out of results?

Apply a permanent control (noindex or authentication), then request recrawl in Search Console. For urgent cases, also submit a temporary removal while your permanent control takes effect (noindex doc).

Related guides

• What does the message "Crawled but not indexed" mean in Google Search Console?