Clarity before claims.A method you can inspect
My AI Search Report.
A practical guide

What is an llms.txt file?

A clear explanation of the voluntary llms.txt v2 proposal: what it is, how it differs from robots.txt, how to structure it, and how to keep it working.

By My AI Search Report
Four-part editorial diagram: question, evidence, checks and next step

An llms.txt file is a voluntary markdown guide you place on your site to help AI agents find concise, LLM-friendly documentation and links, especially markdown versions of key pages. It’s part of a v2 proposal and is not a permissions or blocking file, and it isn’t a Google requirement for AI Overviews or citations (https://llmstxt.org/, https://developers.google.com/search/docs/appearance/ai-features).

What llms.txt is (v2): a curated entry point for agents

  • Location and scope: The file is named llms.txt and can live at the site root (/llms.txt) or any subpath (for example, /docs/llms.txt). A file covers URLs under its path, and if more than one applies, agents should use the most specific one (https://llmstxt.org/).
  • Purpose: It gives a concise overview, then links to detailed, LLM-friendly resources (ideally markdown pages). The llms.txt stays small enough to fit typical context windows; detail lives behind the links and is fetched as needed (https://llmstxt.org/).
  • Use cases: Common for software docs and any site where agents need a guided path to authoritative content. The v2 page notes real-world adoption and tooling interest, including documentation platforms and audits that look for llms.txt as part of agentic browsing checks (https://llmstxt.org/).

How it works with markdown and link relations

  • Markdown alternates: Provide a clean markdown version of important pages at the same URL with a .md extension (for example, page.html.md or page.md). For paths without a filename, use index.html.md or index.md (https://llmstxt.org/).
  • Link relations: Use rel="alternate" type="text/markdown" on HTML pages to point to their markdown version, and rel="describedby" to point to the relevant llms.txt. These can be added as HTML <link> elements or as HTTP Link: headers, which also work for non-HTML resources (https://llmstxt.org/).
Link: </docs/page.html.md>; rel="alternate"; type="text/markdown", </docs/llms.txt>; rel="describedby"

Minimal llms.txt example

This tiny sample follows the v2 structure: a single H1, a short blockquote summary, optional details, and one or more H2 “file lists” with links.

# Example Project
> Concise summary so agents know what this site covers and where to look next.

Key concepts: authentication, pricing, and API limits are documented in markdown.

## Docs
- [Quick start](https://example.com/docs/quickstart.md) : Fast setup with code samples
- [API reference](https://example.com/docs/api.md) : Endpoints, params, and error codes

## Optional
- [Changelog](https://example.com/docs/changelog.md)

Format details and the required section order are specified in v2 (https://llmstxt.org/).

What llms.txt is not

  • Not a permissions or blocking file: robots.txt controls crawling for Search; llms.txt provides guidance and links for agents. The v2 page contrasts their purposes, and Google’s documentation confirms that robots.txt and existing preview controls (nosnippet, data-nosnippet, max-snippet, noindex) govern what appears in Search AI features (https://llmstxt.org/, https://developers.google.com/search/docs/appearance/ai-features).
  • Not a Google requirement or a citation guarantee: Google states there are no additional technical requirements specific to AI Overviews or AI Mode beyond normal indexing and snippet eligibility. llms.txt isn’t listed as a requirement or ranking signal (https://developers.google.com/search/docs/appearance/ai-features).
  • Not a sitemap replacement: Sitemaps enumerate pages; llms.txt is a curated overview for LLMs and can link to external resources where relevant (https://llmstxt.org/).

Set-up and maintenance checks

  1. Choose scope and placement: Put /llms.txt at the root to cover the whole site, or place /docs/llms.txt to cover only /docs/. Remember: the most specific file should be used (https://llmstxt.org/).
  2. Create markdown alternates: For key pages, publish .md versions at the same URLs with the extension changed or appended as described in v2 (https://llmstxt.org/).
  3. Add link relations: On HTML pages, add rel="alternate" type="text/markdown" to the markdown version, and rel="describedby" to the matching llms.txt. You can also add equivalent HTTP Link: headers at the server or CDN level (https://llmstxt.org/).
  4. Keep llms.txt concise: Summarize, then link. Aim for something that comfortably fits typical model context windows; put larger details in the linked markdown (https://llmstxt.org/).
  5. Verify access without exposing private pages: Ensure the file and linked markdown return 200 for public users and expected crawlers; keep authentication-only material out of scope. See our guidance on safe verification steps: How to verify search and AI crawler access without exposing private pages.
  6. Functional test: The v2 page suggests testing by asking an agent questions with only your llms.txt as a starting point. Check that it follows your links and retrieves the right markdown (https://llmstxt.org/).
  7. Monitor over time: In My AI Search Report, llms.txt is inspected but is informational and not a scored signal; it helps context, but presence alone won’t guarantee citations (https://myaisearchreport.com/method/).

Related reading on our site

Quick checklist

  • Publish /llms.txt (or a scoped /path/llms.txt) with an H1, a short blockquote summary, and H2 sections listing key markdown links.
  • Provide markdown alternates at page.md or page.html.md and reference them with rel="alternate" type="text/markdown".
  • Point pages to their covering llms.txt using rel="describedby" (HTML link or HTTP Link: header).
  • Keep llms.txt compact; move details into linked markdown.
  • Return 200 for llms.txt and linked markdown; avoid private or login-only material.
  • Re-test after site moves, redirects or docs reorganizations; fix broken links promptly.

FAQs

Do I need llms.txt to appear in Google’s AI Overviews or AI Mode?
No. Google lists no additional technical requirements beyond being indexable and eligible for snippets, and it uses robots.txt and preview controls to manage what shows in Search (https://developers.google.com/search/docs/appearance/ai-features).

Is llms.txt used for training or inference?
The v2 page says the expectation and observed use is mainly inference, though training runs could use its information too (https://llmstxt.org/).

Why not use a single /.well-known/ file?
The proposal explains that well-known files exist only at the origin root, whereas llms.txt can live at any path and describe just that subtree, which helps authors who only control a subpath (https://llmstxt.org/).

Related guides

Sources and further reading

Provider documentation can change. Check the current guidance before changing crawler or search settings.

How to fix soft 404 errors?

A practical decision tree and step-by-step workflow to diagnose and fix soft 404s, with examples, correct status codes, and Search Console validation.

Is IndexNow free?

Yes. IndexNow URL notifications are free to send via the public endpoints; any costs you encounter would come from a CMS, plugin or hosting provider’s own features, not the protocol itself.