What Is llms.txt and Should Your Site Have One?

A crawler can reach every page on your site and still miss the three pages that best explain your product. Navigation, campaign pages, duplicate URLs, and JavaScript can obscure the useful path. An llms.txt file gives AI tools a short, curated map written by the person who knows the site.

What is llms.txt, and where did it come from?

llms.txt is a Markdown file published at the root of a site, such as https://example.com/llms.txt. It names the site, summarizes what it does, and links to the pages an AI tool should read for reliable context.

Jeremy Howard of Answer.AI published the original llms.txt proposal on September 3, 2024. The accompanying Answer.AI announcement framed it as a way for site owners to identify useful, model-friendly material when an application is assembling context. The intended use was mainly at inference time, when a tool is answering a question, rather than as a directive for model training.

That distinction matters. llms.txt is a community proposal, not an access-control mechanism or a guaranteed distribution channel. It gives a willing reader a cleaner route through your public material. It cannot make an AI service fetch the file, include your pages, cite your brand, or rank you for a query.

The format is deliberately simple. The only required element is an H1 containing the site or project name. A useful file then adds a blockquote summary, optional context, and H2 sections containing Markdown link lists. The proposal gives an Optional section special meaning: tools may skip those links when they need a shorter context.

What belongs in a useful llms.txt file?

Treat the file as an editorial shortlist, not a second sitemap. Start with the pages that answer basic questions about your company, product, documentation, policies, and source of truth. Link to canonical URLs. If you publish clean Markdown versions of pages, link to those; otherwise, normal HTML URLs are valid.

Here is a realistic example for a small software company:

# Northstar Analytics

> Northstar Analytics is a privacy-first web analytics platform for small product teams. It provides cookieless traffic and conversion reporting.

Use the product documentation for implementation details. The privacy page is the source of truth for data handling.

## Start here

- [Product overview](https://northstar.example/product): Features, supported use cases, and plan limits
- [Documentation](https://northstar.example/docs/index.md): Setup and reporting guides
- [API reference](https://northstar.example/docs/api.md): Authentication, endpoints, and response formats

## Company and policies

- [About](https://northstar.example/about): Company background and team
- [Privacy](https://northstar.example/privacy): Data collection and retention policy
- [Contact](https://northstar.example/contact): Support and business inquiries

## Optional

- [Changelog](https://northstar.example/changelog): Recent product updates
- [Customer stories](https://northstar.example/customers): Selected implementation examples

Good descriptions tell the reader what each page resolves. “API authentication and rate limits” is more useful than “Learn more.” Keep claims consistent with the linked pages, remove retired URLs, and update the file when your canonical documentation changes.

Avoid turning llms.txt into a keyword dump, sales pitch, or copy of every page. Do not include private URLs, unpublished facts, or instructions that contradict your public policies. A compact file with clear ownership and current links is easier for both machines and humans to inspect.

Do AI crawlers actually use llms.txt?

Adoption is uneven as of mid-2026. Some documentation systems and AI tools actively publish or consume these files. Perplexity, for example, exposes an llms.txt index for its own developer documentation. That demonstrates a practical use for agent navigation, but it does not prove that PerplexityBot automatically checks every site's file.

Current public guidance from OpenAI, Anthropic, and Perplexity focuses on crawler user agents, crawl access, and robots.txt. The guidance reviewed for this article does not confirm llms.txt as a ranking input. Major engines may fetch it selectively, use other indexes, rely on search partners, or ignore it.

That makes llms.txt cheap insurance, not a ranking lever. It costs little to publish if your core pages are already organized, and it can help tools that know to request it. Your stronger fundamentals remain crawlable pages, accurate metadata, structured data, clear answers, and consistent trust signals. You can check the file's presence and related crawler access in a free Suede Signal audit.

Server logs can show whether a named bot requested /llms.txt, but one request proves only that the file was fetched. It does not prove ingestion, retrieval for later answers, citation, or ranking impact.

How does it differ from robots.txt and sitemap.xml?

These files can coexist because they answer different questions:

File	Primary job	What it contains
robots.txt	State crawl permissions	User-agent rules with allowed or disallowed paths
sitemap.xml	List canonical URLs for discovery	A machine-readable inventory of pages, and sometimes update metadata
llms.txt	Explain what matters and where to find it	A curated Markdown summary with grouped, described links
llms-full.txt	Offer a complete context bundle	The full public documentation or site text assembled into one Markdown file

robots.txt governs access preferences; it does not explain your company. A sitemap helps crawlers discover URLs; it does not tell them which five pages answer a product question best. llms.txt supplies that editorial context, but it does not override a block in robots.txt or make an inaccessible page readable.

llms-full.txt is a related convention used by documentation platforms such as Mintlify. The short file is an index: a tool chooses which links to retrieve. The full file concatenates the underlying public material so a user or tool can load one large document. It can be useful for focused documentation, but it may become bulky, duplicate stale copy, or exceed a tool's practical context. You do not need it to publish a valid llms.txt.

What to do next

• Choose five to fifteen canonical pages that provide the clearest public account of your product, company, documentation, and policies.
• Publish a concise Markdown file at /llms.txt using one H1, a blockquote summary, grouped H2 sections, and described links.
• Confirm the file returns 200, works without authentication, and points only to live public URLs.
• Review robots.txt separately so the crawlers you want can reach the linked pages.
• Add llms.txt to your release checklist and update it when key URLs or claims change.