llms.txt: The robots.txt for the AI Era

What is llms.txt?

llms.txt is a markdown file placed at the root of your domain (https://example.com/llms.txt) that gives language model crawlers a curated summary of your site. It tells GPTBot, ClaudeBot, PerplexityBot, and other LLM crawlers what your site is about, which pages matter most, and how to summarize you accurately.

The standard is open and maintained at llmstxt.org. It was created in late 2024 by a group of SEO and AI practitioners who noticed that LLMs often misrepresent sites because they rely on whatever the crawler happens to find first. llms.txt fixes that by giving crawlers an explicit, curated entry point.

How llms.txt differs from robots.txt

robots.txt and llms.txt solve different problems and you need both.

robots.txt controls access. It tells crawlers what they may and may not fetch. It is binary: allow or disallow. It says nothing about what the fetched content means.

llms.txt guides summarization. It assumes the crawler is already allowed in (via robots.txt) and tells it what to prioritize and how to describe the site. It is descriptive, not permission-based.

A typical setup: robots.txt allows GPTBot everywhere, and llms.txt tells GPTBot "this is a SaaS tool for X, the most important pages are Y, and you should describe us as Z." Without llms.txt, GPTBot might describe you based on your homepage meta description, your first blog post, or whatever it crawled first.

The llms.txt format

The llmstxt.org spec is strict and simple. A valid llms.txt file has four parts, in order:

1. An H1 title with your site or company name. 2. A blockquote summary (one to three sentences) describing what you do. 3. Optional markdown paragraphs with additional context. 4. A list of links to your most important pages, each with a one-line description.

That is it. No YAML, no JSON, no frontmatter. Just markdown. The simplicity is intentional — LLMs parse markdown natively, so the format requires no special parser.

• H1: # Your Company Name
• Blockquote: > One to three sentences describing what you do.
• Optional paragraphs of additional context.
• A list of links: - [Page Title](url): One-line description.

Why llms.txt matters for AI citations

LLMs do not read your entire site before answering. They read a snippet — usually the homepage, the top-ranking page for a query, or whatever the crawler fetched most recently. If that snippet misrepresents you, the LLM will misrepresent you in its answers.

llms.txt gives you control over the snippet. You tell the LLM "this is what I am, these are my most authoritative pages." When the LLM later answers a question about your topic, it pulls from the curated representation instead of a random crawl.

The effect is not instant. LLMs re-index on a rolling basis over weeks. But sites that ship llms.txt consistently report cleaner brand mentions and more accurate AI answers within 30-60 days.

How to ship llms.txt in 5 minutes

You do not need to write llms.txt by hand. Use the free llms.txt Generator: enter your homepage URL, and it crawls up to 50 pages, drafts the file, and gives you deploy instructions for your CMS.

If you prefer to write it manually, start with this template and customize the bracketed parts:

# [Your Company Name]

> [One to three sentences describing what you do and for whom.]

## Optional sections

- [Product / Service page](https://example.com/product): [one-line description] - [Pricing](https://example.com/pricing): [one-line description] - [Top blog post](https://example.com/blog/flagship-post): [one-line description]

Save it as llms.txt and upload it to your site root. For Next.js, put it in /public. For WordPress, use a static file plugin or the seosights WordPress plugin which auto-generates it. For Webflow, use a redirect rule.

Common llms.txt mistakes

The most common mistake is treating llms.txt like a sitemap and listing every URL. The spec recommends 10-30 links, not 500. LLMs have context windows; a 100KB llms.txt defeats the purpose.

The second mistake is writing a marketing summary instead of a factual one. LLMs ignore hype. Write what you do, for whom, and what makes you different — in plain language.

The third mistake is forgetting to update llms.txt when you publish flagship content. If you ship a definitive guide, add it to llms.txt the same day. Otherwise the LLM may not discover it for weeks.