llms.txt: The robots.txt for the AI Era

What is llms.txt?

llms.txt is a markdown file placed at the root of your domain — https://example.com/llms.txt — that gives language model crawlers a curated summary of your site. It tells GPTBot, ClaudeBot, PerplexityBot, and other LLM crawlers what your site is about, which pages matter most, and how to summarize you accurately.

The standard is open and maintained at llmstxt.org. It was created in late 2024 by a coalition of SEO and AI practitioners who noticed that LLMs routinely misrepresent sites because they rely on whatever the crawler happens to find first. In a study of 500 brand-mention queries across ChatGPT and Perplexity, 37% of AI-generated descriptions contained at least one factual error about the brand — errors that originated from the crawler reading the wrong page first.

llms.txt fixes this by giving crawlers an explicit, curated entry point. Instead of guessing what your site does from a random HTML page, the LLM reads a single markdown file that you control. Think of it as a press release for robots — a document you write so that AI systems describe you the way you want to be described.

How llms.txt differs from robots.txt

robots.txt and llms.txt solve different problems and you need both.

robots.txt controls access. It tells crawlers what they may and may not fetch. It is binary: Allow or Disallow. It says nothing about what the fetched content means or how the crawler should interpret it. A typical robots.txt line like User-agent: GPTBot followed by Disallow: /admin/ simply blocks GPTBot from your admin pages — it does not tell GPTBot anything about your business.

llms.txt guides summarization. It assumes the crawler is already allowed in (via robots.txt) and tells it what to prioritize and how to describe the site. It is descriptive, not permission-based. Where robots.txt says "you may look here," llms.txt says "when you describe me, say this, and start with these pages."

A typical setup: robots.txt allows GPTBot everywhere, and llms.txt tells GPTBot "this is a SaaS tool for X, the most important pages are Y, and you should describe us as Z." Without llms.txt, GPTBot might describe you based on your homepage meta description, your first blog post, or whatever it crawled first — and that random selection is the root cause of most AI misrepresentations.

The llms.txt format explained

The llmstxt.org spec is strict and simple. A valid llms.txt file has four parts, in order:

1. An H1 title with your site or company name. 2. A blockquote summary (one to three sentences) describing what you do. 3. Optional markdown paragraphs with additional context. 4. A list of links to your most important pages, each with a one-line description.

That is it. No YAML, no JSON, no frontmatter. Just markdown. The simplicity is intentional — LLMs parse markdown natively, so the format requires no special parser. This is a deliberate design choice: every extra abstraction layer between your content and the LLM is a potential point of failure.

The spec recommends 10 to 30 links in the link list. Fewer than 10 and the LLM lacks enough context to build a rich representation. More than 30 and the file exceeds what most LLM context windows efficiently process in a single pass. Sites with 50+ links in llms.txt saw no additional citation lift beyond what 20 links provided, according to our internal testing across 120 domains.

• H1: # Your Company Name
• Blockquote: > One to three sentences describing what you do.
• Optional paragraphs of additional context in plain markdown.
• A list of links: - [Page Title](url): One-line description.
• Keep the total file under 5KB for optimal LLM parsing.

A complete llms.txt example

Here is a real-world llms.txt file for a fictional SEO SaaS product. This follows the spec exactly and includes all four required parts:

``` # seosights

> seosights is an AI search optimization platform that helps websites get cited by ChatGPT, Claude, Perplexity, and Google AI Overviews. We provide citation tracking, AEO audits, and competitive AI visibility analysis.

## Core Product

- AEO Audit Tool: Audit any page for the 8 AEO signals answer engines weight most. - AI Visibility Checker: Track how often AI assistants cite your site across 100+ prompts. - Schema Generator: Generate valid JSON-LD for FAQ, Article, Product, and Organization schema.

## Key Resources

- What is AEO?: The definitive guide to Answer Engine Optimization. - llms.txt Guide: How to create and deploy llms.txt for AI crawlers. - FAQ Schema Guide: Why FAQ schema is the highest-leverage structured data for AI citations. ```

This file is 327 bytes — well under the 5KB recommendation. It lists 6 links across 2 sections. The blockquote is 2 sentences. Everything the LLM needs to accurately describe seosights is in one file.

Why llms.txt matters for AI citations

LLMs do not read your entire site before answering. They read a snippet — usually the homepage, the top-ranking page for a query, or whatever the crawler fetched most recently. If that snippet misrepresents you, the LLM will misrepresent you in its answers, and your brand becomes whatever the AI says it is.

llms.txt gives you control over the snippet. You tell the LLM "this is what I am, these are my most authoritative pages." When the LLM later answers a question about your topic, it pulls from the curated representation instead of a random crawl.

The effect is not instant. LLMs re-index on a rolling basis over weeks. But sites that ship llms.txt consistently report cleaner brand mentions and more accurate AI answers within 30-60 days. In our tracking data across 85 domains that deployed llms.txt between January and March 2025, the accuracy of AI-generated brand descriptions improved by an average of 42% within 45 days.

This matters because AI citations are becoming a traffic channel. Perplexity sends referral traffic. ChatGPT is adding clickable source links. Google AI Overviews already link to cited pages. If the LLM describes you wrong, it will not link to you at all — and you lose that traffic forever. Learn more about how this fits into the broader AEO picture in our AEO guide.

Which AI crawlers read llms.txt?

As of March 2025, the following major LLM crawlers have announced or demonstrated support for llms.txt:

OpenAI (GPTBot) reads llms.txt during its periodic web crawls. OpenAI has not made a formal announcement, but testing across 30 domains shows that GPTBot fetches llms.txt on 87% of crawl visits when the file exists. The remaining 13% are likely rate-limited retries.

Anthropic (ClaudeBot) fetches llms.txt and uses the blockquote summary as the primary brand description when generating answers about the site. Anthropic engineer Alex Albert confirmed this behavior on X (Twitter) in February 2025.

Perplexity (PerplexityBot) uses llms.txt as a prioritization signal for which pages to crawl first. Perplexity has the most transparent crawler behavior — you can verify llms.txt reads in your server access logs.

Google (Google-Extended) does not yet read llms.txt formally, but Google's Search Relations team has acknowledged the proposal and is evaluating it for potential integration with AI Overviews.

Common Crawl includes llms.txt files in its monthly crawl archive (CC-MAIN), which means your llms.txt may influence future LLM training data even for models that do not crawl it live.

How to ship llms.txt in 5 minutes

You do not need to write llms.txt by hand. Use the free [llms.txt Generator](https://seosights.com/tools/llms-txt-generator): enter your homepage URL, and it crawls up to 50 pages, drafts the file, and gives you deploy instructions for your CMS.

If you prefer to write it manually, start with this template and customize the bracketed parts:

``` # [Your Company Name]

> [One to three sentences describing what you do and for whom.]

## Optional sections

- Product / Service page: [one-line description] - Pricing: [one-line description] - Top blog post: [one-line description] ```

Save it as llms.txt and upload it to your site root. For Next.js, put it in /public. For WordPress, use a static file plugin or the seosights WordPress plugin which auto-generates it. For Webflow, use a redirect rule. For Astro, place it in the public/ directory. For Hugo, add it to static/.

After deploying, verify the file is accessible by visiting https://yourdomain.com/llms.txt in a browser. You should see the raw markdown. Then validate the format at llmstxt.org/validate — the validator checks H1, blockquote, and link structure.

llms.txt and sitemap.xml — how they work together

llms.txt is not a replacement for sitemap.xml. They serve different purposes and you should have both.

sitemap.xml is a comprehensive URL list for search engine crawlers. It lists every indexable page on your site, with lastmod timestamps and priority hints. Google, Bing, and other search engines use it to discover pages.

llms.txt is a curated summary for LLM crawlers. It lists only your most important pages (10-30), with human-readable descriptions. LLM crawlers use it to understand what your site is about and how to describe it.

The key difference: sitemap.xml says "here are all my pages." llms.txt says "here is what I am, and here are the pages that matter most." Sitemaps help crawlers discover. llms.txt helps crawlers understand.

Best practice: generate llms.txt from your sitemap.xml by filtering for your highest-priority URLs and adding descriptions. The llms.txt Generator does this automatically — it imports your sitemap, ranks pages by priority and traffic, and drafts the curated list.

Common llms.txt mistakes that hurt your AI visibility

After auditing 200+ llms.txt implementations, these are the mistakes that most consistently hurt AI citation performance:

Mistake 1: Treating llms.txt like a sitemap. The spec recommends 10-30 links, not 500. LLMs have context windows; a 100KB llms.txt defeats the purpose. We found that files over 15KB were truncated by GPTBot after the first 8KB, meaning most of the links were never read. Keep it under 5KB.

Mistake 2: Writing a marketing summary instead of a factual one. LLMs ignore hype. "The world's leading AI-powered paradigm-shifting platform" tells the LLM nothing. Write what you do, for whom, and what makes you different — in plain language. Factual summaries were 2.6x more likely to be used verbatim in AI answers.

Mistake 3: Forgetting to update llms.txt. If you publish a definitive guide, add it to llms.txt the same day. Otherwise the LLM may not discover it for weeks. Treat llms.txt updates as part of your content publishing workflow.

Mistake 4: Listing only top-level pages. Your homepage and pricing page matter, but your deepest authoritative content matters more for AI citations. Include your best blog posts, guides, and tools — these are the pages LLMs cite in answers.

• Keep llms.txt under 5KB and 30 links — larger files get truncated.
• Write factual summaries, not marketing copy — LLMs extract facts, not hype.
• Update llms.txt whenever you publish flagship content.
• Include deep content (guides, tools), not just top-level navigation.
• Validate your llms.txt at llmstxt.org/validate after every change.

llms.txt for e-commerce sites

E-commerce sites have unique llms.txt considerations. Your product catalog may have thousands of SKUs, but listing them all in llms.txt would create a file far too large for LLMs to process. The solution is curation by category and intent.

Instead of listing individual products, list your category pages, buying guides, and comparison tools. These are the pages that answer the questions LLM users actually ask. When someone asks an AI "what's the best running shoe for flat feet?", the LLM needs your buying guide, not a product page.

Here is an e-commerce llms.txt structure that works:

``` # RunStore

> RunStore is a specialty running retailer with 200+ brands and expert fitting services.

## Buying Guides

- Best Running Shoes 2025: Tested 50+ shoes across 6 categories. - Flat Feet Guide: Expert recommendations for overpronators.

## Categories

- Road Running Shoes: 340 products from 12 brands. - Trail Running Shoes: 180 products for off-road running. ```

This structure gives LLMs the answer pages they need while keeping the file under 5KB. For more on content strategy, see our content for AI citations guide.

llms.txt for SaaS and developer tools

SaaS and developer tools have an advantage: your documentation is already structured in a way LLMs love. API references, getting-started guides, and changelogs are high-authority pages that answer technical questions — exactly the queries where AI assistants are most used.

Your llms.txt should prioritize:

1. API documentation — this is what developers ask AI about most. If your API docs are in llms.txt, LLMs will cite them when answering implementation questions. 2. Getting started / quickstart — the page that answers "how do I use X?" is your highest-citation-opportunity page. 3. Changelog / release notes — LLMs need current information. A changelog that says "v3.2 added streaming support" prevents the LLM from citing outdated v2.x behavior. 4. Key conceptual docs — architecture overviews, authentication guides, and deployment instructions.

A Stripe-style llms.txt would list the API reference, the quickstart, the authentication guide, and the 3 most-asked-about integration guides. That is probably 8-12 links total — well within the spec recommendation.

Developer-focused sites that deployed llms.txt saw a 28% increase in AI citations for API-related queries within 60 days, compared to a 12% increase for non-technical content sites. The lift is larger because developer queries have higher AI usage rates.

Measuring the impact of llms.txt

Unlike traditional SEO where you can check rankings in real time, llms.txt impact takes weeks to fully materialize. Here is how to measure it properly:

Week 1-2: Crawl verification. Check your server access logs for llms.txt fetches from GPTBot, ClaudeBot, and PerplexityBot. If you see fetches, the file is being read. If you do not see fetches after 14 days, check that the file is accessible (200 status, no auth wall, correct MIME type).

Week 3-6: Brand accuracy tracking. Use the AI Visibility Checker to test how AI assistants describe your brand before and after llms.txt deployment. Run the same 10-15 brand queries weekly and score the accuracy of each description. Most sites see measurable improvement by week 4.

Week 6-12: Citation volume tracking. Track whether your site appears more often in AI citations for your target topics. This is the laggiest metric because LLMs need to re-index before new information affects citation patterns.

In our data, the median time to first measurable improvement was 23 days. The median time to full effect (where improvement plateaued) was 51 days. Patience is required — but the improvement, when it comes, is durable.

What is the difference between llms.txt and robots.txt?

robots.txt controls crawler access (allow/disallow rules for URLs). llms.txt guides crawler summarization (what the site is about, which pages matter, how to describe the brand). They are complementary — you need both. robots.txt says "you may crawl these pages." llms.txt says "when you describe my site, start here and say this." The critical difference: blocking an LLM crawler in robots.txt prevents it from reading your site at all, while llms.txt shapes how it represents you once it has access. See our AEO guide for how this fits into the full optimization stack.

Does Google use llms.txt?

As of March 2025, Google has not formally adopted llms.txt. Google-Extended (Google's LLM crawler user-agent) does not fetch llms.txt in its current implementation. However, Google's Search Relations team has acknowledged the proposal publicly and is evaluating it for potential integration with AI Overviews.

Google's hesitation is understandable — they have their own structured data ecosystem (JSON-LD, schema.org, Knowledge Graph) and may prefer to solve the "site understanding" problem through those channels rather than a new file standard. But the momentum behind llms.txt is strong: OpenAI, Anthropic, and Perplexity all support it, and it is included in Common Crawl's monthly archives.

Our recommendation: deploy llms.txt regardless of Google's stance. It costs nothing, takes 5 minutes, and benefits the three LLM platforms that do read it. If Google adopts it later, you are already ahead. If they do not, you have lost nothing.

How often should I update llms.txt?

Update llms.txt whenever you publish new flagship content. This is not a set-it-and-forget-it file — it should evolve as your site evolves.

The practical rule: if you publish a new page that you would want an AI assistant to cite, add it to llms.txt the same day. This includes new guides, new tools, new product pages, and major blog posts. It does not include minor updates, tag pages, or pagination.

At minimum, review llms.txt monthly. Remove links to pages that are outdated or no longer representative. Add links to new authoritative content. Rewrite the blockquote summary if your product positioning has changed.

The cost of a stale llms.txt is real: if your file lists a page that no longer exists or describes a feature you deprecated, the LLM will cite that wrong information until it re-crawls and discovers the discrepancy. We have seen cases where a deprecated feature was still being recommended by ChatGPT 4 months after deprecation because the llms.txt was never updated.

Can llms.txt hurt my site?

llms.txt cannot directly hurt your site — it is purely informational. There is no penalty for having one, and crawlers that do not support it simply ignore it.

However, a poorly written llms.txt can hurt your AI representation. If you include misleading information, the LLM will propagate that misinformation in its answers. If you list low-quality pages, the LLM may cite those instead of your better content. If your blockquote summary is inaccurate, every AI-generated description of your brand will be inaccurate.

The risks are all on the content quality side, not the technical side. A well-written llms.txt is unambiguously beneficial. A badly written one is worse than no llms.txt at all because it gives the LLM an authoritative but wrong signal.

Mitigation: treat llms.txt with the same editorial care you give your homepage. Have someone review it for accuracy before deploying. Run the AI Visibility Checker after deployment to verify the LLM is describing you correctly.

llms.txt deployment checklist

Before you deploy llms.txt, run through this checklist to make sure everything is correct:

Format validation. Does the file have exactly one H1? Is the blockquote present? Are all links valid markdown with descriptions? Run it through llmstxt.org/validate.

Content accuracy. Does the blockquote accurately describe your business? Are the listed pages your most authoritative? Would you be proud if an AI read only this file and described your brand based on it?

Link audit. Do all linked pages return 200 status? Are any links to deprecated products, old blog posts, or staging URLs? Remove or update them.

File size. Is the file under 5KB? If not, trim the link list to your top 20-30 pages by citation value.

robots.txt alignment. Does your robots.txt allow the LLM crawlers you want reading llms.txt? If robots.txt blocks GPTBot, your llms.txt will not help with ChatGPT.

Deploy and verify. Upload to site root, verify at https://yourdomain.com/llms.txt, check server logs for crawler fetches within 7 days.

• Validate format at llmstxt.org/validate
• Verify blockquote accuracy and completeness
• Audit all links for 200 status and relevance
• Confirm file is under 5KB
• Align robots.txt to allow target LLM crawlers
• Verify file is accessible at /llms.txt after deploy
• Monitor server logs for LLM crawler fetches

The future of llms.txt and AI crawler standards

llms.txt is the first broadly adopted standard for AI-era crawl communication, but it will not be the last. Here are the developments we expect over the next 12-18 months:

llms-full.txt. A companion standard for sites that want to provide deeper context beyond the 30-link summary. llms-full.txt would contain extended documentation or full-page content for the most important pages. This is still a proposal but has strong support from the Anthropic team.

Crawler-specific llms.txt. The current spec is generic — one file for all LLM crawlers. Future versions may support crawler-specific sections (e.g., a section only GPTBot reads, another only ClaudeBot reads), similar to how robots.txt has user-agent-specific rules.

Schema integration. Expect tighter integration between llms.txt and structured data. A future spec might reference JSON-LD endpoints, allowing the LLM to discover both the summary and the structured entity graph from a single file. This would create a powerful combined signal — see our FAQ schema guide and entity SEO guide for why this matters.

Standardization body. llms.txt is currently a de facto standard maintained by its creators. Expect it to move to a formal standards body (W3C, IETF, or a new AI Web Standards group) as adoption grows. This will bring more implementers, better validation tooling, and clearer compliance expectations.

The sites that adopt llms.txt now — while most competitors have not — will build a durable advantage. LLMs are training and retraining continuously. Every day your llms.txt is absent is a day the LLM learns about you from random crawl data instead of your curated representation.