The llms.txt File: What It Is, Why It Matters, How to Set It Up

In September 2024, Jeremy Howard proposed a standard called llms.txt. The idea is simple: just as robots.txt tells search engine crawlers what to index and what to skip, llms.txt tells AI models what your site is about and which content is most relevant for them to use.

It is not yet a universal standard, and not every AI platform honors it consistently. But for brands that want to give AI models a direct, machine-readable signal about their identity, category, and authority -- it is worth doing. It takes under an hour to implement and costs nothing.

This guide covers what to put in it, how to structure it, which platforms respond to it, and where it fits in a broader AI visibility strategy.

What llms.txt Actually Does

When an AI model or retrieval system crawls your site, it processes text without context. It sees your homepage copy, your blog posts, your product descriptions -- but it has no direct signal from you about what matters most, what category you belong to, or how you want to be described.

llms.txt is a plain text file at the root of your domain (yourdomain.com/llms.txt) that gives AI crawlers exactly that context. It typically includes:

• A clear, direct description of what your product or company does
• The primary category or problem space you belong to
• Links to your most important pages, organized by type
• Optional: a full-text version (llms-full.txt) with deeper content for AI indexing

Think of it as the briefing document you would want an AI model to read before forming an opinion about your brand.

The File Structure

The format is Markdown-compatible plain text. Here is a representative example for a B2B SaaS tool:

How to Create Your llms.txt

1. Write your descriptionStart with a single paragraph that describes exactly what you do, who you serve, and what category you belong to. Write it the way you would explain your product to a smart person who has never heard of you. Avoid marketing language -- "powerful," "innovative," "leading" -- and use specific, categorical language instead.
2. List your key pagesInclude your product/features page, pricing, documentation if you have it, and any high-value content pages. Do not list every page -- list the 8-12 pages that best represent your company and would give an AI model the fullest understanding of what you do.
3. Add comparison contextThe "Compared To" section is valuable because AI models frequently field comparison queries. If you name your category peers and articulate your differentiation, you give the model better material to work with when someone asks "how does [you] compare to [competitor]."
4. Create the fileSave it as a plain .txt file (not HTML, not rich text). Upload it to your web root so it is accessible at yourdomain.com/llms.txt. If you use a CMS, you may need to add it as a custom file through your hosting provider or CDN.
5. Optionally create llms-full.txtThis is a longer version with more complete content -- your full documentation, key blog posts, and detailed product descriptions in a single file. It is particularly useful for Perplexity and Claude, which can ingest longer context. Keep it under 100KB for reliable processing.

Which AI Platforms Honor llms.txt

What to Put in Your Description -- and What to Avoid

The description section is the most important part of the file. A few principles that make a difference:

Be categorical and specific. "CRM software for small e-commerce businesses" is better than "customer relationship management platform." The more specific the category language, the more accurately an AI model can place you in context when responding to category queries.

Use the language your buyers use. AI models learn from human-generated content. If your buyers search for "inventory tracking for Shopify stores," use that phrase -- not the internal term your team uses for the product.

Avoid superlatives. "The best," "the leading," "the most powerful" -- these phrases have no meaning to a model trying to understand what you do. They take up space that should be used for factual, categorical information.

Name the problem, not just the solution. Instead of "a project management tool," try "a tool that helps engineering managers see where work is blocked and who is overloaded." The problem-framing helps AI models match you to queries that describe a problem rather than a product category.

llms.txt Within a Broader AI Visibility Strategy

llms.txt is infrastructure. It makes sure AI models understand you correctly when they encounter you. But encountering you in the first place requires the external citation signals -- Reddit presence, review site depth, listicle inclusions, and journalist coverage.

Think of the sequence this way: external signals get you cited, and llms.txt ensures that when you are cited, the model has accurate, complete context about who you are. A brand with strong external signals but a weak or missing llms.txt may still rank -- but the description may be incomplete or pulled from less authoritative sources.

Implement llms.txt first because it is fast. Then build the external signals. Both work better together than either does alone.