What Is llms.txt, and Should You Care About It?

Developers and marketers are being told to add llms.txt files to their sites to help large language models (LLMs) “understand” their content.

But what exactly is llms.txt, who’s using it, and—more importantly—should you care?

llmstext.org.

In a nutshell, it’s a text file designed to tell LLMs where to find the good stuff: API documentation, return policies, product taxonomies, and other context-rich resources. The goal is to remove ambiguity by giving language models a curated map of high-value content, so they don’t have to guess what matters.

A screenshot from the proposed standard over on https://llmstxt.org/.

In theory, this sounds like a good idea. We already use files like robots.txt and sitemap.xml to help search engines understand what’s on a site and where to look. Why not apply the same logic to LLMs?

But importantly, no major LLM provider currently supports llms.txt. Not OpenAI. Not Anthropic. Not Google.

As I said in the intro, llms.txt is a proposed standard. I could also propose a standard (let’s call it please-send-me-traffic-robot-overlords.txt), but unless the major LLM providers agree to use it, it’s pretty meaningless.

That’s where we’re at with llms.txt: it’s a speculative idea with no official adoption.

Don’t sleep on robots.txt

llms.txt might not impact your visibility online, but robots.txt definitely does.

You can use Ahrefs’ Site Audit to monitor hundreds of common technical SEO issues, including problems with your robots.txt file that might seriously hamper your visibility (or even stop your site from being crawled).

Anthropic’s actual llms.txt file:

At its core, llms.txt is a Markdown document (a kind of specially formatted text file). It uses H2 headers to organize links to key resources. Here’s a sample structure you could use:

# llms.txt
## Docs
- /api.md
A summary of API methods, authentication, rate limits, and example requests.
- /quickstart.md
A setup guide to help developers start using the platform quickly.
## Policies
- /terms.md
Legal terms outlining service usage.
- /returns.md
Information about return eligibility and processing.
## Products
- /catalog.md
A structured index of product categories, SKUs, and metadata.
- /sizing-guide.md
A reference guide for product sizing across categories.

You can make your own llms.txt in minutes:

  1. Start with a basic Markdown file.
  2. Use H2s to group resources by type.
  3. Link to structured, markdown-friendly content.
  4. Keep it updated.
  5. Host it at your root domain: https://yourdomain.com/llms.txt

You can create it yourself or use a free llms.txt generator (like this one) to make it for you.

I’ve read about some developers also experimenting with LLM-specific metadata in their llms.txt files, like token budgets or preferred file formats (but there’s no evidence that this is respected by crawlers or LLM models).

directory.llmstxt.cloud—a community-maintained index of public llms.txt files.

Here are a few examples:

  • Mintlify: Developer documentation platform.
  • Tinybird: Real-time data APIs.
  • Cloudflare: Lists performance and security docs.
  • Anthropic: Publishes a full Markdown map of its API docs.

But what about the big players?

So far, no major LLM provider has formally adopted llms.txt as part of their crawler protocol:

  • OpenAI (GPTBot): Honors robots.txt but doesn’t officially use llms.txt.
  • Anthropic (Claude): Publishes its own llms.txt, but doesn’t state that its crawlers use the standard.
  • Google (Gemini/Bard): Uses robots.txt (via User-agent: Google-Extended) to manage AI crawl behavior, with no mention of llms.txt support.
  • Meta (LLaMA): No public crawler or guidance, and no indication of llms.txt usage.

This highlights an important point: creating an llms.txt is not the same as enforcing it in crawler behavior. Right now, most LLM vendors treat llms.txt as an interesting idea, and not something that they’ve agreed to prioritize and follow.

LLM visibility, but we lack the tools to do it. So we latch onto ideas that feel like control.

But in my personal view, llms.txt is a solution in search of a problem. Search engines already crawl and understand your content using existing standards like robots.txt and sitemap.xml. LLMs use much of the same infrastructure.

As Google’s John Mueller put it in a Reddit post recently:

AFAIK none of the AI services have said they’re using LLMs.TXT (and you can tell when you look at your server logs that they don’t even check for it). To me, it’s comparable to the keywords meta tag – this is what a site-owner claims their site is about … (Is the site really like that? well, you can check it. At that point, why not just check the site directly?)

John Mueller

Disagree with me, or want to share an example to the contrary? Message me on LinkedIn or X.

Further reading

Similar Posts

Python for SEO, Explained for Beginners
|

Python for SEO, Explained for Beginners

Bywiredgorilla

Python can feel intimidating if you’re not a developer. You see scripts flying around Twitter, hear people talking about automation and APIs, and wonder if it’s worth learning—or even possible—without a computer science degree. But here’s the truth: SEO is filled with repetitive, time-consuming tasks that Python can automate in minutes. Things like checking for…

Want to Grow Your Business? You Need a Growth Strategy
|

Want to Grow Your Business? You Need a Growth Strategy

Bywiredgorilla

Growing a business isn’t just about working harder; it’s about working smarter. A well-thought-out growth strategy helps you identify opportunities, allocate resources effectively, and create a roadmap for sustainable expansion. Whether you’re a startup or an established company, a growth strategy is essential to scale your business in today’s competitive landscape. Here’s a step-by-step guide…

How to Lock or Unlock Your Domain: A Guide to What a Domain Lock Is and What it Does
|

How to Lock or Unlock Your Domain: A Guide to What a Domain Lock Is and What it Does

Bywiredgorilla

Welcome to our guide on locking and unlocking your domain name. Your domain name is your business’ virtual address. If you’re creating a website for your business, it’s important to understand how to secure your domain name. In this guide, we’ll explore the world of domain locks and explain what they are, why they matter,…

What Are the Parts of a Domain Name, and What Do They Do? A Clear Guide to Knowing It All
|

What Are the Parts of a Domain Name, and What Do They Do? A Clear Guide to Knowing It All

Bywiredgorilla

Aristotle once said, ‘The whole is greater than the sum of its parts.” But it’s the parts of a domain name we care about. Learn about them in our guide. Even if technology isn’t your jam, you probably know how to find a particular website: you type the company name into your browser, plus a…