What Is LLMs.txt? The New AI‑Friendly Web Standard Explained

In the rapidly evolving AI era, a new web standard called LLMs.txt is emerging to help large language models better understand and process website content.

What is LLMs.txt?

LLMs.txt is a website‑content standard optimized for large language models (LLMs). Unlike robots.txt or sitemap.xml, it is specifically designed to aid AI systems in comprehending site material.

The standard consists of two core files:

/llms.txt

: provides a concise navigation view of the site’s documents.

/llms-full.txt

: contains the complete documentation in Markdown format.

Why is LLMs.txt needed?

Current AI systems face three main challenges when processing web pages: limited context windows, interference from redundant information such as navigation bars and stylesheets, and a lack of structured data. LLMs.txt solves these issues by offering clear, Markdown‑formatted content.

Structure of LLMs.txt

Example from the author’s blog site

A typical

/llms.txt

file looks like this:

<code># Project Name

> Project Overview

Project background and important notes

## Core Documents

- [Quick Start](url): resource description
- [API Reference](url): API details

## Optional Content

- [Other Resources](url): supplementary information
</code>

The

/llms-full.txt

file contains the full documentation, also in Markdown.

Who is using LLMs.txt?

The standard was introduced by Jeremy Howard, co‑founder of Answer.AI, and has quickly attracted attention. Companies already adopting it include Mintlify (a documentation platform that auto‑generates llms.txt), Anthropic (the creator of Claude AI), and Cursor (an AI coding tool).

Example: Using LLMs.txt in Cursor

Cursor leverages LLMs.txt so its AI assistant can instantly retrieve and understand project documentation.

Typical workflow:

Document Indexing : Cursor automatically detects and indexes any

llms.txt

files in the project.

Contextual Linking : During coding, the AI assistant can dynamically associate relevant documentation content.

Automated Generation of LLMs.txt

To generate an

llms.txt

file with

llmstxt.firecrawl.dev

:

Visit the generator at

llmstxt.firecrawl.dev

.

Enter the target website URL.

Click “Generate” and wait a few minutes for processing, depending on site size.

Conclusion

LLMs.txt represents an AI‑first documentation paradigm. Although still early, it offers a practical solution for AI systems to understand and utilize web content, making it an opportune time for developers and content creators to adopt the standard.