With large language models (LLMs) now deeply embedded in web ecosystems, websites need efficient ways to make their content AI-accessible.

That’s where llms.txt and llms.full.txt come in - serving as roadmaps, directing LLMs straight to essential pages and documentation without requiring them to process complex HTML structures.

Definitions

• llms.txt is a simple text file placed in the root of your website. It works as a map that tells LLMs where to find important pages or API documentation. Think of it as a quick reference guide for AI models. [1]
  
  

• llms.full.txt is the extended version of llms.txt. Instead of just linking to pages, it can contain the full text of those pages in Markdown format, so LLMs can access all the content directly without extra steps. [1]

Why they matter

1. LLMs fetch content differently. They don't load entire web pages like a browser does. With llms.txt, LLMs get a direct list of important pages, cutting out unnecessary HTML. [1]
   
   

2. Saves tokens and speeds up responses. A clear list of key pages means LLMs don't waste tokens crawling irrelevant content, which reduces latency and cost.
   
   

3. Major websites are adopting it. Big names like Anthropic, Cloudflare, and Hugging Face have implemented their own llms.txt and some llms-full.txt. [2]
   
   

4. Growing adoption across the web. BuiltWith now shows >2,000 websites that expose an llms.txt file, up from almost none in early 2024. [3]
   
   

5. Crawl logs on llms.txt are real. Early reports show that GPTbot is actively crawling llms.txt [4]

Examples

Cloudflare's example [5] shows a clean hierarchical structure with sections and nested links to documentation pages in its llms.txt . It starts with a brief introduction about their platform and includes a helpful tip about accessing a Markdown archive.

Loops.so's example [6] demonstrates a more detailed approach, including full documentation content with headers, step-by-step instructions, and even embedded images in its llms-full.txt file.

You can find plenty more examples for both llms.txt and llms.full.txt on directory.llmstxt.cloud and llmstxthub.com.

Create your own llms.txt

Following a consistent nomenclature helps LLMs better navigate and understand your website's content hierarchy. Here's how to structure your file:

Basic Structure

# Main Website Title
Brief description of your website or platform (1-2 sentences)

## Primary Section
- [Page Title](yourwebsite.com/path/to/page) - Brief page description
- [Another Page Title](yourwebsite.com/path/to/another-page) - Brief page description

## Another Primary Section
- [Section Page](yourwebsite.com/section) - Brief page description
  - [Subsection Page](yourwebsite.com/section/subsection) - Brief page description
  - [Another Subsection](yourwebsite.com/section/another-subsection) - Brief page description

Nomenclature Guidelines

1. Headings: Use Markdown heading syntax for hierarchy

   • # for website title

   • ## for primary sections

   • ### for subsections
     
     

2. Page Links: Format as [Page Title](URL)

   • Use descriptive page titles that match actual page headers

   • Include full URLs (not relative paths)
     
     

3. Nesting: Use indentation (2 spaces) for hierarchy

   • Parent pages should be listed before children

   • Related pages should be grouped together
     
     

4. Descriptions (optional): Add brief descriptions under important links

   - [API Documentation](yourwebsite.com/api)- Short description of what users will find in this section

   
   

5. Tips and Callouts: Use Markdown blockquotes for important information

   > [!TIP]
   > Important information or shortcuts users should know
   
   

Following these conventions ensures LLMs can efficiently parse your site structure and direct users to the most relevant information. Remember to place your llms.txt file in your website's root directory for optimal discovery.

Alternatively, you can use our llms.txt generator to create your own llms.txt based on your website’s sitemap.

Sources

[1] https://llmstxt.org

[2] https://directory.llmstxt.cloud/

[3] https://trends.builtwith.com/robots/LLMS-Text

[4] https://www.linkedin.com/posts/volpini_apple-just-said-that-google-search-usage-activity-7329039241034809344-i3rK

[5] https://developers.cloudflare.com/llms.txt

[6] https://loops.so/docs/llms-full.txt

This article was partially written by AI but finalised and fact checked by a human.