What AI engines can read decides what they can cite. See also: can AI crawlers read JavaScript-rendered pages, does llms.txt help you get cited by AI and should you block GPTBot in robots.txt.
What is llms.txt?
llms.txt is a plain Markdown file at /llms.txt that describes a site and links to its most useful pages. Jeremy Howard of Answer.AI published the proposal at llmstxt.org in 2024 (dated 3 September), and it states: "We propose adding a /llms.txt markdown file to websites to provide LLM-friendly content." The proposal's reasoning is that "Agents are best served by concise, expert-level information gathered in a single, accessible location." An llms.txt file is a community proposal, not a web standard, and it does not control crawling: a robots.txt file is the plain-text file at the site root that does.
| File | What it does | Read by AI search engines? |
|---|---|---|
| robots.txt | Allows or blocks each crawler, by path | Yes, OpenAI, Anthropic, Perplexity and Google all document it |
| sitemap.xml | Lists the pages you want crawled, with dates | Yes, as part of normal crawling |
| llms.txt | Summarises the site and links key pages in Markdown | Not documented by any major AI search engine |
Who actually reads llms.txt?
The main readers of llms.txt today are developer tools, not AI search engines, even though the file is common. Data from GeoTest's own scans shows that of 4,775 websites checked between 4 and 27 October 2026, 1,630 sites (34.1%) served an llms.txt file. Coding assistants and documentation tools fetch llms.txt files from software documentation sites to pull in reference material, which is why many developer platforms publish one. For AI search, the evidence points the other way. Google's guidance on AI features, updated in 2025, says: "You don't need to create new machine readable files, AI text files, or markup to appear in these features." OpenAI's crawler documentation tells site owners to manage ChatGPT search through robots.txt and OAI-SearchBot, and notes that "it can take ~24 hours from a site’s robots.txt update for our systems to adjust"; it does not mention llms.txt.
How does GeoTest treat llms.txt?
GeoTest treats llms.txt as reported but not scored. A full site analysis tells you whether the file exists, and it is worth 0 of the site layer's 100 points in the GEO v6.0 rubric. The points go to signals AI search engines demonstrably use instead: robots.txt access for search-indexing crawlers (10 points), a healthy sitemap.xml (8 points) and topical clusters of linked pages (12 points). Google's robots.txt documentation covers the file that does control crawler access.
Is llms.txt the same as a sitemap?
llms.txt is not the same as a sitemap, though both list pages. sitemap.xml is an XML file that lists the URLs a site wants crawled, often with the date each one last changed, and crawlers use it to find pages. llms.txt is written for a model to read: a short Markdown summary with links to the pages that matter most. GeoTest scores the sitemap, worth 8 site-layer points, and a Site Analysis reads it and follows internal links to score up to 200 pages.
Does llms.txt replace robots.txt?
llms.txt does not replace robots.txt, because only robots.txt tells crawlers what they may fetch. A Disallow rule in robots.txt is documented by OpenAI, Anthropic, Perplexity and Google; a line in llms.txt is not a rule at all. OpenAI says a robots.txt change reaches ChatGPT search in about 24 hours, so allow or block AI crawlers in robots.txt and leave llms.txt to describe the site.
What goes in an llms.txt file?
An llms.txt file holds a short Markdown summary of the site and lists of links to its key pages. The proposal at llmstxt.org asks for an H1 with the site's name, a blockquote with a short summary, and then sections under H2 headings that list links, each with a short note on what the page covers. Keep the list to the pages you would hand a new reader first, not the whole sitemap.
How common is llms.txt compared with schema markup?
llms.txt is far less common than schema markup on the sites GeoTest scans. Of 4,775 websites checked between 4 and 27 October 2026, 34.1% served an llms.txt file; of 5,278 homepages scanned in the same weeks, 79.0% carried some JSON-LD and 55.2% marked up an Organization or LocalBusiness. Search engines document how they read schema today, which is one reason Structured Data earns 12 of the 100 page points and llms.txt earns none.
Should you publish an llms.txt file anyway?
You can publish an llms.txt file anyway, because it costs little and it helps if developers or AI agents use your documentation. GeoTest publishes one at geotest.ai/llms.txt. Just do not expect it to change whether ChatGPT or Perplexity cites you. If citations are the goal, the 2023 GEO study points to better uses of the time: it found that adding sources, statistics and quotations to the page itself raised visibility in AI answers, with a 40% lift at the top end. And server-rendered text matters more than any summary file: a Vercel study of 569 million GPTBot requests in 2024 found no sign that AI crawlers run JavaScript.
