llms.txt and making your docs crawlable by AI

Add an llms.txt file if you want, but do not expect it to do much on its own. It is a proposed community convention, not an official standard, and no major AI engine has committed to reading it. It is not a Google ranking factor. Adoption is real but uneven, and several engines ignore it entirely. So put it low on your list. The things that actually get a devtool cited by ChatGPT, Perplexity, and Google’s AI Overviews are older and more boring: clean HTML, a real answer in the first paragraph, structured data, and crawlers you have not accidentally locked out.

I say this as someone whose own site ranks second for a competitive AI-visibility query, cited by ChatGPT, Perplexity, and Google’s AI Overview ahead of much bigger names. None of that came from a magic text file. It came from writing pages a model can read without a fight.

What is llms.txt, actually?

llms.txt is a markdown file you place at your domain root, like example.com/llms.txt, that points to the pages you most want a language model to read. The idea came from Jeremy Howard in 2024. Think of it as a curated table of contents for machines: links to your best docs, your API reference, your key concept pages, sometimes with short descriptions. Some projects also ship an llms-full.txt that inlines the actual content so a model does not have to crawl page by page.

The pitch is reasonable. LLM context windows are finite, HTML is noisy, and giving a model a clean map of your good pages sounds helpful. I like the intent.

Here is the honest part. It is a proposal. There is no governing body, no shared spec everyone agreed to, and no confirmed commitment from OpenAI, Anthropic, Google, or Perplexity to fetch or trust the file. As of now it behaves more like a nice-to-have that a few tools respect than an input the big answer engines rely on. Treat any blog post promising it will “get you into ChatGPT” with suspicion. Nobody outside those companies knows their retrieval stack, and the companies are not saying.

So should you ship one? Sure, if it costs you an hour. It does not hurt, it signals you are paying attention, and if adoption grows you are already there. Just do not reorganize your whole content strategy around it.

What actually makes your docs legible to AI?

Answer engines do not have a secret llms.txt-shaped door. They mostly read your site the way a browser and a search crawler do, then their models decide what is quotable. Which means the wins are the ones good writers and decent front-end engineers already knew about.

Put the answer in the first paragraph. Models lift the sentence that directly answers the question. If your docs open with three paragraphs of context before defining the thing, the model either skips you or grabs a competitor who led with the definition. Every concept page should answer “what is this and what does it do” before it does anything else. This is the single highest-leverage change on this list.

Write clean semantic HTML. Real <h1> through <h3> structure, one clear heading per section, <p> tags around prose, ordered and unordered lists that are actually lists. Code in <pre><code>. If your headings are styled <div>s and your content only renders after client-side JavaScript, you are making the model guess at your structure, and some crawlers will not run your JS at all. Server-rendered HTML that reads cleanly with JavaScript off is the safest thing you can ship.

Add structured data. Schema.org JSON-LD for articles, FAQs, how-tos, software applications, and breadcrumbs gives machines an explicit, unambiguous statement of what a page is. It has helped traditional search for years and it gives answer engines a cleaner signal than parsing your prose. FAQ schema is especially worth it because the question-and-answer shape maps directly onto how people query these tools.

Do not block the crawlers you want citing you. This one bites people constantly. Teams add a blanket robots.txt disallow, or their WAF or bot-management rules quietly block AI user agents, and then they wonder why they never show up in AI answers. If you want Perplexity or ChatGPT to cite you, you have to let their crawlers in. Look up the current user agents (GPTBot, OAI-SearchBot, PerplexityBot, Google-Extended, ClaudeBot, and friends), decide deliberately which you allow, and stop blocking the ones you actually want traffic and citations from. Blocking training crawlers while allowing retrieval crawlers is a legitimate stance, just make it on purpose instead of by accident.

Make content a model can lift cleanly. Self-contained sections. Definitions near the top. Concrete code samples with the language declared. Plain tables instead of screenshots of tables. Alt text on diagrams. If the only place your setup steps exist is inside a video or an image, a model cannot quote them, so it will quote whoever wrote them as text.

None of this is exotic. It is the same discipline behind docs-led growth: docs that are easy for a human to skim are, not coincidentally, easy for a model to parse.

How does this connect to GEO?

Generative engine optimization is just SEO’s job description updated for a world where the answer often gets synthesized instead of clicked. The goal shifts from “rank in the ten blue links” to “be the source the model quotes and names.” Everything above is the technical layer of GEO. The strategy layer, which content to write, which questions to own, how to earn the third-party mentions that models weight heavily, is a bigger topic I get into in GEO for devtools.

The mistake I see founders make is treating GEO as a checklist of files to add. It is not. It is the same thing content has always been about, being genuinely the best answer to a real question, plus making sure the machine can read that answer without tripping over your markup. llms.txt is a tiny tactical piece. The legible content underneath it is what actually earns the citation.

And you cannot improve what you cannot see, so at some point you need to actually track your AI search visibility and watch whether these changes move anything. Ship, then check. Do not assume.

What should I do this week?

If you have a couple of hours, here is where the hours go, roughly in order of payoff:

• Open your top ten docs and concept pages. Rewrite each opening so the first sentence answers the page’s core question. No preamble.
• Check robots.txt and your bot-management rules for accidental blocks of AI crawlers you want, then unblock the ones you want citing you.
• Confirm your key pages render as real HTML without JavaScript. View source, or load with JS disabled, and make sure the content is actually there.
• Add JSON-LD structured data to your docs and any FAQ pages. Start with FAQ and Article schema.
• Replace any setup steps or key explanations that live only inside images or videos with real text.
• Then, if you have time left, write an llms.txt pointing to your best pages. Last, because it matters least.

That order is deliberate. The first three will do more for your AI visibility than the file everyone is currently writing blog posts about. Do the boring work first. The convention can wait.