llms.txt Explained: What It Is, Who Uses It, and How to Add It to Next.js

You've probably seen a stray llms.txt link in someone's site footer, or noticed a new file sitting next to robots.txt and sitemap.xml on a docs site you visited. Maybe you've even been told "you need this or ChatGPT won't recommend your site." Before you spend an afternoon building one, it's worth asking the boring-but-important question nobody answers clearly: does anything actually read it?

The honest answer is more nuanced than the hype suggests — and also more nuanced than the dismissals. llms.txt is real, it has a real spec, and real companies use it. It's also not a magic AI-SEO switch, and it isn't officially supported by the major search engines or model providers the way robots.txt is.

By the end of this guide, you'll know exactly what llms.txt is, what it isn't, who actually fetches it today, and — if you decide it's worth it — how to add one to a Next.js 16 App Router site in a few minutes.

---

Why This Is Confusing

llms.txt gets compared to robots.txt constantly, which is exactly why people misunderstand it. robots.txt is a decades-old, genuinely standardized protocol that tells crawlers what they're allowed to request — if you haven't set yours up yet, our Google Search Console setup guide covers robots.ts and sitemap.ts from scratch. llms.txt looks similar — a plain text file at your site's root — but it does something completely different: it's a curated, human-and-LLM-readable summary of your site's most important content, meant to be read on demand when someone (or their AI assistant) is actively trying to understand your project.

It's also less than two years old, was proposed by one person, and isn't controlled by any standards body. That's not a knock on it — it just means "should I add this?" deserves a more grounded answer than "yes, everyone's doing it."

---

What Is llms.txt, Exactly?

llms.txt is a plain Markdown file you publish at https://yoursite.com/llms.txt. It was proposed on September 3, 2024 by Jeremy Howard of Answer.AI (the team behind fast.ai and FastHTML), as a way to solve a specific problem: large language models have limited context windows, and a full HTML page — full of navigation bars, ads, and JavaScript — is a wasteful, imprecise way to hand a model information about your site.

The idea is to give models a shortcut: one concise file, in a format they already parse well (Markdown), that says "here's what this site is, and here's where to find more detail if you need it."

Your Website

├── robots.txt
│      ↓
│   Search Crawlers

├── sitemap.xml
│      ↓
│   Index Discovery

└── llms.txt
       ↓
 AI Coding Agents

💡 Tip: llms.txt is a community-maintained specification, not a ratified web standard like HTTP or robots.txt's RFC. It's hosted and documented at llmstxt.org, with a public GitHub repo where people can propose changes.

The Required Format

The spec is intentionally strict about structure so both humans and simple parsers can read it reliably. A valid llms.txt file contains these sections, in this order:

1. An optional byte-order mark
2. An H1 with your project or site name — this is the only required section
3. A blockquote with a short summary containing the key facts someone needs to understand the rest of the file
4. Zero or more plain paragraphs or lists with more context (no headings allowed here)
5. Zero or more H2-delimited sections, each containing a Markdown list of links — these are your "file lists" pointing to more detailed pages

Here's the minimal shape:

# Project Name

> One or two sentences summarizing what this is and why it matters.

Optional extra context or notes go here — things like "this API isn't
compatible with X" or "written for developers who already know Y."

## Docs

- [Getting Started](https://example.com/docs/start.md): Setup and first steps
- [API Reference](https://example.com/docs/api.md): Full endpoint reference

## Optional

- [Changelog](https://example.com/changelog.md): Full version history

What's happening here: the H1 and blockquote are the only parts a model must be able to rely on — everything else is there to help it decide what to fetch next. That ## Optional section has special meaning in the spec: it's explicitly for secondary material that can be skipped if a model needs to save context space. Anything genuinely optional (a full changelog, an old migration guide) belongs there instead of in your main sections.

⚠️ Common Mistake: Treating llms.txt like robots.txt and trying to write "allow" or "disallow" rules into it. It has no access-control syntax at all — it's purely informational. If you want to control what AI crawlers can fetch, that's still robots.txt's job.

The Companion Idea: .md Versions of Your Pages

The original proposal also suggests something separate but related: serving a clean Markdown version of any page at the exact same URL, with .md appended. So example.com/docs/routing would also be reachable at example.com/docs/routing.md — the same content, minus the nav bar, ads, and JS chrome. This is optional, and most sites that adopt llms.txt skip it, but it's the other half of the same idea: give models plain text instead of forcing them to parse a full rendered page.

---

llms.txt vs robots.txt vs sitemap.xml

This is the comparison that actually matters, because each file solves a different problem:

File	Purpose	Who reads it	Format
robots.txt	Tells crawlers what they're allowed to request	Search engine and AI crawlers, generally respected	Simple directive syntax
sitemap.xml	Lists every indexable URL on your site, for discovery	Search engine crawlers	XML
llms.txt	A curated, concise summary of your most important content	Coding agents, some chatbots, on-demand — inconsistently	Markdown

The llms.txt spec itself is upfront about this distinction: it's designed to complement robots.txt and sitemap.xml, not replace either. A sitemap is exhaustive and built for indexing; llms.txt is deliberately small and curated, and it's meant to be pulled in at inference time — when someone's actually asking a question — rather than during a training run or a full crawl.

---

Does Anyone Actually Read It? (The Honest Answer)

This is the part most articles either oversell or dismiss entirely, so let's be precise.

What's confirmed:

• Coding-focused AI tools — Cursor, GitHub Copilot, Windsurf, Claude Code, and similar IDE agents — do fetch llms.txt (and often llms-full.txt) when they're pointed at a documentation site or a library's docs, as part of pulling in relevant context before writing code.
• Companies that ship developer tools — Anthropic, Stripe, Cloudflare, Vercel, Supabase, Mintlify, and others — publish one, largely for exactly that audience: developers and their AI coding assistants.

What's not confirmed:

• Google has not stated that any of its crawlers use llms.txt to inform Search or AI Overviews. When asked directly why a site's llms.txt wasn't showing up in crawl logs, Google's John Mueller responded that no AI crawler has publicly confirmed that it extracts information via llms.txt.
• A large-scale study covering roughly 300,000 domains (SE Ranking, late 2025) found adoption sitting around 10% of sites — modest growth after well over a year of industry conversation.
• Search Engine Land reported that in a small sample of sites that added llms.txt, most saw no measurable change in AI referral traffic afterward.

So the realistic picture in 2026: llms.txt is a genuinely useful convention for developer-facing content consumed by coding agents and AI-assisted IDEs — but it is not a confirmed input to ChatGPT Search, Google AI Mode, or Perplexity's general web answers, and there's no solid evidence yet that adding one moves the needle on general AI citations or search rankings.

💡 Tip: Treat llms.txt the way you'd treat any unofficial-but-plausible SEO tactic: cheap to try, worth doing if it fits your site, but not something to obsess over or expect measurable ranking gains from.

---

Should Your Next.js Site Have One?

It depends entirely on what your site is:

• You run a docs site, an SDK, an open-source library, or a developer tool (like DevStacked's tools and guides): yes, worth it. Your actual audience includes other developers whose coding assistants will genuinely benefit from a clean, curated summary — this is the use case the spec was built for.
• You run a general blog, marketing site, or e-commerce store: lower priority. There's no confirmed benefit yet for general-purpose AI search visibility, and your time is probably better spent on the fundamentals — structured data, internal linking, and content depth — which are confirmed to matter for both traditional SEO and AI citations. See the complete technical SEO checklist for where to focus instead, and the JSON-LD Generator for structured data that's actually proven to affect rich results.
• Either way: it costs almost nothing to add, doesn't conflict with anything else on your site, and gives you a clean, canonical "here's what this is" file you can point people (and their tools) to directly.

---

When You Probably Don't Need One

It's worth being just as direct about when to skip this. Adding llms.txt isn't harmful, but it isn't free either — it's one more file to keep accurate, and a stale or careless one is arguably worse than none at all. Skip it (for now) if:

• You're pre-launch or pre-traffic. A curated summary file is only useful once there's something worth curating. Get your core pages, metadata, and content live first — see how to add metadata in Next.js 16 if that part isn't done yet.
• Your site is small and already simple to crawl. A five-page marketing site doesn't need a curation layer — a model (or a human) can already understand it from the page content and your sitemap alone.
• You don't have the bandwidth to keep it accurate. An llms.txt that still lists a deprecated API or a renamed pricing plan six months later actively misleads anything that reads it. If you can't commit to updating it alongside your content, it's better left out until you can.
• You're chasing it purely for "AI SEO." As covered above, there's no confirmed ranking or citation benefit yet. If that's your only motivation, your time is better spent on things with a proven payoff — structured data, canonical URLs, and Core Web Vitals, all covered in the technical SEO checklist.

💡 Tip: If you're unsure, the safest default is: build it for genuinely developer-facing content (docs, tools, APIs) where coding agents are a real audience, and skip it everywhere else until the spec's adoption picture is clearer.

---

Step-by-Step: Adding llms.txt to a Next.js 16 App Router Site

This guide assumes Next.js 16 with the App Router, TypeScript in strict mode, and content that changes over time (like blog posts) — so we'll generate the file dynamically instead of hand-maintaining a static one.

Step 1: Decide Static or Dynamic

If your site's content is small and stable (a handful of pages that rarely change), a plain static file is simplest.

If you have a blog or a growing set of tools/docs pages — which is the more common case — generating the file from your existing data source keeps it accurate without manual upkeep every time you publish.

We'll build the dynamic version, since it's the one that actually needs explaining. If you want the static route, skip to the callout at the end of Step 3.

Step 2: Create a Route Handler for /llms.txt

Next.js Route Handlers can serve any content type at any path — including a plain text file — so this is the right tool for a file that needs to be generated per-request (or cached and revalidated). If Route Handlers are new to you, our full breakdown of Route Handlers in Next.js 16 covers the route.ts convention this step relies on.

// app/llms.txt/route.ts
import { getAllPosts } from "@/lib/posts";
import { TOOLS } from "@/lib/tools-data";
import { appURL } from "@/lib/utils";

export const dynamic = "force-static"; // safe to prerender — same content for every visitor

export function GET() {
  const posts = getAllPosts();

  const docsSection = posts
    .slice(0, 25) // keep the file focused — see the note on curation below
    .map((post) => `- [${post.title}](${appURL}/blog/${post.slug}): ${post.description}`)
    .join("\n");

  const toolsSection = TOOLS.map(
    (tool) => `- [${tool.name}](${appURL}${tool.href}): ${tool.tagline}`
  ).join("\n");

  const content = `# DevStacked

> DevStacked is a developer-focused site with practical Next.js, React, Supabase,
> and Stripe implementation guides, plus a set of free browser-based developer tools.

Guides are implementation-driven, covering real architecture decisions and
production code rather than surface-level tutorials.

## Tools

${toolsSection}

## Guides

${docsSection}

## Optional

- [Full sitemap](${appURL}/sitemap.xml): Every indexable page on the site
`;

  return new Response(content, {
    headers: {
      "Content-Type": "text/plain; charset=utf-8",
    },
  });
}

What's happening here:

• The route lives at app/llms.txt/route.ts — Next.js folder names can include a dot, so this maps directly to the URL /llms.txt, no rewrites needed.
• export const dynamic = "force-static" tells Next.js this route's output doesn't depend on the incoming request, so it can be generated once at build time instead of on every visit — the same optimization your sitemap.ts and robots.ts files already get.
• The response sets Content-Type: text/plain; charset=utf-8 instead of the default JSON or HTML content type, since llms.txt is meant to be read as plain Markdown text, not rendered.
• Only the 25 most recent posts are included in the main section — see the note below on why bigger isn't better here.

⚠️ Common Mistake: Dumping your entire sitemap into llms.txt. The spec exists specifically because full-page dumps are what sitemap.xml is already for — llms.txt is supposed to be a curated shortlist. If every single page is listed, you've just built a worse, non-standard sitemap that a model has no more reason to trust than the real one.

Step 3: Verify the Output

Run your dev server and visit the route directly:

npm run dev

Then open http://localhost:3000/llms.txt in your browser. You should see plain Markdown text (not rendered HTML) — if your browser tries to download it or shows raw HTML instead, double-check the Content-Type header in Step 2.

💡 Static-site alternative: If your content doesn't change often, skip the Route Handler entirely and just drop a hand-written llms.txt file into /public/llms.txt. Anything in public/ is served as-is at the site root, so public/llms.txt becomes yoursite.com/llms.txt automatically — no route code needed. This is the simpler option for a personal site or a small marketing site with a handful of pages.

Step 4: Link to It From robots.txt (Optional, but Tidy)

Since the spec says llms.txt is meant to coexist with robots.txt, it's a reasonable place to reference it — the same way you already point to your sitemap:

// app/robots.ts
import { appURL } from "@/lib/utils";
import { MetadataRoute } from "next";

export default function robots(): MetadataRoute.Robots {
  return {
    rules: {
      userAgent: "*",
      allow: "/",
    },
    sitemap: `${appURL}/sitemap.xml`,
  };
}

There's no official field for referencing llms.txt inside robots.txt — this step is just a discoverability nicety, not part of the spec. A simple comment-style mention or a link from your footer/README works just as well.

Step 5: Keep It Curated, Not Comprehensive

The single biggest quality mistake is treating llms.txt like a dumping ground. A few concrete guidelines from the spec authors, worth following:

• Use plain, unambiguous language — avoid jargon you haven't already defined
• Give every linked resource a short, informative description, not just a bare link
• Put genuinely secondary material (old changelogs, deprecated docs) under the ## Optional heading so it's easy for a model to skip
• If your file starts feeling long, that's a sign to trim it, not add a summary section on top of it

---

Frequently Asked Questions

Does llms.txt help my Google rankings?

No confirmed evidence of that yet. Google's own John Mueller has said no AI crawler has confirmed using llms.txt for content extraction, and it isn't part of Google's documented Search ranking systems. Treat it as separate from your regular SEO work — Google Search Console and a solid technical SEO checklist are still where the confirmed ranking signals live.

Is llms.txt an official web standard?

No — it's a community-maintained convention proposed by Jeremy Howard of Answer.AI in September 2024, documented at llmstxt.org. There's ongoing community discussion about formalizing it further, but as of now there's no ratified standards-body specification the way there is for robots.txt.

Will ChatGPT or Perplexity actually read my llms.txt?

Not reliably, based on current evidence. The clearest, best-documented use case today is coding tools and IDE agents (Cursor, GitHub Copilot, Claude Code, and similar) fetching a project's llms.txt when a developer is working with that library. General-purpose chat search products haven't confirmed using it the same way.

Should I generate llms.txt dynamically or just write it by hand?

If your content is small and stable, hand-writing a static file in public/llms.txt is simpler and easier to review. If you have a blog or a growing tools page, generating it from the same data source your site already uses (like getAllPosts()) keeps it accurate without extra manual work every time you publish.

Can llms.txt block AI crawlers from my content?

No — it has no access-control syntax at all. It's purely descriptive. If you want to allow or block specific AI crawlers, that's still handled through robots.txt, using each crawler's documented user-agent string.

---

Wrapping Up

llms.txt is a real, well-specified convention — not an official standard, but not vaporware either. It has a clear, honest use case today: helping AI coding tools and IDE agents quickly understand a developer-facing site without wading through rendered HTML. For a project like a Next.js tools-and-guides site, it's a cheap, low-risk addition. What it isn't, at least as of now, is a proven lever for general AI search visibility or Google rankings — so add it because it genuinely helps your developer audience, not because someone promised it'll get you cited by ChatGPT.

If you're already generating sitemap.ts and robots.ts from your post data, adding app/llms.txt/route.ts following the same pattern takes about ten minutes — and now you know exactly what it will, and won't, actually do.

Continue Learning

• Google Search Console Setup for Next.js (2026 Beginner's Guide)
• The Complete Technical SEO Checklist for Next.js Developers (2026 Guide)
• Route Handlers Explained in Next.js 16 (Beginner's Guide)
• How to Add Metadata in Next.js 16 (2026 Beginner's Guide)
• Google Tag Manager & GA4 Setup for Next.js 16 (2026 Guide)

Free SEO Tools

• Meta Tag Generator for Next.js & HTML — generate SEO meta tags, Open Graph tags, canonical URLs, and JSON-LD in one pass.
• JSON-LD Generator for Next.js & HTML — generate structured data checked live against Google's actual Rich Results requirements, not just schema.org validity.