AI crawlers, llms.txt, and deciding what to allow

There is no single category of AI crawler. Agents fetching pages to build a training corpus, agents fetching to retrieve an answer for a user asking right now, and agents fetching on behalf of a person who pasted your URL are three different things with three different consequences — and most robots.txt files treat them identically.

Three purposes, three separate decisions

Complicating this, some providers use one agent for several purposes and others publish distinct agents per purpose, and the names change. Any list written down today will be partly wrong within a year, so the durable practice is to check your own server logs for what is actually fetching you and read each provider's current documentation before writing a rule.

robots.txt: what it does and what it cannot do

robots.txt is a request, honoured voluntarily. Major providers do honour it; anything determined to ignore it will. It is the right tool for stating a preference and the wrong tool for enforcing one.

# Conventional search indexing: allow.
User-agent: Googlebot
Allow: /

# A specific AI agent, blocked by name. Naming agents individually is
# the only way to make the training/retrieval distinction at all.
User-agent: SomeTrainingBot
Disallow: /

# Everything else, including agents that do not exist yet.
User-agent: *
Allow: /
Disallow: /api/

Sitemap: https://example.com/sitemap.xml

# Notes that matter:
# - Wildcards in user-agent names are not supported. "AI*" matches nothing.
# - Disallow prevents crawling, not indexing. A blocked URL can still be
#   listed from external links, with no description.
# - To keep a page out of an index, allow the crawl and serve noindex.
#   A page blocked in robots.txt is never fetched, so the noindex is
#   never read.

That final note catches people every time. Blocking and de-indexing are opposite mechanisms: one prevents the fetch, the other requires it. Using the first to achieve the second produces a URL that stays listed and can no longer be described.

llms.txt: a convention, not a standard

An /llms.txt file is a proposed convention: a plain-language map of a site, in Markdown, aimed at making the important content easy for a language model to locate without parsing navigation and layout. It is worth being precise about its status, because it is widely oversold.

# Example Company

> One-sentence description of what the organisation does, in plain
> language and with no marketing adjectives.

## Services
- [Service name](https://example.com/services/slug): what it covers,
  in one line.

## Insights
- [Article title](https://example.com/insights/slug): the question it
  answers, in one line.

## Facts
- Founded: 2026
- Registered in: India
- Contact: hello@example.com

Generate this from the same typed content the pages render, so it
cannot drift out of date. A hand-maintained llms.txt is stale within
two content changes.

If the decision is to block, enforce it properly

A robots.txt directive stops well-behaved agents. Where content genuinely must not be fetched — licensed material, paywalled archives, anything with a contractual restriction — the control has to be a control.

1. 01Authentication. Content behind a login is not available to a crawler, which is the only reliable boundary.
2. 02Verified-agent allowlisting at the edge. Identify legitimate crawlers by reverse DNS or published address ranges rather than by the user-agent string, which anyone can set to anything.
3. 03Rate limiting per agent. Prevents corpus-scale fetching without blocking the occasional retrieval.
4. 04Log and review. You cannot make a decision about traffic you have never looked at, and the agent list changes constantly.

A defensible default

For most organisations whose published content is marketing, documentation and expertise, the sensible position is: allow retrieval, allow search, and decide about training deliberately.

• Allow live retrieval agents. Being the cited source is the entire point of publishing the content
• Allow user-initiated fetches. Refusing them means a prospective customer is told your site would not answer
• Decide on training per organisation. If your published content is how you demonstrate expertise, exclusion mainly removes you from the corpus your competitors are in. If it is licensed or proprietary, exclusion is obviously correct
• Keep private paths blocked regardless — internal search, parameterised URLs, anything generating infinite space

Whichever way it goes, write down the reasoning. This is the kind of decision that gets revisited in eighteen months by someone who wants to know whether it was deliberate, and a robots.txt with unexplained agent names is indistinguishable from an accident.