<SYSTEM>This is the full developer documentation for Starlight llms.txt Plugin</SYSTEM>

# Starlight llms.txt Plugin

> Automated llms.txt and llms-full.txt generation for Astro Starlight documentation sites.

The **starlight-llms-txt** plugin integrates with Astro Starlight to generate standard-compliant `llms.txt` and `llms-full.txt` feeds. These endpoints provide large language model (LLM) agents, crawlers, and AI developer tools with structured, token-efficient, and federated documentation context.

## Core Capabilities

Standard Compliance

Generates structured `/llms.txt` files conforming to the emerging LLM context standard with curated link sections.

Full Content Concatenation

Builds single-file `/llms-full.txt` feeds stripped of UI boilerplate, navigation bars, and HTML noise.

Multi-Locale Routing

Automatically produces localized feeds (e.g., `/ja/llms.txt`, `/ko/llms-full.txt`) matching your Starlight locales.

Federated Sites Integration

Aggregates remote documentation endpoints across modular multi-repo ecosystems via `llms-federated-sites.json`.

## Architecture & Feed Pipeline

```text
+-------------------------------------------------------------------------+
|                        Astro Starlight Build Engine                     |
|                                                                         |
|  +-----------------------+     +-------------------------------------+  |
|  | Markdown/MDX Content  |     | starlight-llms-txt Integration Hook |  |
|  +-----------+-----------+     +------------------+------------------+  |
|              |                                    |                     |
|              v                                    v                     |
|  +-----------------------+     +-------------------------------------+  |
|  | Clean Markdown AST    | --> | llms.txt / llms-full.txt Generator  |  |
|  | (Stripped Frontmatter)|     | (Tier Trees & Federated Federation) |  |
|  +-----------------------+     +------------------+------------------+  |
+---------------------------------------------------|---------------------+
                                                    |
                                                    v
+-------------------------------------------------------------------------+
|                         Generated Static Endpoints                      |
|   /llms.txt  |  /llms-full.txt  |  /llms-small.txt  |  /llms-tiered.txt |
+-------------------------------------------------------------------------+
```

## Quick Start

### Installation

Install the package into your Starlight project:

```bash
npm install @f5-sales-demo/starlight-llms-txt
```

### Configuration

Add the plugin to your `astro.config.mjs`:

```javascript
import { defineConfig } from 'astro/config';
import starlight from '@astrojs/starlight';
import starlightLlmsTxt from '@f5-sales-demo/starlight-llms-txt';


export default defineConfig({
  integrations: [
    starlight({
      title: 'Example Documentation',
      plugins: [
        starlightLlmsTxt({
          description: 'Technical documentation for Example Corp cloud services.',
          details: 'Includes architecture guides, CLI references, and API specifications.',
          promote: ['getting-started', 'architecture'],
          demote: ['archive', 'deprecated'],
        }),
      ],
    }),
  ],
});
```

# Configuration Reference

> Complete options and schema reference for the starlight-llms-txt plugin.

## Plugin Options

The `starlightLlmsTxt` plugin accepts the following configuration parameters:

| Option           | Type       | Default          | Description                                                                   |
| :--------------- | :--------- | :--------------- | :---------------------------------------------------------------------------- |
| `description`    | `string`   | Site description | High-level summary of the documentation site displayed in the header block.   |
| `details`        | `string`   | `undefined`      | Extended guidance or context instructions for AI agents processing the feed.  |
| `promote`        | `string[]` | `[]`             | List of section slugs to prioritize near the top of `llms.txt`.               |
| `demote`         | `string[]` | `[]`             | List of section slugs to place at the bottom of the index.                    |
| `exclude`        | `string[]` | `[]`             | Glob patterns or route paths to exclude from `llms-full.txt`.                 |
| `federatedSites` | `string`   | `undefined`      | File path to a JSON file containing federated remote documentation endpoints. |

## Federation Schema

When linking multiple repositories across an organization, provide a `llms-federated-sites.json` file:

```json
[
  {
    "label": "Service API Reference",
    "url": "https://docs.example.com/api/llms.txt",
    "description": "OpenAPI specifications and endpoint contracts",
    "category": "developer-tools"
  }
]
```

# starlight-llms-txt

> Generate llms.txt files to train large language models on your Starlight documentation website

## Demo

`starlight-llms-txt` generates the following files for this site:

* llms.txt

  [llms.txt file for this site](/starlight-llms-txt/llms.txt)

* llms-full.txt

  [llms-full.txt file for this site](/starlight-llms-txt/llms-full.txt)

* llms-small.txt

  [llms-small.txt file for this site](/starlight-llms-txt/llms-small.txt)

## Next steps

Install the plugin

Check the [getting started guide](/starlight-llms-txt/getting-started/) for installation instructions.

Configure the plugin

Edit your config in the `astro.config.mjs` file.

# Configuration

> How to customize the llms.txt and llms-full.txt files generated by the starlight-llms-txt plugin

The `starlight-llms-txt` plugin can be configured in your Astro config file:

astro.config.mjs

```js
import starlight from '@astrojs/starlight';
import { defineConfig } from 'astro/config';
import starlightLlmsTxt from 'starlight-llms-txt';


export default defineConfig({
 site: 'https://example.com/',
 integrations: [
  starlight({
   title: 'My Docs',
   plugins: [
    starlightLlmsTxt({
     projectName: 'Very Cool Tool',
    }),
   ],
  }),
 ],
});
```

## Options

The following configuration options are available to pass to the plugin:

### `projectName`

**Type:** `string`\
**Default:** the value of Starlight’s `title` option

Provide a custom name for this project or software. This will be used in `llms.txt` to identify what the documentation is for.

For example, with `projectName: "FastHTML"`, the generated `llms.txt` file would start:

llms.txt

```md
# FastHTML


## Documentation Sets


- [Complete documentation](https://example.com/llms-full.txt): the full documentation for FastHTML
```

### `description`

**Type:** `string`\
**Default:** the value of Starlight’s `description` option

Set a custom description for your documentation site to share with large language models. Can include Markdown syntax. Will be displayed as a blockquote in `llms.txt` immediately after the file’s title.

According to [llmstxt.org](https://llmstxt.org/) this should be:

> a short summary of the project, containing key information necessary for understanding the rest of the file

### `details`

**Type:** `string`\
**Default:** `undefined`

Provide additional details to add after the `description` in `llms.txt`.

According to [llmstxt.org](https://llmstxt.org/) this should be:

> Zero or more markdown sections (e.g. paragraphs, lists, etc) of any type except headings, containing more detailed information about the project and how to interpret the provided files

### `optionalLinks`

**Type:** `Array<{ label: string; url: string; description?: string }>`\
**Default:** `[]`

An array of optional links to add to the `llms.txt` entrypoint.

URLs provided here can be skipped by the LLM if a shorter context is needed. Use it for secondary information which is not already in your docs content.

### `customSets`

**Type:** `Array<{ label: string; paths: string[]; description?: string; }>`\
**Default:** `[]`

Specify additional subsets of your docs pages to generate and link to from `llms.txt`.

This can be helpful for large docs sites where you need to break up your content into categories, so that LLMs can choose what data to ingest.

Each custom set should define a label and an array of page slugs or glob patterns that match page slugs to include. Glob patterns are processed using [`micromatch`’s matching features](https://github.com/micromatch/micromatch#matching-features).

```js
customSets: [
 {
  label: 'Reference',
  description: 'full reference documentation for my project',
  paths: ['reference/**'],
 },
 {
  label: 'Tutorial',
  description: 'step-by-step tutorial for how to build a new project',
  paths: ['tutorial/**'],
 },
];
```

### `promote`

**Type:** `string[]`\
**Default:** `['index*']`

Use the `promote` option to specify pages that should be sorted to the top of the `llms-full.txt` and `llms-small.txt` files. The value should be an array of page slugs or glob patterns that match page slugs. Glob patterns are processed using [`micromatch`’s matching features](https://github.com/micromatch/micromatch#matching-features).

The default value promotes the index page. If you’d like to promote more or other pages, add them to the array:

```ts
// Example: In addition to the index page, also promote `getting-started`
// and any pages in the root directory to the top of the output
promote: ['index*', 'getting-started*', '!*/*'],
```

### `demote`

**Type:** `string[]`\
**Default:** `[]`

Use the `demote` option to specify pages that should be sorted to the bottom of the `llms-full.txt` and `llms-small.txt` files. The value should be an array of page slugs or glob patterns that match page slugs. Glob patterns are processed using [`micromatch`’s matching features](https://github.com/micromatch/micromatch#matching-features).

If a page matches patterns in both `promote` and `demote`, it will be demoted.

### `exclude`

**Type:** `string[]`\
**Default:** `[]`

Use the `exclude` option to specify docs pages that should be excluded when generating `llms-small.txt`. This allows you to filter out non-essential documentation for models with a smaller context window.

The value of `exclude` is an array of page slugs or glob patterns that match page slugs. Glob patterns are processed using [`micromatch`’s matching features](https://github.com/micromatch/micromatch#matching-features).

In the following example, a specific, outdated page is excluded as well as all pages in the `src/content/docs/tutorial/` directory:

```ts
exclude: ['old-page', 'tutorial/**'],
```

### `rawContent`

**Type:** `boolean`\
**Default:** `false`

When enabled, returns raw Markdown content without processing. Useful for large documentation sites where processing time is a concern.

You must enable the `rawContent` option if your content includes components built with UI frameworks like React, Vue, Svelte, etc.

```ts
rawContent: true,
```

### `minify`

A map of content types to filter out of docs when creating `llms-small.txt` for small context windows. Set an element to `true` to filter it out or to `false` to keep it.

Use the [`customSelectors`](#minifycustomselectors) option to provide an array of additional CSS-style selectors that match elements to exclude.

In the following example, the default exclusion of notes is overridden and a custom in-page navigation component is added to the elements to exclude:

```ts
minify: {
 note: false,
 customSelectors: ['.my-page-nav'],
},
```

#### `minify.customSelectors`

**Type:** `string[]`\
**Default:** `[]`

Specify custom CSS-style selectors to exclude. Selectors are tested with [`hast-util-select`’s matching features](https://github.com/syntax-tree/hast-util-select#support) and should match your site’s HTML output.

In the following example, `customSelectors` is used to exclude an `<interactive-demo>` custom element and an element showing a project’s sponsors that has the `sponsors-banner` class name:

```ts
customSelectors: ['interactive-demo', '.sponsors-banner'],
```

#### `minify.note`

**Type:** `boolean`\
**Default:** `true`

Exclude the `note` variant of Starlight’s aside component.

#### `minify.tip`

**Type:** `boolean`\
**Default:** `true`

Exclude the `tip` variant of Starlight’s aside component.

#### `minify.caution`

**Type:** `boolean`\
**Default:** `false`

Exclude the `caution` variant of Starlight’s aside component.

#### `minify.danger`

**Type:** `boolean`\
**Default:** `false`

Exclude the `danger` variant of Starlight’s aside component.

#### `minify.details`

**Type:** `boolean`\
**Default:** `true`

Exclude the `<details>` HTML element.

#### `minify.whitespace`

**Type:** `boolean`\
**Default:** `true`

Collapse whitespace. When `minify.whitespace` is set to `true`, all whitespace — including new lines — is collapsed to a single space character.

### `pageSeparator`

**Type:** `string`\
**Default:** `"\n\n"`

The separator used when concatenating page entries together.

```ts
pageSeparator: '\n----------\n',
```

### `sidebarNav`

**Type:** `boolean`\
**Default:** `false`

When enabled, `llms.txt` includes a `## Sections` block listing the site’s pages in hierarchical order. Pages are grouped by first path segment (e.g. `demo/phase-1-build` under a `demo` group). If a group has an index page (`demo/index.mdx`), its frontmatter title and description are used for the group heading; otherwise the path segment is title-cased.

Each entry’s `description` (from frontmatter) is appended automatically when present:

```md
## Sections


- [Overview](https://example.com/overview/): Product overview and architecture
- [Demo](https://example.com/demo/): 4-phase demo exercise
  - [Phase 1 — Build](https://example.com/demo/phase-1-build/): Deploy infrastructure via API
```

Respects the [`promote`](#promote) and [`demote`](#demote) options for ordering. Draft pages and non-default locales are excluded.

### `federatedSites`

**Type:** `Array<{ label: string; url: string; description?: string }>`\
**Default:** `[]`

An array of links to other sites’ `llms.txt` entry points. Use this on a docs portal to federate out to product-specific documentation. Empty array (the default) omits the block entirely, so leaf product sites don’t need conditional config.

TS config example:

```ts
federatedSites: [
 { label: 'WAF', url: 'https://example.com/waf/llms.txt', description: 'Web application firewall' },
 { label: 'CSD', url: 'https://example.com/csd/llms.txt', description: 'Client-side defense' },
]
```

Output:

```md
## Federated Sites


- [WAF](https://example.com/waf/llms.txt): Web application firewall
- [CSD](https://example.com/csd/llms.txt): Client-side defense
```

The block is placed between `## Sections` and `## Notes` — local navigation before cross-site traversal.

### `perPageMarkdown`

**Type:** `boolean | { extensionStrategy?: 'append' | 'replace'; excludePages?: string[] }`\
**Default:** `false`

Enable generation of individual Markdown (`.md`) files for each documentation page. This implements the second part of the [llmstxt.org standard proposal](https://llmstxt.org/#proposal), allowing LLMs to fetch specific documentation pages on-demand rather than processing the entire documentation.

Set to `true` to enable with defaults, or pass an object for advanced configuration.

```js
// Enable with defaults
perPageMarkdown: true,
```

```js
// Enable with custom configuration
perPageMarkdown: {
 extensionStrategy: 'replace',
 excludePages: ['404', 'index*'],
},
```

#### `perPageMarkdown.extensionStrategy`

**Type:** `'append' | 'replace'`\
**Default:** `'append'`

File naming pattern for the generated Markdown files.

* `'append'`: Adds `.md` to the existing URL (e.g., `/docs/getting-started` → `/docs/getting-started.html.md`).
* `'replace'`: Replaces the extension with `.md` (e.g., `/docs/getting-started` → `/docs/getting-started.md`).

#### `perPageMarkdown.excludePages`

**Type:** `string[]`\
**Default:** `['404']`

An array of page IDs to exclude from per-page Markdown generation. Supports glob patterns processed using [`micromatch`’s matching features](https://github.com/micromatch/micromatch#matching-features); for example, `['index*']` excludes all pages whose IDs start with `index`.

```js
excludePages: ['404', 'index*', 'admin/**'],
```

# Getting Started

> How to install and set-up starlight-llms-txt in your Starlight docs site

`starlight-llms-txt` is a plugin for the [Starlight](https://starlight.astro.build/) documentation site framework that auto-generates `llms.txt`, `llms-full.txt`, and `llms-small.txt` context files for large language models based on your documentation site’s content. It can also generate individual Markdown files for every docs page.

You can learn more about `llms.txt` files at [llmstxt.org](https://llmstxt.org/).

## Prerequisites

You will need to have a Starlight site set up. If you don’t have one yet, you can follow the [“Getting Started”](https://starlight.astro.build/getting-started) guide in the Starlight docs to create one.

## Installation

1. `starlight-llms-txt` is a Starlight [plugin](https://starlight.astro.build/reference/plugins/). Install it by running the following command in your terminal:

   ```bash
   npm install starlight-llms-txt
   ```

2. Configure the plugin in your Starlight [configuration](https://starlight.astro.build/reference/configuration/#plugins) in the `astro.config.mjs` file. A `site` URL is also required if you haven’t already set this:

   astro.config.mjs

   ```diff
   import starlight from '@astrojs/starlight'
   import { defineConfig } from 'astro/config'
   +import starlightLlmsTxt from 'starlight-llms-txt'


   export default defineConfig({
   +  site: 'https://example.com/',
     integrations: [
       starlight({
         title: 'My Docs',
   +      plugins: [starlightLlmsTxt()],
       }),
     ],
   })
   ```

3. [Start the development server](https://starlight.astro.build/getting-started/#start-the-development-server):

   ```bash
   npm run dev
   ```

4. Visit [`localhost:4321/llms.txt`](http://localhost:4321/llms.txt) to preview the plugin in action.

## Next steps

See the [Configuration](/starlight-llms-txt/configuration/) guide to learn how to adjust the output of the plugin.