add llms.txt to docs

[futurepaul]

What is the problem this feature would solve?

Most llms are under-trained on bun, which is especially noticeable with new or novel bun features (such as Bun.serve or the new static option). When using llms with bun I have to add a lot of instructions on which apis to use, or copy and paste docs, or go in and correct node APIs to bun APIs manually.

What is the feature you are proposing to solve the problem?

/llms.txt is a new emerging standard for presenting a website in an llm-friendly format. In this case, I'm proposing to add a build step to docs generation to output an llms.txt and llms-full.txt to the docs url. So if an llm is attempting to use http://bun.sh/docs they can find http://bun.sh/docs/llms.txt or http://bun.sh/docs/llms-full.txt

Here's a directory of these llms.txt files for reference: https://directory.llmstxt.cloud

I wrote a simple sample script that would output these files, but I don't know where this actually belongs or what would run it, so I'm just putting it here instead of a PR.

In my personal tests using Cursor and this prompt:

make a static server that imports an index.html file and serves it. follow bun best practices

I got the best results by adding llms-full.txt to my Cursor Docs. Similar results by using Cursor's built-in Docs indexer on the regular http://bun.sh/docs site. Cursor didn't do much with the simple link list of llms.txt (results weren't much better than without the llms.txt) but I don't know how other llms will handle it.

Here's my script:

import docs from "./nav";

const ROOT_URL = "https://bun.sh/docs";

const header = `# Bun

## Docs`;

async function generateOutline() {
  // Generate markdown content with dividers and links
  const content = docs.items
    .map(item => {
      if (item.type === "divider") {
        return `\n### ${item.title}\n`; // Add newlines around divider for better readability
      }
      // For pages, create markdown link with description
      if (item.disabled) {
        return `- ${item.title} _(Coming soon)_: ${item.description}`;
      }
      return `- [${item.title}](${ROOT_URL}/${item.slug}): ${item.description}`;
    })
    .join("\n");

  await Bun.write(`${import.meta.dir}/llms.txt`, `${header}\n${content}`);
}

// Generate full content by reading each markdown file
async function generateFullContent() {
  const fullContent: string[] = [header];

  for (const item of docs.items) {
    if (item.type === "divider") {
      fullContent.push(`\n# ${item.title}\n`);
      continue;
    }

    if (item.disabled) {
      fullContent.push(`\n## ${item.title} _(Coming soon)_\n\n${item.description}\n`);
      continue;
    }

    try {
      // Handle nested paths by splitting the slug on '/' and joining with the directory
      const basePath = `./${item.slug.split("/").join("/")}`;
      let mdPath: string;

      // Try direct .md file first
      try {
        mdPath = Bun.resolveSync(`${basePath}.md`, import.meta.dir);
      } catch {
        // If that fails, try index.md in a directory
        mdPath = Bun.resolveSync(`${basePath}/index.md`, import.meta.dir);
      }

      const file = Bun.file(mdPath);

      if (await file.exists()) {
        fullContent.push(`\n## ${item.title}\n`);

        const content = await file.text();
        // Remove HTML comments
        let cleanContent = content.replace(/<!--[\s\S]*?-->/g, "");

        // TODO: handle  blocks like codetabs, etc.
        fullContent.push(cleanContent);
      } else {
        console.warn(`File does not exist: ${mdPath}`);
        continue;
      }
    } catch (error) {
      console.warn(`Could not resolve file for ${item.slug}:`, error);
      continue;
    }
  }

  // Write the full content to llms-full.txt
  await Bun.write(`${import.meta.dir}/llms-full.txt`, fullContent.join("\n"));
}

// Generate both files
await generateOutline();
await generateFullContent();

What alternatives have you considered?

Cursor's built-in docs indexer keeps getting better, this is actually the first time I've tried it where it did better than me manually telling it about bun apis.

Also, now that I've generated these myself I don't NEED it to be a part of the official project, I can self-host these for myself.

I could also continue to write bun code by hand the old-fashioned way!

llms.txt
llms-full.txt