Agent-Discovery Layer — Endpoint Reference

Phase 2b deliverable. Defines the machine-readable surface of the wiki: the endpoints any AI agent can fetch to discover, index, and consume this wiki without human help. All endpoints are generated automatically from the content/ pages' front-matter by scripts/generate-agent-endpoints.py — none are hand-maintained.

Endpoints

Endpoint	Content-Type	Generator	Purpose
`/llms.txt`	`text/plain` (markdown)	`generate-agent-endpoints.py`	llms.txt v2 (llmstxt.org) index: H1 + blockquote + H2 category sections, each a list of `[title](url): summary`.
`/llms-full.txt`	`text/plain` (markdown)	`generate-agent-endpoints.py`	Concatenated body markdown of every page — for agents that ingest everything.
`/index.json`	`application/json`	`generate-agent-endpoints.py`	Programmatic index: one object per page (id, url, raw_url, title, summary, category, tags, last_updated, source_count, status) + site metadata.
`/raw/<category>/<slug>.md`	`text/markdown`	`generate-agent-endpoints.py`	Byte-identical copy of each source `.md` (front-matter included) at a predictable URL.
`/sitemap.xml`	`application/xml`	MkDocs (built-in `sitemap` plugin)	XML sitemap for crawlers.
`/robots.txt`	`text/plain`	`generate-agent-endpoints.py`	Allow-all crawler policy.

URL conventions

HTML page URL: https://wiki.imparlabs.com/content/<category>/<slug>/ (content lives under docs/content/, a symlink into content/, so MkDocs publishes it at /content/<category>/<slug>/).
Raw markdown URL: https://wiki.imparlabs.com/raw/<category>/<slug>.md — a clean, predictable path that disambiguates same-named pages across categories (e.g. hardware/overview.md and inference-engines/overview.md would collide at /raw/overview.md; the full path avoids this).
llms.txt links point to the raw .md URLs, per the v2 spec recommendation that an llms.txt link list point to LLM-friendly (markdown) content so agents can ingest clean source without parsing HTML.

`/llms.txt` — the v2 spec

Verified against the authoritative spec at llmstxt.org (fetched during implementation, September 2024 proposal v2, August 2026 update). The format, in order, is:

An H1 with the name of the project/site — # Local AI Agent Wiki.
A blockquote short summary — > A knowledge wiki about running open-weight LLMs locally, built from verified research. Machine-readable for AI agents.
Zero or more non-heading markdown paragraphs of context.
Zero or more H2 sections (## Section Name) each containing a markdown list of [Page Title](url): optional one-line description.

The "Optional" section convention (for skippable secondary info) is not used here — every page is essential. The file is generated by scripts/generate-agent-endpoints.py from content front-matter, grouped into one H2 per wiki category (Inference Engines, Models, Hardware, Application Stack, Practice & Ops, Overview), with each entry linking to that page's raw .md endpoint.

Why a custom generator instead of only the mkdocs-llmstxt plugin? The Phase 0 architecture decision chose the mkdocs-llmstxt plugin to emit llms.txt. That plugin is configured in mkdocs.yml (owned by Phase 1b) and requires a sections block listing every page; the committed config omits it, so the build aborts on its own. generate-agent-endpoints.py is the authoritative, self-contained producer for the full agent layer: it reads content/ directly and can run as a post-build step, so the surface is verifiable without the mkdocs.yml edit. When the plugin is configured it emits its own copies; this script overwrites them with the spec-correct form (single source of truth = content front-matter).

`/llms-full.txt`

Every page's body markdown (front-matter stripped) concatenated in nav order, separated by blank lines. For agents that want the whole corpus offline in a single fetch.

`/index.json`

{
  "name": "Local AI Agent Wiki",
  "site_url": "https://wiki.imparlabs.com",
  "page_count": 52,
  "categories": ["Inference Engines", "Models", ...],
  "endpoints": { ... },
  "pages": [
    {
      "id": "inference-engines.engines-llama-cpp",
      "url": "https://wiki.imparlabs.com/content/inference-engines/engines-llama-cpp/",
      "raw_url": "https://wiki.imparlabs.com/raw/inference-engines/engines-llama-cpp.md",
      "title": "llama.cpp — The Substrate of Local Inference",
      "summary": "What llama.cpp is, ...",
      "category": "inference-engines",
      "tags": ["engine"],
      "last_updated": "2026-10-09",
      "source_count": 49,
      "status": "ready"
    },
    ...
  ]
}

This is the programmatic entry point a future MCP server or agent indexer should read first. last_updated comes from the front-matter date (set by the publish pipeline); status is one of verified / ready / partial / unverified.

Raw markdown endpoints

Each page is copied byte-for-byte from source/<rel>.md to raw/<rel>.md in the site output. The smoke test verifies the served bytes match the on-disk source for all 52 pages. The front-matter is preserved intentionally: it carries the machine-readable metadata (title, summary, tags, status, sources) an agent needs to evaluate a page without fetching the HTML.

`/sitemap.xml` and `/robots.txt`

MkDocs' built-in sitemap plugin emits /sitemap.xml (one <url> per HTML page plus the home/404). generate-agent-endpoints.py writes /robots.txt.

Robots policy decision: allow-all. The wiki's explicit purpose (from the project index: "anyone can point their agents to [the site]") is to be consumed by AI agents, so we do not block GPTBot, ClaudeBot, CCBot, or Googlebot. Each is listed explicitly with Allow: /, and a Sitemap: directive points crawlers at /sitemap.xml. This is the documented default from the Phase 0 research, which notes it "needs Paulo's explicit confirmation" — this implements that default; a future ops review can tighten it.

Generator

python3 scripts/generate-agent-endpoints.py [--content content] [--out site] [--site-url URL]

Reads content/**/*.md, validates required front-matter (title, category, tags, date, source_count, status, summary), and writes all non-HTML endpoints into --out (the MkDocs site_dir). It must run after mkdocs build so the HTML + sitemap.xml are present, and it overwrites the plugin's llms.txt/llms-full.txt with the spec-correct versions.

Integration into the publish pipeline

Per the Phase 0 publish_design, the future scripts/publish.py should: validate → copy raw .md → mkdocs build → generate-agent-endpoints.py → smoke-test → commit. This task delivers the generator and smoke-test so that pipeline step is a plug-in call.

MCP: recommendation (no build)

Recommendation: do not build an MCP server in this task. Ship the static endpoints first. This follows the Phase 0 architecture decision (docs/00-architecture-research.md, §3.4), summarised here:

MCP is mid-deprecation-cycle. The 2026-07-28 spec rewrote MCP as fully stateless and deprecated five features (Roots, Sampling, Logging, Dynamic Client Registration, legacy HTTP+SSE) with a 12-month window (source: Cloudflare, "The next generation of MCP"). Building against a moving target adds churn for zero content benefit.
MCP reaches a narrower audience than llms.txt. MCP requires (a) an agent harness that supports MCP, (b) a user who knows the server exists, and (c) has configured it. llms.txt works for any HTTP client that can fetch a URL. For a public knowledge wiki whose value is "anyone can point their agents at it", the floor matters more than the ceiling. The Astro case study is instructive: Astro removed its llms.txt in April 2026 to redirect effort to an MCP server, and its docs' agent-friendliness score fell from 78 (C) to 59 (F) as a result (Dachary Carey's write-up).
If/when added, it should be thin. The recommended design (already baked into this task's index.json) is a stdio or streamable-HTTP MCP server whose single search tool reads the generated index.json + fetches /raw/<slug>.md. Designing index.json as a first-class deliverable now keeps that door open at near-zero future cost.

A docs-MCP-server pattern is well-established (e.g. speakeasy-api/docs-mcp, johnmathews/unified-documentation-server, the Arxiv paper MCP Server Architecture Patterns for LLM-Integrated Applications 2026), confirming the pattern is sound — just not yet for this corpus, which is small (52 pages) and fully covered by the static layer.

Decision: static endpoints are the primary surface in this task. An MCP server is a legitimate addition for a future card — the index.json + raw .md surface is designed so it can be bolted on cheaply.