localdoc

Large language models that power coding agents are limited by the knowledge encoded during pretraining. When answering questions about library APIs, configuration options, or version-specific behavior, three strategies dominate in practice: (i) injecting entire documentation corpora into the prompt, (ii) delegating retrieval to a remote retrieval-augmented generation (RAG) service [1] [2], or (iii) relying on parametric memory alone. Each strategy exhibits well-known failure modes. Full-document injection is token-inefficient and frequently truncated; remote RAG introduces privacy, latency, and vendor-lock-in concerns; parametric recall is prone to outdated answers and confident hallucination.

localdoc occupies an intermediate design point. It indexes documentation once on the local machine and, at query time, returns a context pack: a token-budgeted, source-attributed set of excerpts. Critically, localdoc performs retrieval only. It does not invoke a language model to synthesize answers; the consuming agent remains responsible for generation. The design goal is therefore compression of documentation context, rather than conversational question answering over documents.

This article describes the system architecture, indexing pipeline, hybrid retrieval model, and the alternatives that shaped those choices.

Problem formulation

A classical RAG system retrieves a subset of a documentation corpus $\mathcal{D}$ and asks an LLM to answer from it [1]. localdoc exposes only the retrieval half: given a query $q$ and a token budget $B$ (default $2400$ ), it returns a context pack $P(q)$ that the agent injects into its working context. Generation stays with whatever model the agent already uses. This keeps the index offline, avoids shipping $\mathcal{D}$ to a third-party service, and matches how coding agents already work—inject evidence, then emit code.

In practice, localdoc can output this pack as either flat Markdown or structured JSON. Here is a truncated example of the JSON payload:

json
{
  "query": "How does Prisma migrate on first run?",
  "budget": 2400,
  "estimatedTokens": 412,
  "savedPercent": 0.83,
  "sections": [
    {
      "rank": 1,
      "title": "Runtime migrations",
      "sourceUri": "https://docs.example.com/deploy#migrations",
      "sectionPath": "Deploy > Runtime migrations",
      "text": "Prisma migrations ship inside the binary and apply at startup…"
    },
    {
      "rank": 2,
      "title": "Embedded schema",
      "sourceUri": "https://docs.example.com/schema",
      "sectionPath": "Schema > Embedding",
      "text": "The schema is compiled into the executable; end users never invoke the Prisma CLI…"
    }
  ]
}

In this example, savedPercent: 0.83 indicates that localdoc eliminated 83% of the token overhead compared to injecting the uncompressed document hits.

System architecture

localdoc is a single Bun-compiled executable designed around one shared retrieval path. There is no long-running server by default; Model Context Protocol (MCP) is opt-in via localdoc mcp serve [7]. Persistent state lives in a local libSQL database with FTS5 [6] and vector extensions.

Rendering diagram…

Layer	Technology
Runtime	Bun + TypeScript
Database	libSQL (`@libsql/client`), Prisma 7 schema
Lexical search	SQLite FTS5 (`unicode61`), weighted BM25 [3]
Dense search	`F32_BLOB` + `vector_top_k` / cosine
Chunking	Chonkie (`Recursive` / `Table` / `Code`)
Embeddings (default)	Model2Vec sidecar, `minishlab/potion-base-8M` [5]
Embeddings (optional)	OpenAI-compatible API
Rerank (optional)	Local MiniLM, Cohere, or `/rerank`
Crawl	HTTP fetch + robots/sitemap; Playwright on demand

Configuration resides at ~/.config/localdoc/config.yml. Extracted markdown caches under ~/.localdoc/extracted/; Model2Vec weights under ~/.localdoc/models/. Prisma migrations are embedded in the binary and applied at runtime, so end users never invoke the Prisma CLI.

Indexing pipeline

Ingest is orchestrated by ingestTarget (localdoc add / localdoc update). Each target kind—web, GitHub, local folder, or OpenAPI—has its own fetch path, but all converge on the same sanitize → hash → chunk → embed store loop. Source status moves through pending → indexing → ready (or error).

Rendering diagram…

Discovery

For web corpora, URL discovery follows a cascade that terminates at the first successful strategy [8]:

llms-full.txt — a single concatenated agent-oriented dump
llms.txt — a structured link list
sitemap.xml
nav-crawl — same-origin link following from the root

Preferring llms*.txt when present avoids a full site crawl and yields pages already curated for agents. Crawl policy defaults include a page cap of 500, concurrency 4, a 30s timeout, and robots.txt respect. Versioned and localized URL duplicates are collapsed in favor of unversioned, English-preferring pages.

GitHub sources use a shallow sparse clone biased toward documentation paths, with an HTTP API fallback. Folder ingest globs markdown, HTML, notebooks, source, and OpenAPI specs while skipping build artifacts and files larger than 2 MB. OpenAPI specifications are expanded into one document per operation, with local $ref resolution—atomic operations avoid pathological chunk boundaries and prevent diversity caps from collapsing an entire API surface into two hits.

Extraction and incremental invalidation

Pages are fetched over HTTP first. Playwright Chromium is downloaded on demand when the crawl policy is auto/always and the response fails, resembles a challenge page, or yields boilerplate-only content. Site adapters (Mintlify, GitBook, Docusaurus, ReadMe, Sphinx, and a generic fallback) convert HTML to sanitized markdown while preserving breadcrumb trails as section context.

Each sanitized page is keyed by its SHA-256 content hash. A document is skipped during update when the hash matches, the document status is ok, and the stored extractor_version equals the current extractor constant (presently 3). Bumping that constant forces rechunking when extract or sanitize rules change, even if remote bytes are identical. The flag --recreate bypasses this guard.

Structure-aware chunking

Naive fixed-window chunking is a poor fit for technical documentation, which interleaves prose, tables, and code. localdoc first partitions markdown into kind-tagged spans, then applies a specialized chunker:

Kind	Strategy	Default parameters
Prose	Recursive	size 512 characters, overlap 64 characters
Tables	Row-aware	3 rows per chunk
Code files	AST-/signature-aware	512–1024 characters
OpenAPI ops	Atomic	single chunk per operation

For code and OpenAPI units, the system stores an optional denser field embed_text (signatures and doc comments, or flattened YAML) used solely as the embedding input. Retrieval still returns the full readable body. The embedding string combines the header with either the dense embed_text (if present) or the fallback body:

Header(c) + (embed_text(c) || body(c))

where the header concatenates optional file path, document title, section path, and a short summary (≤ 300 characters). This asymmetric design—dense signals for retrieval, full text for the agent—addresses the empirical observation that raw code bodies and JSON schemas embed poorly.

Default vectors are produced by a Model2Vec Rust sidecar (potion-base-8M) in batches of 20 [5]. The model is weaker than large transformer embedders but inexpensive enough to index an entire documentation site on a laptop. An OpenAI-compatible path is available when semantic quality must dominate offline constraints.

Retrieval model

At query time, localdoc executes hybrid search followed by budgeted packing.

Rendering diagram…

Lexical retrieval

FTS5 MATCH queries are constructed with a multilingual stopword list and the unicode61 tokenizer [6]; English Porter stemming is intentionally omitted to preserve identifiers. Queries with two or three content terms use a proximity operator NEAR(..., 5); longer queries use conjunction with a disjunctive fallback. Ranking employs weighted BM25 [3] over four columns $(text, heading, section\_path, title)$ with weights

\mathbf{w} = (1.0,\; 8.0,\; 2.0,\; 6.0).

Elevating headings and titles relative to body text is intentional: documentation pages are often best retrieved by section identity rather than diffuse body overlap. Up to 40 lexical candidates are retained. Only documents with status ok participate. Whitespace-tokenized FTS is weak for Chinese, Japanese, and Korean (CJK) and other non-segmented scripts; dense recall is therefore essential for those queries.

Dense retrieval

The query is embedded with the same embedder used at index time. Approximate nearest neighbors via vector_top_k are preferred, then rescored by cosine similarity:

\mathrm{sim}(\mathbf{q}, \mathbf{v}) = \frac{\mathbf{q}\cdot\mathbf{v}}{\|\mathbf{q}\|\,\|\mathbf{v}\|}.

The candidate limit is 40. Source-filtered queries bypass ANN and brute-force the filtered subset: in practice, filtered ANN paths exhibited pathological latency relative to a small exact scan. Kind and keyword filters over-fetch from ANN (factors of 5–20) and post-filter in application code when possible.

Fusion and optional reranking

Lexical and dense rankings are combined with Reciprocal Rank Fusion (RRF) [4]. For each candidate $c$ appearing at ranks $r_{\mathrm{fts}}(c)$ and/or $r_{\mathrm{vec}}(c)$ ,

\mathrm{score}(c) = \sum_{s \in \{\mathrm{fts},\,\mathrm{vec}\}} \frac{1}{k + r_s(c) + 1},

with default $k = 60$ . RRF requires no learned fusion weights and remains robust when one modality fails. An optional second-stage reranker (local MiniLM, Cohere rerank-v3.5, or an OpenAI-style /rerank endpoint) may reorder a capped candidate set; it is disabled by default to preserve the offline-first posture.

Results are diversified with a per-document cap of 2 and truncated to 12 hits before packing.

Context packing

Packing maps the ranked list into budget $B$ . Token length is estimated as ceil(charCount / 4). Soft fairness constrains any single section to at most 45% of $B$ . Near-duplicates are dropped via containment and shared-prefix heuristics; truncation respects paragraph, then sentence, then word boundaries. The JSON pack exposes savedPercent = 1 - packTokens / rawHitTokens, a compression ratio against the tokens of the raw retrieved hits. That quantity makes the system’s purpose measurable: agents receive the same sources under a strictly smaller token envelope.

Agent integration

localdoc integrates at different levels of the developer loop: interactive investigation via the CLI/TUI, background context provider via MCP, and direct agent instructions via IDE skills.

CLI — add, update, query, list, inspect, doctor, fetch, mcp serve, tui
MCP — tools query, list, and inspect over stdio [7]
Skills — localdoc install-skill installs a SKILL.md for Cursor, Claude Code, Codex, OpenCode, and related environments

The skill workflow is deliberately conservative: enumerate indexed sources, query when present, and add a new source only with explicit user approval. This reduces accidental ingestion of untrusted documentation into the agent’s context channel.

Rejected alternatives

Several early designs were discarded before settling on the current shape.

Hosted chat over docs. The first prototype answered questions with a local LLM. That collapsed privacy goals (the corpus still had to be trusted by a generation stack), duplicated work coding agents already do, and made evaluation fuzzy—“good answer” instead of “good evidence under budget.” Retrieval-only packs won because the output is inspectable and model-agnostic.

Vectors alone. Pure dense search missed exact API identifiers, error codes, and protocol literals that BM25 recovers cheaply [3]. Adding FTS and fusing with RRF closed that gap without training a fusion model [4].

Symmetric embed = display text. Embedding raw code bodies and JSON schemas produced weak nearest neighbors. Splitting embed_text (signatures, doc comments, flattened YAML) from the full body fixed recall while keeping agent-facing text readable.

Always-on Playwright. Shipping Chromium in the release binary ballooned download size for sites that never needed it. On-demand install keeps the default path small; JS-heavy or challenge pages pay the cost only when the HTTP fetch fails.

Filtered ANN for source-scoped queries. Early vector queries tried to push source filters into ANN. On laptop-scale corpora the filtered path was slower and less predictable than an exact scan of the subset, so source-scoped dense search now brute-forces.

Always-on reranker. A second-stage MiniLM/Cohere pass helped on noisy corpora but broke the offline-first default and added latency to every query. It remains available, off by default.

Limitations

localdoc is optimized for workstation-scale documentation corpora, not for leaderboard retrieval metrics. A few constraints follow from that scope.

Embedding quality. The default Model2Vec model is a deliberate quality–throughput tradeoff: good enough to index a full docs site on a laptop, not competitive with large transformer embedders [5]. An OpenAI-compatible path exists when semantic quality must win.

Lexical gaps. FTS5 without language-specific segmentation remains weak for CJK and similar scripts; those queries lean on dense recall. Exact identifier search in English and code-heavy docs is the regime where BM25 shines.

Operational costs. The first Playwright download is heavy, paid only when HTTP extraction fails. Source-scoped dense queries skip ANN and scan the filtered subset instead—predictable on small corpora, not asymptotically elegant.

None of these are accidental. They are the price of keeping indexing offline, the binary small, and the default path free of cloud dependencies.

Conclusion

localdoc provides a fast, offline hybrid retrieval engine over documentation corpora, emitting token-budgeted context packs directly into coding agent prompts. By separating retrieval from generation, indexing structure-aware chunks with asymmetric embed text, and fusing BM25 with dense vectors via RRF, the system targets a pragmatic gap between stuffing entire sites into prompts and outsourcing RAG to the cloud [2]. The resulting artifact is a small, attributable evidence set—source-grounded context without shipping the corpus elsewhere.

Description