Indexing reference¶

Everything a question is answered from went through this pipeline: a repository is cloned, some of its files are kept, each kept file is cut into chunks, and the chunks are ranked against the question. This page states the rules and the numbers.

For why the corpus is defined the way it is, see what the bot is allowed to read. For what happens after retrieval, see how an answer is produced.

How a repository is fetched¶

A qualifying repository is shallow-cloned into memory — depth 1, single branch, no tags — from the default branch the forge reports, falling back to main when it reports none. Nothing is written to disk: the corpus predicate is a security boundary, and content that never touches the filesystem cannot be left behind by a crash between the clone and the check that should have rejected it.

The commit at the head of that clone is what gets recorded against the source, and what every citation from it is pinned to.

Which files are indexed¶

Each file in the clone is classified by path. Only four kinds are indexed at all.

Everything else is classified none and never read.

What is deliberately not indexed¶

The 1 MiB bound is on the file, not the repository. Documentation and Go source sit far below it; what it catches is generated API dumps and vendored binaries.

Where markdown is cut into chunks¶

A chunk is the unit that gets retrieved, and for markdown it is a heading section. The limits below apply to markdown, READMEs and changelogs — Go source is chunked by a different rule and is not subject to the byte cap.

A section longer than 4000 bytes is split, and the parts overlap by 200 bytes so a passage lying across a boundary is still retrievable from one side. The splitter prefers a blank-line paragraph boundary when one falls in the second half of the window; failing that it cuts at the byte limit.

The limit is a byte budget rather than a token count. Tokenisation depends on which model is composing, and that can change; a byte bound is stable and close enough for a retrieval unit.

A heading with nothing under it is merged into the chunk before it. On its own it would be a chunk that matches on its title and then says nothing, which is the worst thing an index can return, because it looks like an answer.

The anchor a citation lands on¶

Each chunk carries the URL fragment its heading renders as. Punctuation is removed rather than replaced: ReleaseSource.Private becomes releasesourceprivate, not releasesource-private. Letters, digits, - and _ survive; whitespace becomes a single -; everything else disappears.

Where Go source is cut into chunks¶

Go files are parsed rather than split textually. One chunk per symbol:

• The package doc comment becomes the first chunk, titled package <name>.
• Every exported function, type, and variable becomes a chunk holding its doc comment and its signature — not its body. Bodies are imports, struct tags and boilerplate: a large volume of low-signal tokens competing with real documentation for the same ranking.
• Every constant is indexed, exported or not. A constant holds a default, a threshold or an interval, and its value is what "what is the default X?" is actually asking for.
• Unexported functions, types and variables are left out.

Each code chunk also carries the split form of its identifier as search-only text — defaultTakeCount is reachable by typing "default take count" — indexed but never displayed and never sent to the model as prose.

Code chunks are not bounded by the 4000-byte cap. That limit lives in the markdown splitter and nothing applies it here, so a long package doc comment becomes one long chunk: the largest in an August 2026 build was 5450 bytes, against a 4000-byte ceiling for every markdown chunk in the same index.

How chunks are ranked¶

Retrieval is BM25 over SQLite FTS5. There is no vector search, no embedding model and no re-ranker.

A heading match is stronger evidence than a body match: a section titled "Hot reload" is about hot reloading, where a page mentioning it in passing is not. Search terms sit between the two — a split identifier is strong evidence, but it is derived rather than written, so it does not outrank a heading somebody chose.

The tokenizer is porter unicode61, so terms are stemmed.

Raw FTS5 bm25() scores are negative, more negative meaning a better match. phpbotscout flips them once at the boundary, so every score you see — in ask --retrieval-only, in a decline's closest passages — is positive and higher-is-better.

How a question becomes a query¶

The question is not passed to FTS5 as typed. It is tokenised into words, each word is lowercased and quoted as a literal, and the terms are joined with OR.

Three consequences worth knowing:

• Punctuation cannot reach the query language. Passing a question through unchanged does not merely risk odd results, it fails: how do I hot-reload config? errors with no such column: reload, and an apostrophe is a syntax error.
• hot-reload searches for hot and reload, and so matches a page that writes it as two words.
• OR, not AND. BM25 does the ranking; requiring every term would turn one unusual word into no results at all.

A question containing no letters or digits produces an empty query, and retrieval returns nothing rather than matching everything.

How a citation URL is built¶

Blob links are pinned to the commit that was indexed, not to a branch. A citation that silently follows a branch can end up pointing at text contradicting the answer it was supporting.

Documentation URLs are derived from the docsite layout rather than fetched: docs/ is the site root, index.md is a directory's own page, and every other page becomes a directory named after the file.

What the index holds¶

SQLite, one file, at index.path. Schema version 3.

Rejected sources are stored on purpose: that is what lets index sources explain an absence instead of just omitting it.

Scale, as measured¶

A full build of the phpboyscout namespace in August 2026: 53 qualifying sources, 1786 documents, 11928 chunks, in a little over a minute. Roughly 6.7 chunks per document.

Those numbers move with the estate; treat them as an order of magnitude, and run index status for the current ones.