Indexing reference¶

Everything a question is answered from went through this pipeline: a repository is cloned, some of its files are kept, each kept file is cut into chunks, and the chunks are ranked against the question. This page states the rules and the numbers.

For why the corpus is defined the way it is, see what the bot is allowed to read. For what happens after retrieval, see how an answer is produced.

How a repository is fetched¶

A qualifying repository is shallow-cloned to disk (depth 1, single branch) from the default branch the forge reports, falling back to main when it reports none. The clone also holds the tag of the repository's latest release, fetched at depth 1, and no other tag. The clone is kept, under index.clones, and each later run fast-forwards it, so a source that has not moved costs one round trip rather than a clone.

With index.ssh.key set, every clone and fetch goes over SSH with that key, against host keys pinned in index.ssh.known_hosts. Without it they go over HTTPS. The two are never mixed: a source with no usable SSH URL fails rather than falling back.

Either way, the clone URL comes from the forge's API, so it is checked before a credential goes with it. It must name the configured forge (gitlab.url.api, or gitlab.com when that is unset), and over HTTPS it must use the same scheme, so the token is never sent in plain text. A source whose URL fails the check is reported as a failed fetch. An HTTPS fetch with no token sends nothing and is not checked.

A clone that cannot be fast-forwarded is deleted and cloned again. That covers a clone with local changes, an upstream whose history was rewritten, a default branch that moved, and a clone too damaged to open. A failure to reach the forge keeps the clone, since cloning again would only spend more of the same rate limit.

The corpus predicate runs before a repository is cloned, so everything on disk came from a source that qualified when it was fetched. The directory is created mode 0700. A source that stops qualifying keeps its clone until the next run that completes discovery, which deletes it. A crash, a stopped run or a failed discovery leaves it there for longer. Every source is public when it is cloned, so what lingers is content that was public.

A source whose verdict is unresolved keeps its clone. That verdict means a fact could not be checked this run, such as when discovery was rate limited or a source's zensical.toml could not be read. It does not mean the source stopped qualifying, so the clone is there for the next run.

Which commit each kind comes from¶

Code comes from the release because that is what people can install: indexed from the branch, the bot could describe a function or option that no release ships yet. Docs stay on the branch because most documentation sites deploy from it, and a doc fix is published without a release; pinning docs to a tag would hide that fix and make a citation disagree with the page it links to. The changelog is the same at the release and on the branch, since the release commit is what writes it.

The latest release is the forge's own answer, not the newest tag, so a tag whose binaries are not uploaded yet does not count. If the release cannot be looked up, for any reason but "there is none", the docs are still refreshed and the code stays at the last release it was indexed from; the failure shows in index status. A rate limit stops the run instead.

Each document is pinned to the commit it was read from, and every citation links to that commit. A source's tree is read only when its branch head or its release commit differs from what is already indexed. The first run after upgrading to this behaviour re-indexes every source once.

Up to two repositories are fetched at once. Indexing and every database write happen one source at a time.

Which sources are indexed¶

A repository is indexed when it is public, in the phpboyscout group, not archived, not excluded, and listed in the estate feed, projects.json, read from index.feed.url. The feed is canon: a repository it does not list is rejected with the clause feed, whatever else is true of it.

A listed repository with a documentation site is indexed in full. One with no site is indexed only in phpboyscout/go/, phpboyscout/rust/ and phpboyscout/images/, family members included, and then only for its README, code and changelog: a docs/ page with no published site has no page to cite. If a source loses its site but is still admitted, its doc documents are removed on the next run. index sources shows the clause that decided every repository.

Which files are indexed¶

Each file in the clone is classified by path. Only four kinds are indexed at all.

Everything else is classified none and never read.

What is deliberately not indexed¶

The 1 MiB bound is on the file, not the repository. Documentation and Go source sit far below it; what it catches is generated API dumps and vendored binaries.

Where markdown is cut into chunks¶

A chunk is the unit that gets retrieved, and for markdown it is a heading section. The limits below apply to markdown, READMEs and changelogs — Go source is chunked by a different rule and is not subject to the byte cap.

A section longer than 4000 bytes is split, and the parts overlap by 200 bytes so a passage lying across a boundary is still retrievable from one side. The splitter prefers a blank-line paragraph boundary when one falls in the second half of the window; failing that it cuts at the byte limit.

The limit is a byte budget rather than a token count. Tokenisation depends on which model is composing, and that can change; a byte bound is stable and close enough for a retrieval unit.

A heading with nothing under it is merged into the chunk before it. On its own it would be a chunk that matches on its title and then says nothing, which is the worst thing an index can return, because it looks like an answer.

The anchor a citation lands on¶

Each chunk carries the URL fragment its heading renders as. Punctuation is removed rather than replaced: ReleaseSource.Private becomes releasesourceprivate, not releasesource-private. Letters, digits, - and _ survive; whitespace becomes a single -; everything else disappears.

Where Go source is cut into chunks¶

Go files are parsed rather than split textually. One chunk per symbol:

• The package doc comment becomes the first chunk, titled package <name>.
• Every exported function, type, and variable becomes a chunk holding its doc comment and its signature — not its body. Bodies are imports, struct tags and boilerplate: a large volume of low-signal tokens competing with real documentation for the same ranking.
• Every constant is indexed, exported or not. A constant holds a default, a threshold or an interval, and its value is what "what is the default X?" is actually asking for.
• Unexported functions, types and variables are left out.

Each code chunk also carries the split form of its identifier as search-only text — defaultTakeCount is reachable by typing "default take count" — indexed but never displayed and never sent to the model as prose.

Code chunks are not bounded by the 4000-byte cap. That limit lives in the markdown splitter and nothing applies it here, so a long package doc comment becomes one long chunk: the largest in an August 2026 build was 5450 bytes, against a 4000-byte ceiling for every markdown chunk in the same index.

How chunks are ranked¶

Retrieval is BM25 over SQLite FTS5. There is no vector search, no embedding model and no re-ranker.

A heading match is stronger evidence than a body match: a section titled "Hot reload" is about hot reloading, where a page mentioning it in passing is not. Search terms sit between the two — a split identifier is strong evidence, but it is derived rather than written, so it does not outrank a heading somebody chose.

The tokenizer is porter unicode61, so terms are stemmed.

Raw FTS5 bm25() scores are negative, more negative meaning a better match. phpbotscout flips them once at the boundary, so every score you see — in ask --retrieval-only, in a decline's closest passages — is positive and higher-is-better.

How a question becomes a query¶

The question is not passed to FTS5 as typed. It is tokenised into words, each word is lowercased and quoted as a literal, and the terms are joined with OR.

Function words go because headings outweigh bodies ten to one, and many pages have FAQ headings phrased as questions. Searched for, "what can I put in a message" matched "Can I put a circuit breaker on my server?" better than the page about messages. A question made only of function words, such as "how are you", is searched for as typed.

Four consequences worth knowing:

• Punctuation cannot reach the query language. Passing a question through unchanged does not merely risk odd results, it fails: how do I hot-reload config? errors with no such column: reload, and an apostrophe is a syntax error.
• hot-reload searches for hot and reload, and so matches a page that writes it as two words.
• OR, not AND. BM25 does the ranking; requiring every term would turn one unusual word into no results at all.

A question containing no letters or digits produces an empty query, and retrieval returns nothing rather than matching everything.

How the passages are chosen¶

An answer is composed from 8 passages.

• No document takes more than two slots. A changelog's entries all match a question alike, and without the cap one changelog could fill all eight.
• A named module's slots are searched without its name. Inside go/config every page belongs to go/config, so "config" tells pages apart only where a heading happens to contain it. "How does config precedence work in go/config" searches go/config's pages for "precedence".
• The remaining slots search with the full question, name included, because there the name is what finds the module's mentions in other projects.

How a citation URL is built¶

Blob links are pinned to the commit that was indexed, not to a branch. A citation that silently follows a branch can end up pointing at text contradicting the answer it was supporting.

Documentation URLs are derived from the docsite layout rather than fetched: docs/ is the site root, index.md is a directory's own page, and every other page becomes a directory named after the file.

What the index holds¶

SQLite, one file, at index.path, shared with the questions, answers, votes and review verdicts serve and review record. The tables that make up the index are these:

Answers only cite documents from a source that qualifies. A source that stops qualifying keeps its rows until something replaces them, but nothing is retrieved from them.

Re-indexing a source removes its documents for files that are no longer in the repository, or that no longer yield a document, such as a Go file that stopped parsing. That happens only once every file in the new commit has been written, so a run that fails part-way removes nothing. A source is re-indexed when its commit moves, or by index build, which re-indexes every source.

Rejected sources are stored on purpose: that is what lets index sources explain an absence instead of just omitting it.

Scale, as measured¶

A full build of the phpboyscout namespace in August 2026: 53 qualifying sources, 1786 documents, 11928 chunks, in a little over a minute. Roughly 6.7 chunks per document.

Those numbers move with the estate; treat them as an order of magnitude, and run index status for the current ones.