Knowledge index and the ask command¶

Status: APPROVED 2026-07-28. Phase: 2 (delivery plan). Needs 0.3, 0.7. Nothing from Track A. Implements: R-KB-1 through R-KB-7, R-KB-11, R-ANS-5, R-ANS-5a, R-ANS-5b, R-ANS-8, R-ANS-9, R-CLI-1, R-OPS-25, R-OPS-26, R-OPS-28.

1. What this phase is¶

The bot learns what it knows. It discovers which repositories qualify, reads their documentation, indexes it, and answers a question from the terminal with citations.

No Discord. Phase 1b reads a channel and Phase 3 will join the two, but this phase is measured entirely from the command line — which is deliberate, because it is the phase that answers the product's real question: can keyword retrieval over this corpus actually answer the questions people ask? Everything downstream assumes the answer is yes, and nothing before this phase tests it.

The most valuable phase in the plan, and the one most worth being honest about. If the benchmark says retrieval is weak, that is a finding, not a failure — it is what 2c exists to act on.

2. Corpus discovery¶

The corpus is a predicate, not a list (R-KB-1). A source qualifies when all hold:

1. it is in the phpboyscout GitLab group, including subgroups;
2. it carries a zensical.toml;
3. it publishes a reachable GitLab Pages site;
4. the API reports visibility public;
5. it is not excluded.

Exclusions are evaluated first and are absolute (R-KB-2). This ordering is not a detail. phpboyscout/infra satisfies the predicate — it is public, it is documented, it has a zensical site — and it must never be indexed. An exclusion applied after discovery is one that can be forgotten; applied before, it cannot.

index sources explains each verdict, naming which clause admitted or rejected every candidate. A predicate nobody can inspect is a curated list with extra steps.

Enumeration depends on an upstream capability¶

Discovery must enumerate the namespace from the API, never from disk. A checkout cannot say whether a repository is still public, still in the group, or even which project it is — several working directories in this estate resolve to a repository with a different name, and one made private yesterday looks identical on disk. Since the predicate is a security boundary, the visibility check has to be live.

forge has no such capability today: it is release-focused. Adding it is tracked upstream as forge#2, which unparks the Repositories and Contents capabilities the provider-contract-widening spec left waiting for a concrete consumer.

phpbotscout therefore declares the narrow interface it needs and builds discovery against it, exercised by a fake. No forge SDK enters this repository — the depfootprint guard would fail if one did — and wiring the released capability is an adapter, not a rewrite. Until it lands, index build cannot enumerate for real; everything downstream of enumeration is testable without it.

What is fetched¶

Qualifying repositories are cloned shallowly and in memory through gitlab.com/phpboyscout/go/repo, which already provides exactly this and keeps a forge SDK out of our graph. The clone yields the commit SHA that provenance requires (R-KB-5).

Markdown source, not the rendered site. The rendered page carries the navigation, sidebar and footer on every single page, so shared chrome inflates term frequencies across the entire corpus and dilutes exactly the ranking the index exists to provide. Markdown is also already structured for chunking, and R-KB-1a already mandates markdown-only for the blog — so this makes the two consistent rather than special-casing one.

Citations still point at the public site. The mapping is deterministic because we control the toolchain:

docs/how-to/01-author-a-provider.md
  → https://chat-platform.go.phpboyscout.uk/how-to/01-author-a-provider/

docs/index.md          → <site>/
docs/reference/index.md → <site>/reference/

The site's base URL comes from zensical.toml's site_url, which every qualifying repository has by construction — it is part of the predicate.

3. Chunking¶

One chunk per H2/H3 section, carrying its heading path as context, never split inside a fenced code block.

This is the phase's largest quality lever, and it is also the citation decision. A whole-page chunk dilutes term frequency until a long relevant page ranks below a short tangential one, and it can only cite the page — leaving the asker to find the paragraph. A section chunk scores against focused text and cites the exact anchor zensical already generates:

chunk   Author a provider › 2. Construction must not need credentials
text    New validates configuration. Connect reaches out. Nothing in
        construction may require a working token…
cite    …/how-to/01-author-a-provider/#2-construction-must-not-need-credentials

Two boundary rules, both about not producing chunks that match everything and say nothing:

• A section longer than the chunk ceiling splits with overlap, keeping its heading path.
• A heading with almost nothing under it merges upward into its parent.

4. Code is indexed as prose, or not at all¶

Doc comments and exported declarations only — the package doc, and each exported symbol's doc comment and signature. Test files and anything marked generated are skipped.

Doc comments are prose written to be read, so BM25 scores them like documentation. Raw bodies are imports, struct tags and boilerplate: a large volume of low-signal tokens competing for the same ranking as the documentation. This estate writes unusually thorough doc comments, which makes the trade clearly worth taking here even where it might not be elsewhere.

Code citations line-anchor to the GitLab blob at the indexed SHA, so a citation keeps meaning after the file moves on.

5. Storage¶

One SQLite database for the index and every other piece of persistent state (R-OPS-25) — FTS5 for retrieval, ordinary tables for everything else. One writer by construction, so SQLite's headline weakness does not apply.

Driver: pure Go, and cgo is not needed after all¶

R-OPS-26 permits cgo with the native driver and names modernc.org/sqlite as the fallback. Verified 2026-07-28: the fallback is sufficient. modernc.org/sqlite ships FTS5 with BM25 built in — no cgo, no build tag:

query: reload OR config       (lower score = better)
  Hot reload                     -2.1346
  Precedence                     -0.0000
  Write fidelity                 -0.0000

The ranking is real, and the near-zero scores are correct BM25: config appears in most documents so its IDF collapses, while reload is rare and therefore discriminating. Column weights work too — bm25(chunks, 10.0, 1.0) weights a heading match above a body match.

So CGO_ENABLED=0 stays, which keeps the build simple and cross-compilation free. This is recorded as a finding rather than a preference: R-OPS-26 anticipated possibly needing cgo, and it turns out we do not.

Schema¶

Versioned migrations from the first commit, applied on startup, never hand-edited (R-OPS-28). Twelve-month audit retention means the schema outlives several releases of the code that writes to it.

sources     -- one row per qualifying repository, with its verdict and last refresh
documents   -- one row per file, with path, site URL, commit SHA, indexed_at
chunks      -- one row per section: heading path, body, byte range, anchor
chunks_fts  -- FTS5 virtual table over (heading, body), external content
schema_migrations

chunks_fts uses external-content so the text is not stored twice, and provenance lives on documents where a citation can join to it.

Migrations are embedded SQL applied in order under one transaction each, with the applied version recorded. A migration runner for a single-writer SQLite database is small enough that a dependency would cost more than it saves.

6. The Retriever contract¶

Ships this phase regardless of what the hybrid spike later concludes (R-KB-11):

type Retriever interface {
    Retrieve(ctx context.Context, query string, limit int) ([]Result, error)
}

type Result struct {
    Text    string
    Score   float64
    Source  Provenance   // repo, path, site URL, anchor, commit SHA, indexed_at
}

FTS5 is the first and only implementation. The contract exists so 2c can measure a vector retriever against the identical path rather than building it twice, and so the benchmark harness depends on an interface rather than on SQLite.

Scores are normalised on the way out. FTS5's bm25() returns more negative is better, which is correct but a trap for every caller that assumes higher is better — including the relevance threshold in R-ANS-6. The contract exposes a score where higher is better, once, at the boundary.

7. The Composer seam¶

Demonstrated 2026-07-28 against claude-local. One passage — phpbotscout's own policy page — asked two questions:

Same passage, same retrieval score, opposite outcomes. That is the case a threshold cannot reach, and it is cal-32 — the question that scored 29.01 and motivated all of this.

The composer decides sufficiency, because the score cannot. Calibration showed retrieval scores do not separate answerable questions from unanswerable ones (Q-2-7), so "did we retrieve something that answers this?" is a judgement the composer must make and be able to answer no to. An answer that cannot be supported by the retrieved passages is not returned.

type Composer interface {
    Compose(ctx context.Context, question string, results []Result) (Answer, error)
}

Phase 2 wires one provisional provider through gitlab.com/phpboyscout/go/chat to make ask answer. That choice is explicitly not a decision — 2b swaps implementations through this seam and records the verdict.

The seam is what makes R-CLI-1 honest: ask and the eventual Discord flow share one path, and 2b benchmarks that path rather than a reconstruction of it.

Every answer carries citations naming the retrieved chunk's anchor (R-ANS-9). An answer that cannot cite is not returned; it becomes a knowledge gap, which is the outcome the product is built to surface.

8. Commands¶

ask --retrieval-only prints the ranked chunks and scores without composing, which is how retrieval quality is measured independently of whichever provider happens to be wired.

9. The refresher service¶

serve gains a second managed service alongside the Discord gateway from 1b: a ticker calling the same Refresh the CLI command calls. One code path, so the scheduled and manual routes cannot drift.

Refresh interval is configurable per source (R-KB-4). Indexing is incremental and resumable, and a failed run leaves the previous index intact (R-KB-7) — each source's update commits in its own transaction, so a failure part-way through a run costs that source's refresh, not the index.

Staleness is reported rather than hidden (R-KB-6): answering from a stale index with a caveat beats not answering.

10. Resolved questions¶

All six resolved 2026-07-28.

Q-2-1 — what ask does in Phase 2. The full path with composition behind a seam, one provisional provider. Phase 2's exit criterion says ask answers with citations, but the provider is not chosen until 2b, which itself runs candidates "through the identical retrieval path". A seam satisfies both: the path exists once, and 2b swaps implementations through it.

Q-2-2 — index source. Markdown, citation URLs derived. Rendered pages repeat navigation and footer chrome on every page, which distorts term frequency corpus-wide; markdown is already structured for chunking; and R-KB-1a already requires markdown-only for the blog.

Q-2-3 — chunking. One chunk per H2/H3 section, heading path retained, never splitting inside a fenced block. The chunk boundary and the citation anchor are the same decision.

Q-2-4 — code. Doc comments and exported declarations only. Raw bodies are volume without signal for the questions a support bot receives.

Q-2-5 — signposts. Deferred to Phase 3. Surfaced while walking this: R-KB-1d's third check — whether the target actually addresses the question — is question-dependent, so signpost validation can never be purely an indexing operation. It is split across index time and answer time by nature, which places it with the answering path. Recorded so Phase 3 does not rediscover it.

Q-2-6 — refresh. Both the CLI commands and a scheduled service, sharing one code path.

11. Open questions¶

Q-2-7 — RESOLVED, negatively: a BM25 score cannot be the threshold.

Measured 2026-07-28 against the curated calibration corpus — 36 questions in three bands, over 500 documents and 3,072 chunks from ten projects.

Re-measured 2026-07-28 with code indexed — 1,068 documents, 6,015 chunks. The code band improved (mean 30.52 → 35.94) and, more usefully, six of ten code questions now return an actual symbol rather than tangentially related prose. The overlap is unchanged: excluding every unanswerable question still costs 37% of the answerable ones, down from 40%. Adding a whole new content type moved the ranking without moving the conclusion, which is what "structural" means here.

The bands overlap and no threshold separates them. The best-scoring unanswerable question beats eleven of the thirty answerable ones; excluding every unanswerable question would cost 37% of the answerable ones. Even the docs band alone does not separate — its worst is 26.77 against an unanswerable high of 34.88.

The reason is structural rather than a tuning failure. BM25 measures term overlap, not whether a passage answers a question. "Which exact AI model is phpbotscout using in production today?" scores 29.01 against phpbotscout's own policy page, because it is about phpbotscout and moderation and questions — and that page does not say. A high score on a topically related passage that does not contain the answer is the exact shape of the failure, and it is what a score cannot see.

Consequences, and they are structural:

• R-ANS-6's "below a relevance threshold, escalate to the tool loop" cannot key on a raw retrieval score. The gate has to be a judgement about sufficiency, not a number about term overlap.
• The Composer therefore carries a real responsibility: deciding whether the retrieved passages actually answer the question, and declining when they do not. That is what turns an unanswerable question into a knowledge gap rather than into a confident wrong answer with a genuine citation.
• This is measured evidence for 2c before the spike begins. It does not show semantic retrieval would fix it — an embedding of a topically-related passage is also topically related — but it does show keyword scoring alone cannot, which is what R-KB-8 asks the spike to establish.

A configurable floor still ships, to discard genuine noise. It is a floor, not a decision procedure, and the spec no longer pretends otherwise.

Q-2-7a — the original question, retained for context: how would a threshold be set? R-ANS-6 sends anything below a threshold to the agentic tool loop, and R-ANS-13a governs hedging. A normalised BM25 score has no natural cut-off — it depends on corpus size and query length. The proposal is to derive it from the benchmark corpus rather than guess: run all 30 questions, plot score against whether the expected citation was retrieved, and pick the threshold that separates them. That cannot be done until 0.3 is complete, so this phase ships a configurable threshold with a provisional default and the calibration is an explicit task at the end of the phase.

Q-2-9 — do link-list sections need suppressing? First retrieval against the real corpus — 563 chunks from three repositories — surfaced a consistent failure shape. Sections that are lists of links ("Next", "Related", "Where next", "See also") rank well and answer nothing: they are dense with terms and empty of prose, which is precisely what BM25 rewards. Two of six trial questions were won by one.

Q: how do I authenticate to a forge?
   11.68  Getting started › Where next
   11.66  Credential pinning › Related

The right answer, how-to/authenticate.md, placed nowhere. Candidate remedies are a link-density penalty at index time, excluding sections whose body is mostly links, or leaving it to the composer to discard. Deliberately not fixed yet: the honest order is to finish 0.3, measure it against the benchmark, and tune against a number rather than against six ad-hoc questions. Recorded now so the calibration task has a hypothesis to test rather than starting cold.

Q-2-8 — does ask need the index to be fresh? If the index is empty or badly stale, should ask refuse, warn, or answer anyway? R-KB-6 says degrade with a caveat, which suggests answering with a warning — but an empty index is different from a stale one, and answering confidently from nothing is the worst available outcome.

11a. Measured behaviour, 2026-07-28¶

The full calibration set through ask, against 1,130 documents and 6,897 chunks.

The one docs decline is correct. The per-platform link-clickability detail it was asked for exists only in docs/development/specs/0008-tiktok.md, which the corpus deliberately excludes — every file in that repository containing the word is under docs/development. The bot reported that the user-facing documentation does not cover it, which is true, and which is the knowledge-gap signal this product exists to produce.

Identifiers are split into their words for search. The tokenizer treats defaultTakeCount as a single token, so a question asking about the "default take count" could not reach it — and the natural phrasing is what a question always contains, never the identifier. Confirmed directly: defaultTakeCount retrieved the symbol at 17.30, default take count retrieved something else entirely. The split form is stored in a search-only column, indexed and never shown, so it reaches nobody as though it were prose.

Unexported constants are indexed; unexported functions and types are not. Every code question that failed the first run asked "what is the default X?" — a take count, a heartbeat interval, a byte limit — and every one of those answers lives in a constant. Exportedness describes API surface, not whether a value is a behaviour people ask about. Indexing them moved the code band from 5/10 to 8/10, with cal-28 now reporting this repository's own sectionTop = 2, sectionBottom = 3 correctly.

"Unanswerable" splits in two, and the split explains an instability. Three of the six have false premises the corpus can refute — there is no hosted Keryx service, there is no username-based moderation lookup, the AI provider is an unfilled placeholder. Those three oscillated between declining and answering across runs, because both readings are defensible and nothing said which to prefer. The prompt now says a passage contradicting the question's premise is an answer: refuting a false premise is more useful than shrugging, and it removed the oscillation entirely — three runs each, no variance.

The remaining three decline consistently. They are genuine gaps: an unreleased GA date, other people's private pipelines, an external approval decision.

The last code decline is a vocabulary mismatch, and is left alone deliberately. cal-29 asks how many "repeated unanchored spoken observations" corroborate a spelling. The answer is defaultCorroboration = 5, whose doc comment says "a name said five times" — sharing almost no vocabulary with the question. It is indexed, findable by direct wording, and ranks 13^th for the question as asked, behind symbols that match the rarer terms. BM25 is behaving correctly and cannot bridge that gap.

Raising the retrieval limit would fix this one question and cost tokens on every other, which is fitting to a sample of one. It is recorded instead as the first clean vocabulary-mismatch case in the set, which is exactly the measurement R-KB-8 asks the 2c spike to make.

Generated godoc was considered and rejected on measurement. It renders the package doc, exported symbols, doc comments and signatures — what is already extracted here, minus the unexported constants that turned out to be several of the answers. The one genuine addition is Example functions, and the estate has eight of them across every repository, in packages no calibration question asks about.

Worth noting what the provider question now returns. It finds the placeholder in the policy page, says the provider has not been chosen, and then explicitly rejects a distractor — an unrelated --provider openai --model "gpt-5.4" example belonging to the documentation CLI. It located the plausible wrong answer and explained why it does not apply, which is the behaviour R-ANS-6b was written to get.

12. Exit criteria¶

1. index sources lists every qualifying repository and explains each verdict, including why each excluded source was excluded.
2. index build produces an index; index refresh re-reads only changed sources.
3. A failed refresh leaves the previous index queryable.
4. ask answers benchmark questions with correct citations, measured against the R-OPS-13 corpus.
5. ask --retrieval-only reports the ranked chunks, so retrieval is measurable independently of composition.
6. The retrieval benchmark is committed and re-runnable, reporting recall against expected citations — this is what 2b and 2c both measure against.
7. serve refreshes on schedule and reports staleness.
8. pkg/ packages hold ≥90% coverage; no private repository can appear in the index (R-ANS-8), asserted by a test against the exclusion list and the visibility check.

13. Testing¶

Unit — discovery predicate, exclusion ordering, chunking boundaries, the docs→URL mapping, and score normalisation, all against fixtures. No network.

The exclusion test is the one that matters. A private or excluded repository reaching the index is a P1 security defect, not a bug, so the predicate is tested adversarially: an excluded source that satisfies every other clause, a source that becomes public after being excluded, and a source whose zensical.toml appears without a public site.

Integration — against the live phpboyscout group, gated on INT_TEST_GITLAB=1, asserting that discovery finds the sources 0.7 enumerated and no others.

Benchmark — the 30-question corpus, reporting recall of expected citations. Committed and re-runnable so 2b and 2c measure against the same figure rather than a remembered one.

BDD — features/knowledge-index.feature for the operator-facing flows: building, refreshing, asking, and asking against a stale index.