How an answer is produced¶

A question takes one of three routes: it is answered with citations, it is declined, or something breaks. This page explains the path, and the reasoning behind the parts of it that surprise people.

The path a question takes¶

1. The question becomes a search query. It is tokenised into words, each quoted as a literal term, and the terms joined with OR. Nothing from the question survives as query syntax.
2. Passages are retrieved and ranked — eight of them by default — by BM25, weighted so a heading match counts for more than a body match.
3. If nothing is retrieved, the question is declined without a model being called. There is nothing to ground an answer in, so there is nothing worth asking about.
4. The passages and the question go to the model, numbered, each fenced between explicit delimiters, under instructions to answer only from them.
5. The model answers or declines, returning structured output: the prose, the passage numbers it used, and a decline flag with a reason.
6. The citations are checked. An answer citing nothing is refused. An answer citing a number outside the set it was given is refused.
7. A refused answer becomes a decline, carrying the reason it was refused.

Why declining is a result, not a failure¶

A component that can only produce answers produces one for every question — including questions whose correct answer is "we do not know". Those answers arrive carrying a real citation to a page that does not say what was claimed, which is worse than silence: every signal a reader uses to judge trustworthiness is intact, and only the content is wrong.

So declining is a first-class outcome. Three situations produce one:

• Nothing matched. Retrieval found no relevant passages.
• The composer declined. The passages discuss the subject but do not answer the question.
• The answer could not be grounded. Prose was drafted, but it cited nothing or cited something outside the retrieved set.

An error — a non-zero exit — is reserved for something genuinely breaking: the index unreachable, the provider failing. An error never means "we do not know". Keeping those distinct is what lets a knowledge gap be told apart from an incident, and the gap backlog depends on it: a declined question is a documentation gap with a timestamp on it.

Why there is no relevance threshold¶

The obvious design is a score floor: if the best passage scores below x, decline. It does not work here.

Calibration against the curated corpus showed that no score separates a question the corpus answers from one whose subject it merely covers. The two bands overlap, and a floor set high enough to exclude the unanswerable questions takes about 40% of the answerable ones with it. BM25 measures term overlap; it does not measure whether a passage answers anything.

So the judgement is made where the evidence is — against the retrieved passages themselves, by the model, with declining explicitly available to it. A floor is still supported internally to discard outright noise, but it is not the gate and is not exposed as a configuration key.

The most damaging mistake in this whole path is the one that looks best: a question whose subject the passages discuss, answered as though they addressed it. The instruction the composer runs under names that case specifically, because it is both the most common and the hardest to spot after the fact.

Why contradicting the question is a good answer¶

If the passages contradict the question's premise, that is an answer, not a decline. "There is no hosted service" and "no such option exists" are useful replies; declining would leave the asker believing the thing exists and that the bot simply failed to find it.

This is why a documented limitation is worth more than another feature description. The bot cannot infer an absence from silence — it can only report one that somebody wrote down. Everything on what phpbotscout does not do exists to be that written-down absence.

Why passages are fenced, and what that does not fix¶

Passage text is untrusted. It comes from public repositories anyone may open a merge request against, and once questions arrive from a public Discord channel, the question is untrusted too.

Each passage is wrapped in explicit -----BEGIN PASSAGE----- / -----END PASSAGE----- delimiters, and the instruction states that text inside a passage is never an instruction, whatever it says.

Delimiting does not make prompt injection impossible. What it does is make the boundary explicit and enforceable — it is the part a prompt can actually hold. The content form of the same attack — plausible, wrong prose rather than a planted instruction — is not defended by prompting at all, and is handled structurally instead by keeping issue text out of the corpus entirely. See what the bot is allowed to read.

Why citations are numbers¶

The model is given numbered passages and must cite by number. That makes a citation checkable rather than a claim: every number it returns is verified against the passages actually supplied, and anything outside that set is refused.

A citation that resolves to a real page is the strongest trust signal an answer has, which is exactly why it is the one worth verifying mechanically. An answer nobody can check is a claim wearing evidence.

Citations point at the commit that was indexed rather than at a branch, for the same reason: a link that silently follows a branch can end up pointing at text that contradicts the answer it was supporting.

Why a small model is enough¶

The provider is configurable, and what runs by default is claude-local — a locally installed, already-authenticated claude binary — which keeps development off metered APIs. The model selected for answering is Claude Haiku 4.5 (claude-haiku-4-5), through Anthropic's API.

That choice came from measurement rather than preference. Three tiers were run over the calibration corpus: the middle tier scored identically to the small one on both answerable bands while costing four times as much, and the largest bought a marginal recall gain at 7.6 times the cost — more than its list-price ratio, because it emits far more output tokens per answer.

The reason a small model holds up is structural. Composition here is not the hard part: retrieval has already found the passages, the instruction is narrow, and the model's job is to write a short answer from text placed in front of it and to notice when that text does not answer the question. A larger model is better at knowing things, and knowing things is precisely what this component is forbidden from doing.

It is a decision with a stated trigger for revisiting: once citation accuracy can be measured directly, a model that cites more faithfully would be worth paying for.

What this design does not give you¶

• No memory. Each question is answered from its own passages. Nothing is carried between questions, by design as much as by economy: in a shared channel, one person's question becoming context for the next person's is a privacy problem before it is a quality one.
• No reasoning beyond the passages. The model is instructed to ignore anything it knows about these projects from elsewhere. A correct answer that is not in the corpus is still a decline.
• No semantic search. Retrieval is keyword-based; a question sharing no vocabulary with the page that answers it will not find it. See what phpbotscout does not do.