RAG in TrueBearing AI
An AI on its own will confidently make up immigration rules. So we never let it answer from memory. Here’s the trick that keeps every answer real and traceable.
A The library it draws from
Every file in okf/ is one small, true fact plus the official page it came from. Think index cards: fact on the front, source on the back.
FSW and CEC (TEER 0/1) require CLB 7 in all four abilities. Ability is scored per skill — the lowest one governs eligibility.
resource: URL becomes the citation on every answer.The shape of those files is not something I invented. It follows the Open Knowledge Format — markdown with YAML frontmatter, one concept per document, a resource field pointing at the authoritative source, and markdown links between concepts. The reason it fits a RAG system so well is that OKF was designed for exactly this problem: the knowledge a model needs is usually scattered across wikis, catalogs, code comments and senior engineers’ heads, and every team ends up re-solving the same context-assembly problem from scratch.
Adopting it bought two concrete things. First, producer/consumer independence — a card hand-written by a policy researcher and a card generated by a scraper are the same artifact to the ingestion script, so the corpus can grow either way. Second, no lock-in: it is just files, so the corpus is greppable, diffable, renderable on GitHub, and portable to whatever model or framework comes next. The knowledge outlives the stack that reads it.
type field of every concept. How the Open Knowledge Format can improve data sharing ↗B Getting the library ready
Each card is already one small, self-contained fact — so there’s nothing to split. The whole card becomes one vector: a list of numbers that captures its meaning, so we can later find cards by similarity instead of exact keywords.
768-dim vectorpgvectorWhy embeddings at all? Keyword search fails the moment a user’s words differ from the government’s. Someone asks “do I need to redo my IELTS?” and the card that answers it says language test validity — zero words in common, but nearly identical meaning. An embedding maps both into the same region of a 768-dimension space, so closeness in that space is closeness in meaning. Retrieval becomes a nearest-neighbour lookup instead of a string match.
Ingestion is deliberately a batch script, not a runtime path. It runs on deploy and on demand, never on a user’s request. That keeps embedding cost proportional to how often the corpus changes rather than how much traffic the app gets, and it means a malformed card fails loudly in CI instead of halfway through answering someone’s question.
C Answering a question
The question gets turned into a vector too, then matched against the shelf. The closest few cards — and nothing else — are handed to the AI with strict orders.
must matchtop 5Step 4 is where the guarantee is enforced. The system prompt does not politely suggest using the sources — it constrains the model to them: answer only from the supplied cards, cite the card you used for each claim, and if the cards do not contain the answer, say so. Since the cards are the only immigration content in the context window, a hallucinated rule has nowhere to come from.
The answer streams token by token rather than arriving complete. That is a perceived-latency decision, not a throughput one — total time is unchanged, but the user sees words at the first-token mark instead of staring at a spinner until the last one. Citations resolve from the retrieved cards’ resource fields, so the links are a property of the retrieval, not something the model is trusted to produce correctly.
D The choices behind it
Most of the engineering here was deciding what not to build. Each choice below traded a capability away on purpose, in exchange for a guarantee that was worth more.
Generic RAG splits long documents on a character count, which cuts sentences in half and strands a claim from the caveat that qualifies it. Because every OKF card is authored as a single self-contained fact, the natural chunk boundary already exists — so there is nothing to split. That removes the entire class of bugs where a retrieved fragment is technically present in the corpus but misleading out of context, and it means a chunk always carries its own citation rather than inheriting one from a parent document.
Knowledge is reviewed the same way code is: a pull request, a diff, an approver. Every change to a legal claim has an author, a timestamp, and a revert path — which is what makes the corpus auditable rather than merely current. It also collapses the deploy story: there is no migration and no admin UI to maintain, and a non-engineer can add a fact by adding a file.
The corpus is small and slow-moving — hundreds of cards, not millions. At that scale a specialised vector service buys latency that Postgres already delivers, while costing a second system to operate, sync, and pay for. Keeping vectors in the same Postgres that holds application data means retrieval joins directly against user state in one round trip, and one backup covers both.
This is the load-bearing decision of the whole product. An LLM that does arithmetic is plausibly wrong in ways nobody can audit, and a CRS score is the number a user makes life decisions on. So the scoring engine re-runs the official math in plain TypeScript — exact, repeatable, unit-testable — and the model never sees a calculation it could “help” with. The same engine computes lever deltas by re-running itself with one input changed, so “+18 points if you reach CLB 9” is arithmetic, not a guess.
A bigger context window is not a better answer. Padding the prompt with marginal matches raises token cost, slows the first token, and — worst — gives the model enough loosely related material to synthesise a claim no single card supports. Five keeps the grounding tight. When the retrieved cards do not answer the question the system says it does not know, because a refusal is a correct answer and a confident wrong one is a liability.
Query vectors and stored vectors must come from the same model or the geometry is meaningless — nearest-neighbour search silently returns noise instead of failing loudly. The model version is pinned in config and stored alongside every row, so a mismatch is a startup error rather than a quiet quality regression. Changing models is a deliberate re-ingest of the whole corpus, not a config tweak.
E What it costs at runtime
The useful thing about a five-stage pipeline is that you can measure each stage separately — and find out that the one you spent the most design effort on is not the one costing you anything.
| Stage | p50 | p95 | Note |
|---|---|---|---|
| Embed the question | 180ms | 303ms | One API call, 768-dim |
| Vector search | 139ms | 477ms | Almost entirely HTTP round trip |
| Prompt assembly | <1ms | <1ms | In-process, no I/O |
| First token from model | 2.94s | 4.22s | Dominates the budget |
| Full streamed answer | 3.16s | 4.83s | Length-dependent |
| End to end | 3.60s | 5.37s | Retrieval + generation |
54 runs — 18 eval questions × 3 repetitions — through the same streaming path the app serves, against the live Supabase and Gemini APIs from one machine on residential broadband. Nearest-rank percentiles, no interpolation. Every stage includes its network round trip, which is the honest way to read the table: “vector search” is microseconds of Postgres work wrapped in ~130ms of HTTP.
The shape of that table is the whole point: vector search is not the bottleneck, and it is not close. Retrieval — the embedding call plus the search — is about 320ms of a 3.6s answer, roughly 9% of the budget, while the model accounts for close to 90% of the wait before the first token appears. That is what made a dedicated vector database unjustifiable here: it would have optimised the cheapest stage in the pipeline, and at twelve cards the index scan is not even the expensive part of its own stage — the HTTP round trip is.
That distribution also tells you where tuning pays. Raising retrieval from top-5 to top-20 would barely move the search time but would inflate the prompt, which lands directly on the expensive stage — slower and less accurate at once. Keeping the context small is a latency optimisation and a quality optimisation that happen to be the same lever.
The same script scores retrieval, not just speed. Each eval question is labelled with the cards that genuinely answer it, and the questions are deliberately phrased the way a user would ask — “I have two part-time jobs, do they count as full-time?” against a card that talks about 1,560 hours. Retrieval that only works when the question echoes the source is keyword search wearing an embedding costume, and this set is built to catch that. On the current corpus it retrieves every expected card for 15 of 15 covered questions, and declines all 3 questions the corpus does not cover.
That 100% deserves an asterisk, so here it is. The corpus is twelve cards and retrieval returns five of them, so the haystack is small enough that a mediocre embedding would still score well. The number is a regression guard — it will tell me loudly when a model swap or a corpus change breaks something — not evidence that retrieval is solved. It gets interesting at a few hundred cards, and I would expect it to drop when the corpus grows enough to contain near-duplicates.
One thing the benchmark surfaced that I had assumed otherwise: the search RPC applies no similarity floor. Every query returns five cards, including the ones about proof of funds and work-permit timelines that the corpus knows nothing about. So the refusal is not retrieval declining to match — it is entirely the prompt’s instruction holding, which makes those 3 refusals a measurement of the prompt rather than a property of the system. A distance threshold in the RPC is the obvious next change, and until it lands, the refusal count is the thing I watch.
F The rule that keeps it honest
The deterministic CRS engine computes every point and every lever delta by re-running the math. Exact, repeatable — an AI never calculates a score.
RAG only explains and cites — how a factor works, what it requires. It answers strictly from retrieved cards, and refuses when the corpus doesn’t cover it.
Keeping these two apart is what lets each be tested properly. The engine has unit tests asserting exact point totals against published tables — a regression there is a failing build, not a subjective judgement. The RAG side is evaluated on whether it retrieved the right cards and refused when it should have. Two systems, two definitions of correct, neither able to corrupt the other.
G Why it was worth building this way
Every choice above was made for what it does to the operating model, not just the response. These are the properties the design is meant to guarantee — the reasons the extra constraint was worth accepting.
The through-line is that trust is a feature with an architecture. In a domain where a wrong answer costs someone a filing fee or a year of their life, “the model is usually right” is not a product. Grounding every claim in a citable source and every number in re-runnable arithmetic is what turns a chatbot into something a person can actually act on.
Status, so the claims above are read correctly: TrueBearing AI is a pre-launch project I build solo. Everything here is a property of the design and of the eval numbers in section E — measured on my own machine against a twelve-card corpus — not an outcome observed on production traffic or reported by a team.