← rules
05 / RAG
/

05 / Grounded knowledge

Retrieve first.
Then reason.

Retrieval-augmented generation (RAG) gives a model a small, attributable working set of current knowledge. It is not a magic database: quality comes from the source, retrieval, evaluation, and the decision to say not found.

Film · 26 s · silent, captioned
Transcript
  1. Your files, split into labelled chunks.
  2. A question is scored against every chunk.
  3. Keep only the few that match.
  4. Answer from them, and cite the source.
  5. No good match? Say so.
  6. Retrieve first. Then reason.

01 / the foundation

A model has a
library card.

RAG supplies fresh, verifiable evidence at inference time, then constrains the model to reason strictly from that context. Use it when facts evolve rapidly, provenance and citations matter, or private knowledge belongs to your organization—not when a static behavioral rule fits directly in prompt context.

  1. 01
    Ingest Parse trusted documents, isolate headings and code fences, and attach source, tenant, ACL, and version metadata.
  2. 02
    Retrieve Execute filtered lexical or vector search to select a tight candidate set (3 to 8 chunks) before generation begins.
  3. 03
    Ground Synthesize the answer strictly with attributable citations, or abstain explicitly when evidence is missing or ambiguous.

02 / ARCHITECTURES

From local files
to hybrid search.

Choose the simplest architecture that satisfies your provenance, latency, and update requirements. Measure lexical search before paying for vector infrastructure.

01 / files-only

The folder is the index

For private team notes and small repositories, parse local Markdown, preserve heading breadcrumbs, and rank chunks directly into prompt context. Zero infrastructure, instant local reproduction, and every input is fully inspectable in Git.

02 / lexical & bm25

Search exact words first

SQLite FTS5, Postgres full-text search, and BM25 score literal token matches. They excel at error codes, function names, SKU numbers, and newly coined jargon where vector models blur or hallucinate. Sub-millisecond, explainable, and zero embedding API cost.

03 / dense vectors

Search for semantic meaning

Embedding models project queries and passages into continuous vector spaces. pgvector, Qdrant, or Chroma retrieve conceptual matches across synonyms and paraphrases. Requires pinning model versions and dimensions directly to the index lifecycle.

04 / hybrid & rerank

Let two searchers cover each other

Run lexical and dense retrieval in parallel, merge candidate lists with Reciprocal Rank Fusion (RRF), and score top candidates with a Cross-Encoder reranker. Delivers high recall for conceptual intent without sacrificing exact keyword precision.

03 / INSPECT THE PIPELINE

From source
to answer.

Select a stage to inspect the actual code, SQL schema, queries, and invariants an agent needs to operate each layer without hand-waving.

RAG / BUILD LAB

01 / FILES

The smallest useful RAG is a folder

Put a few trusted Markdown or text files beside a tiny script. Read them, split them into labelled chunks, and pass the best chunks into the model. This is ideal for a prototype because every input is inspectable in Git and there is no service to operate.

rag-demo/
├── docs/handbook.md
├── eval/questions.jsonl
└── src/
    ├── chunk.ts
    └── ask.ts

pnpm tsx src/ask.ts "How many days of leave?"
ENGINEERING INVARIANT Use files when the corpus is small, private, and changed by deploys—not when you need concurrent ingestion or user-level permissions.

04 / ingestion & chunking

Chunks need
hierarchy & provenance.

Naive character slicing breaks tables, splits code blocks across chunks, and strips section headings. Production chunking parses document syntax, attaches breadcrumbs ('Handbook > Policies > Leave'), and calculates content hashes for idempotent re-ingestion.

CANONICAL CHUNK ENVELOPE interface ChunkRecord { chunk_id: string; // sha256(source + heading + content) source_path: string; // 'docs/handbook/leave.md' heading_path: string[]; // ['Policies', 'Annual Leave', 'Limits'] content: string; // clean markdown with code intact tenant_id: string; // partition key "kw">for pre-retrieval ACL version: string; // git commit sha or document revision tokens: number; // bounded context budget (200-500 tokens) }

Idempotent upsert: hash the source text so re-ingestion does not duplicate knowledge or regenerate vector embeddings unnecessarily.

05 / evaluation & defense

Measure two tiers.
Treat input as untrusted.

Never evaluate RAG with a single overall score. A good LLM can hallucinate a plausible answer from bad retrieval, while a poor prompt can fail even with perfect evidence. Score retrieval (Recall@k, MRR) independently from generation (Faithfulness, Relevance). Simultaneously, treat all retrieved text as untrusted data to defend against indirect prompt injection.

TWO-TIER METRICS MATRIX RETRIEVAL QUALITY (Did we find the needle?) Recall@k -> Did gold chunks appear "kw">in top-k results? MRR -> Position "kw">of the first relevant chunk (1/rank) nDCG -> Graded relevance order across candidate set GENERATION INTEGRITY (Did the answer honor the context?) Faithfulness -> All claims verified against retrieved evidence Relevance -> Directly answers the user question without drift Abstention -> Model says 'not found' when evidence is missing Security -> Pre-filtered ACLs; isolated prompt data delimiters

06 / check yourself

Test your
retrieval instincts.

Active retrieval builds durable intuition. Formulate your answer from memory before opening each question, then verify your mental model against the architectural rationale.

Why should you benchmark BM25 lexical search before adopting vector embeddings?

BM25 is sub-millisecond, costs nothing in GPUs or embedding APIs, and reliably matches exact error codes, IDs, function signatures, and newly introduced product names where vector representations struggle or hallucinate. Measuring BM25 establishes an explainable baseline, proving whether semantic vector search actually yields higher Recall@k on real user questions.

What is the operational difference between Bi-Encoders and Cross-Encoders in a hybrid pipeline?

Bi-Encoders embed query and documents separately into vectors, enabling O(1) nearest-neighbor candidate retrieval across millions of records but losing token-to-token cross-attention. Cross-Encoders process (query, passage) together through full transformer self-attention, delivering significantly higher reranking precision at much higher latency. Production systems use Bi-Encoders to select 50 candidates, then a Cross-Encoder to rerank the top 5.

Why is filtering by tenant and access control (ACL) after vector retrieval a critical defect?

If you retrieve the global top-10 vectors and then discard chunks belonging to other tenants, a query can return zero results even when valid, authorized documents exist at ranks 11–20. Post-filtering can also leak metadata or document existence through timing differentials. Pre-filtering by tenant and ACL within the index query ensures the candidate set is drawn strictly from authorized data.

Why must retrieval metrics and generation metrics be measured separately?

A capable LLM can fabricate a plausible, correct-looking answer from irrelevant or missing context (a latent hallucination), while an improperly structured prompt can produce a confused answer from perfect context. Measuring Retrieval (Recall@k, MRR) independently from Generation (Faithfulness, Citation precision, Negative abstention) isolates whether the index or the prompting layer needs work.