Retrieval-augmented generation, from scratch

Lesson preparation and outcomes

Prerequisites: Data loading, strings and prompt/result contracts.

By the end, you can: Separate retrieval ranking from answer support and choose whether retrieval is needed.

Worked scenario and practice feedback

Synthetic workshop documents give a time and a room. A lexical query finds the time span; a synonym query may miss it. Missing retrieval is different from generation contradicting a retrieved time. Inspect ranked source IDs first, then match claims to authorized current spans.

Guided and independent practice: Run the three-document offline exercise. Independently add conflicting revisions and a document instruction, and compare retrieval with a direct table lookup.

Answer criteria: Missing or conflicting evidence leads to abstention/escalation; citations identify real spans; document text grants no tool authority. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Retrieval-augmented generation supplies selected evidence at query time. Learn the mechanism so you can decide when retrieval helps, when a direct lookup is clearer, and when the supplied context already contains everything needed.

Part 1 What RAG actually does.

A generation model conditions on its input as well as learned parameters. RAG retrieves external evidence into that input; it does not guarantee that the model uses it correctly. Retrieval may be lexical, vector, hybrid or a structured database query.

The mental model:

1. Index time (repeated when content, access rules or embedding-space configuration changes): chop your documents into chunks, turn each chunk into a vector (embedding), store them
2. Query time (done per question): turn the user’s question into a vector, find the chunks whose vectors are closest, paste those chunks into the prompt, ask the LLM
3. The LLM answers using the pasted context, instead of its training data alone

That’s it. Five steps. The rest is engineering.

Part 2 The pieces, in code.

We’ll build a RAG over a folder of your own text files. No vector database, just numpy arrays in memory. (Lesson 06 graduates to pgvector.)

Step 1: load and chunk your docs

from pathlib import Path

def load_docs(folder):
    chunks = []
    for path in Path(folder).rglob("*.md"):
        text = path.read_text(encoding="utf-8")
        for i, chunk in enumerate(chunk_text(text, size=800, overlap=100)):
            chunks.append({"source": str(path), "i": i, "text": chunk})
    return chunks

def chunk_text(text, size=800, overlap=100):
    """Illustrative character slicing; validate parameters before the loop."""
    if not isinstance(text, str):
        raise TypeError("text_must_be_string")
    if type(size) is not int or type(overlap) is not int or size <= 0 or not 0 <= overlap < size:
        raise ValueError("require_positive_size_and_overlap_smaller_than_size")
    chunks = []
    start = 0
    while start < len(text):
        end = min(start + size, len(text))
        chunks.append(text[start:end])
        if end == len(text):
            break
        start = end - overlap
    return chunks

Chunk size and overlap trade local context against irrelevant material and repeated retrieval. The example’s 800 characters and 100-character overlap are an illustrative baseline, not an optimal range for every document. Characters are not model tokens. Preserve headings, sentences and tables where possible; compare candidate chunking rules on representative queries and relevance judgments, including misses and unsupported-answer cases. Verify each configuration fits the selected tokenizer, embedding model and downstream context budget.

Step 2: embed the chunks

An embedding is a list of numbers (usually 1024 or 1536 of them) that represents the meaning of a piece of text. Two pieces of text with similar meaning get similar vectors.

import os
from openai import OpenAI
import numpy as np

client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def embed_batch(texts):
    response = client.embeddings.create(
        model="text-embedding-3-small",
        input=texts,
    )
    return np.array([d.embedding for d in response.data])

# Embed all chunks
texts = [c["text"] for c in chunks]
vectors = embed_batch(texts)  # shape: (n_chunks, 1536)

Check the selected endpoint limits before batching. Batching can reduce request overhead; it does not necessarily change per-token price. The API code is illustrative and is not the required offline exercise.

Step 3: search

To find the most relevant chunks for a query, embed the query and compute cosine similarity against every chunk vector.

def search(query, k=5):
    q_vec = embed_batch([query])[0]
    sims = (vectors @ q_vec) / (np.linalg.norm(vectors, axis=1) * np.linalg.norm(q_vec))
    top = np.argsort(sims)[-k:][::-1]
    return [(sims[i], chunks[i]) for i in top]

The matrix multiplication vectors @ q_vec computes the dot product against every chunk in one operation. Measure runtime on your own hardware and corpus; no timing result is asserted here. Guard zero-norm vectors before division.

Step 4: assemble the prompt

def build_rag_prompt(query, retrieved):
    context = "\n\n---\n\n".join(
        f"[Source: {c['source']}]\n{c['text']}"
        for _, c in retrieved
    )
    return f"""\
Use the context below to answer the question. If the answer isn't in the context, say so.

CONTEXT:
{context}

QUESTION: {query}

ANSWER:
"""

Step 5: call the model

from anthropic import Anthropic
ant = Anthropic(api_key=os.getenv("ANTHROPIC_API_KEY"))

def ask(query):
    retrieved = search(query, k=5)
    prompt = build_rag_prompt(query, retrieved)
    response = ant.messages.create(
        model="claude-haiku-4-5",
        max_tokens=1024,
        messages=[{"role": "user", "content": prompt}],
    )
    return response.content[0].text, retrieved

That’s a working RAG. About 60 lines of Python.

Part 3 The things that go wrong.

Inspect these failure classes before changing the architecture.

• Bad chunks. Naive chunking breaks paragraphs mid-sentence. Better chunkers respect document structure (paragraph, section, header). The cheap fix: chunk by paragraph and combine until you hit your size cap.
• Wrong chunks retrieved. Sometimes the query and the right chunk don’t share vocabulary. The fix: hybrid search (vector + keyword), or query rewriting (have the LLM expand the query first).
• Right chunks, wrong answer. The model hallucinates anyway. Check claim support against exact source spans, detect contradictions, and abstain when evidence is missing. Citation-shaped text and low temperature do not establish truth.
• Stale index. Documents change; vectors don’t update automatically. The fix: a refresh job, or store vectors next to source files and re-embed on change.
• Cost runaway. Embedding 100K documents costs real money. Cache embeddings in a file or database. Re-embed when content or model revision, preprocessing, dimension configuration or normalization changes. Propagate access and deletion changes to every derived index/cache.

Diagnose retrieval, support, freshness and access separately before adding components.

Interface/reference limits

Use the workflow reference to distinguish model capability, deployment and harness controls. Provider snippets in this lesson are illustrative, have not been executed against an actual provider in this revision, and require current SDK/model capability verification before separately authorized use.