Retrieval baseline from first principles

What you will be able to do

1. Chunk content with stable source identifiers.
2. Implement transparent lexical and vector-like baselines.
3. Measure retrieval separately from answer generation.

Prerequisites: dictionaries, loops, functions, exceptions and installed Python or paper tracing. Reuse earlier Builder artifacts only when the task requires them.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

Retrieval quality asks whether the system found the needed evidence; answer quality asks whether generation used it correctly. Mixing them hides root causes. Begin with structure-aware chunks that preserve document, section, access label, version, and span. A lexical score is explainable and strong for exact terms; an embedding score may capture semantic similarity but introduces model and version dependencies. Evaluate with query-to-relevant-chunk judgments, recall@k, precision@k, and failure classes. Do not add generation until retrieval evidence is visible.

Mechanism and implementation

Choose retrieval when relevant evidence is outside supplied input. Direct lookup may be simpler for a tiny catalog. Start with visible lexical scores and relevance judgments, then compare vector or hybrid search on the same cases. Model revision and preprocessing define embedding space; equal dimensions do not prove compatibility.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

docs = {"a": "workshop opens nine", "b": "lunch opens noon", "c": "workshop room four"}
def rank(query):
    words = set(query.split())
    scored = [(len(words & set(text.split())), key) for key, text in docs.items()]
    return [key for score, key in sorted(scored, key=lambda x: (-x[0], x[1])) if score > 0]
assert rank("workshop opens")[0] == "a"
relevant = {"a", "c"}
assert len(set(rank("workshop")[:2]) & relevant) / len(relevant) == 1
print("constructed recall@2 = 1")

Engineering decisions

• Choose chunk boundaries that preserve meaning and provenance.
• Select k based on context budget and recall evidence.
• Classify a miss as corpus, chunking, indexing, query, or ranking failure.

Worked example

A six-document synthetic policy corpus includes exact acronyms and paraphrases. The learner creates stable chunks, evaluates keyword overlap, and compares results against 30 judgments. Misses are labeled rather than averaged away.

Diagnose the failure

Report retrieval recall separately from answer support. The printed number is arithmetic on invented records, not a model benchmark. Define tie-breaking before comparing rankings.

Part 2 Guided practice

Calculate overlap scores manually. Add a synonym with no shared term and record the lexical miss. Independently compare an alternative constructed ranking using the same judgments.

Answer criteria and feedback: Report retrieval recall separately from answer support. The printed number is arithmetic on invented records, not a model benchmark. Define tie-breaking before comparing rankings.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Print scores for one failing query. Confirm token normalization and chunk IDs before changing ranking. Avoid tuning to one query; rerun the full judgment set. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://airc.nist.gov/Docs/Generative_AI_Profile.pdf
• https://lucene.apache.org/core/

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.