Evaluation design and release gates

What you will be able to do

1. Design representative and adversarial cases.
2. Combine deterministic, human, and calibrated automated grading.
3. Predeclare release and rollback thresholds.

Prerequisites: dictionaries, loops, functions, exceptions and installed Python or paper tracing. Reuse earlier Builder artifacts only when the task requires them.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

An evaluation suite is a decision instrument, not a demo collection. Begin with intended use, affected users, unacceptable outcomes, and likely changes. Sample routine, boundary, adversarial, abstention, and regression cases. Record case provenance and authorization. Deterministic graders work for exact structure and citations; qualified humans judge nuance; model-assisted graders can scale triage but must be calibrated against human judgments and cannot alone pass a critical gate. Report slices and confidence limits; averages cannot compensate for a critical authorization failure.

Mechanism and implementation

A gate combines task-quality rules with critical safety rules. Define it before inspecting a candidate. Pair case IDs across versions and retain difficult cases. Judges need rubric anchors, blinding and calibration against human review. Small samples constrain what can be claimed.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

cases = [{"id": "a", "pass": True, "critical": False},
         {"id": "b", "pass": True, "critical": False},
         {"id": "c", "pass": False, "critical": True}]
quality = sum(c["pass"] for c in cases) / len(cases)
critical = [c["id"] for c in cases if c["critical"] and not c["pass"]]
release = quality >= 0.6 and not critical
assert not release
print("no-go", critical)

Engineering decisions

• Tie every case to a risk or requirement.
• Predeclare critical failures and acceptable uncertainty.
• Decide which disagreements require adjudication.

Worked example

A 75-case RAG suite measures authorization leakage, citation correctness, supported-answer rate, abstention, and latency. The release rule requires zero leakage, at least 90% citation correctness, and no regression greater than the predeclared tolerance on priority slices.

Diagnose the failure

An aggregate score cannot erase critical failure. Report counts and missing cases. Deleting failures to improve the score invalidates evidence; correct an erroneous test only with preserved rationale.

Part 2 Guided practice

Predict average and decision. Add a hundred routine passes and verify the critical failure still blocks release. Independently report slices and preserve one judge disagreement for adjudication.

Answer criteria and feedback: An aggregate score cannot erase critical failure. Report counts and missing cases. Deleting failures to improve the score invalidates evidence; correct an erroneous test only with preserved rationale.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

If the decision changes when one case is removed, report fragility. Add cases only through change control; do not delete failing cases to improve the score. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://airc.nist.gov/airmf-resources/airmf/
• https://platform.openai.com/docs/guides/evals

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.