Evaluation engineering and statistical evidence

What you will be able to do

1. Design stratified comparisons tied to risks and populations.
2. Quantify uncertainty with paired evidence.
3. Calibrate automated graders against humans.

Prerequisites: Builder-level interface validation, Python, testing, metadata logging and state-transition tracing. Hardware benchmarks and model accounts are not required.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

A release comparison should preserve pairing: run the same cases on both systems and analyze per-case differences. Stratify by risk and user-relevant slices before sampling. Confidence intervals express sampling uncertainty, not all uncertainty. Rare critical failures often need targeted tests and conjunctive rules rather than an average rate. Human rating protocols define criteria, anchors, blinding, and adjudication. Automated judges are measurement instruments whose agreement, bias, and failure modes must be tested against qualified human judgments.

Mechanism and implementation

Pair cases to isolate changes. Hold harness constant for model comparisons and model constant for harness comparisons. Predeclare metric, critical gates, slices and uncertainty method. Repeated cases from one source are dependent and cannot be counted as independent users.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

old = {"a": 1, "b": 0, "c": 1}
new = {"a": 1, "b": 1, "c": 0}
assert old.keys() == new.keys()
delta = [new[k] - old[k] for k in sorted(old)]
assert delta == [0, 1, -1]
assert sum(delta) == 0
print("one gain and one regression; aggregate unchanged")

Engineering decisions

• Choose estimand and decision threshold before data collection.
• Preserve paired case identifiers.
• Separate statistical, operational, and safety significance.

Worked example

Two systems are compared on 40 paired synthetic cases. The new system improves routine cases but creates two critical disclosure failures. The release decision is no-go despite a positive average difference.

Diagnose the failure

The inline example has an aggregate tie; the larger lab has an aggregate improvement. Either can hide a critical regression. Confidence intervals concern sampling assumptions, not permission or all uncertainty. Misaligned IDs block scoring.

Part 2 Guided practice

Identify gains/regressions before calculating a mean. Declare c critical and report slices. Independently remove a candidate case and fail the comparison explicitly rather than dropping the pair.

Answer criteria and feedback: An aggregate improvement can hide a critical regression. Confidence intervals concern sampling assumptions, not permission or all uncertainty. Misaligned IDs block scoring.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Recalculate with a fixed random seed and verify case alignment. If one system is missing a case, flag it; do not silently drop the pair. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://airc.nist.gov/airmf-resources/airmf/
• https://www.itl.nist.gov/div898/handbook/

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.