Machine learning foundations for system builders

What you will be able to do

1. Distinguish learning regimes and their evidence needs.
2. Train/evaluate a small classifier with clean splits.
3. Interpret loss, overfitting, calibration, and shift.

Prerequisites: Builder-level interface validation, Python, testing, metadata logging and state-transition tracing. Hardware benchmarks and model accounts are not required.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

A model estimate is conditional on data, objective, algorithm, and operating distribution. Features are inputs; labels operationalize a target and may encode error or policy choices. Separate train, validation, and test sets before fitting transformations. Learning curves compare training and validation behavior; a widening gap suggests variance, not an automatic remedy. Calibration asks whether predicted probabilities match observed frequencies. Distribution shift can change feature prevalence, label relationship, or user behavior without changing code.

Mechanism and implementation

Learning estimates behavior under a chosen loss. Accuracy can hide class imbalance and unequal error costs. Calculate confusion counts before selecting a threshold. Calibration compares probabilities with frequencies. Freeze entity/time splits before fitting transformations to avoid leakage.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

truth = [1, 1, 0, 0]
scores = [0.9, 0.4, 0.6, 0.1]
def counts(threshold):
    predicted = [int(s >= threshold) for s in scores]
    return {"tp": sum(y == 1 and p == 1 for y, p in zip(truth, predicted)),
            "fn": sum(y == 1 and p == 0 for y, p in zip(truth, predicted)),
            "fp": sum(y == 0 and p == 1 for y, p in zip(truth, predicted))}
assert counts(0.5) == {"tp": 1, "fn": 1, "fp": 1}
assert counts(0.3)["fn"] == 0
print(counts(0.5))

Engineering decisions

• Define the target and acceptable error asymmetry.
• Prevent split leakage.
• Select accuracy, calibration, and slice evidence appropriate to consequence.

Worked example

A synthetic urgency classifier achieves high accuracy but is overconfident on shifted vocabulary. The learner computes Brier score and calibration bins, documents the failure, and recommends abstention or retraining evidence.

Diagnose the failure

These four examples illustrate threshold arithmetic, not population performance. Inspect label alignment before retraining when results differ.

Part 2 Guided practice

Compute counts by hand and describe changes at threshold 0.3. Independently fit a mean using an entity-disjoint training set only and explain how deployment shift could change error rates.

Answer criteria and feedback: These four examples illustrate threshold arithmetic, not population performance. Inspect label alignment before retraining when results differ.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Recompute one example by hand. Confirm labels and probability range before interpreting aggregate metrics. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://airc.nist.gov/airmf-resources/airmf/
• https://scikit-learn.org/stable/modules/calibration.html

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.