Performance, reliability, and resource governance

What you will be able to do

1. Test latency, throughput, availability, and degradation.
2. Apply caching, batching, routing, and fallback with evidence.
3. Enforce per-request resource budgets.

Prerequisites: Builder-level interface validation, Python, testing, metadata logging and state-transition tracing. Hardware benchmarks and model accounts are not required.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

Performance governance starts with user-facing SLOs and an error budget, then links them to request admission, concurrency, model routing, cache behavior, and dependency failure. Report median and tail latency; p95 or p99 often governs experience. A cache must not bypass freshness or authorization. Fallbacks require quality and safety evaluation. Resource budgets cap tokens, tool calls, elapsed time, retries, and compute estimates to contain loops and denial-of-wallet conditions. The lab uses unit estimates only and initiates no purchase.

Mechanism and implementation

Optimize the bottleneck supported by traces, then recheck quality and permissions. Rate, admission, context, call and token budgets do different jobs. A fallback needs its own data-policy and capability evaluation and must not silently lose required tools or schemas.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

jobs = [{"id": "a", "calls": 1, "tokens": 100},
        {"id": "b", "calls": 4, "tokens": 200},
        {"id": "c", "calls": 1, "tokens": 2000}]
def admit(job):
    return job["calls"] <= 3 and job["tokens"] <= 1000
assert [j["id"] for j in jobs if admit(j)] == ["a"]
durations = sorted([10, 20, 30, 40, 100])
assert durations[2] == 30
print("accepted a; constructed median 30 units")

Percentile contract for the downloadable lab

Use the empirical nearest-rank convention: sort n finite nonnegative durations; for 0 < q <= 1, select position ceil(q*n) using one-based positions; q=0 selects the minimum. Empty input and q outside [0,1] are errors. Thus [1,2,3,100] has p95 100, not 3. Other conventions, including interpolated quantiles, are valid but must be named rather than mixed. The synthetic trace practices calculation and release decisions; its sample p95 is not a guarantee about real service tails. See NumPy’s documented methods.

Engineering decisions

• Choose SLO and error-budget policy.
• Tune one bottleneck without regressing quality.
• Define budget exhaustion behavior.

Worked example

A load trace shows p95 latency breaches after queue depth 20. A bounded concurrency change improves p95, while the quality regression check stays within tolerance. Requests exceeding tool budget abstain.

Diagnose the failure

No request crosses a cap or silently drops a required feature. Five timings illustrate arithmetic, not tail reliability. Changed inputs are a confound to disclose.

Part 2 Guided practice

Predict budget denials and median. Add a total-time cap including retries. Independently propose a cheap fallback missing required schema support and block it before execution.

Answer criteria and feedback: No request crosses a cap or silently drops a required feature. Five timings illustrate arithmetic, not tail reliability. Changed inputs are a confound to disclose.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Sort raw durations and verify the percentile convention. Change one control at a time and rerun both performance and quality checks. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://sre.google/sre-book/embracing-risk/
• https://airc.nist.gov/Docs/Generative_AI_Profile.pdf

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.