Production architecture and inference serving

What you will be able to do

1. Translate workload and SLOs into capacity.
2. Compare managed, hosted, and on-premises options.
3. Design queuing, backpressure, caching, and degradation.

Prerequisites: Builder-level interface validation, Python, testing, metadata logging and state-transition tracing. Hardware benchmarks and model accounts are not required.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

Architecture begins with workload: arrival rate, concurrency, token distribution, latency objective, availability, data boundary, and failure consequence. Average latency is not capacity evidence; use percentiles and burst behavior. Queues absorb short bursts but create wait time and can amplify overload. Backpressure rejects or sheds work before collapse. Caches need identity, authorization, version, and freshness keys. Graceful degradation might shorten context, use a validated smaller model, return retrieval-only results, or abstain. Every option needs an exit and rollback strategy.

Mechanism and implementation

Capacity depends on arrival rate, service time, burstiness and variance. Little’s law relates average in-system work, arrival rate and residence time under stable conditions; it is not a tail guarantee. Bound queues and admission before overload. Local deployments can be production systems with proper operational ownership.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

arrival, mean_seconds = 2, 3
assert arrival * mean_seconds == 6
capacity = 4
admitted = []
for request_id in range(7):
    if len(admitted) < capacity:
        admitted.append(request_id)
assert len(admitted) == 4
print("admitted", len(admitted), "deferred", 7 - len(admitted))

Engineering decisions

• Define SLO and saturation signals.
• Select deployment boundary from authorization evidence.
• Choose degradation that preserves safety and user expectations.

Worked example

Two simulated architectures have different service rates and queue limits. The capacity model shows one meets median latency but fails p95 during bursts. The ADR selects the second and documents an offline fallback.

Diagnose the failure

Units must cancel correctly. A queue cap alone cannot prove p95 latency. Measure load-generator limits before attributing failures to serving.

Part 2 Guided practice

Calculate average in-system work and contrast it with queue capacity. Add defer/reject statuses and a retry deadline. Independently compare approved local and hosted routes for the same workload.

Answer criteria and feedback: Units must cancel correctly. A queue cap alone cannot prove p95 latency. Measure load-generator limits before attributing failures to serving.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Check units: requests/second, seconds/request, tokens/request, and concurrency. Reproduce one estimate manually before trusting the simulator. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://sre.google/sre-book/service-level-objectives/
• https://kubernetes.io/docs/concepts/workloads/autoscaling/

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.