Production deployment patterns and infrastructure

Lesson preparation and outcomes

Prerequisites: HTTP outcomes, serving concepts and resource estimates.

By the end, you can: Design admission, readiness and recovery for a declared workload and deployment.

Worked scenario and practice feedback

At stable 2 requests/second and mean 3 seconds in system, average work in system is 6. A cap of four bounds admission but does not by itself satisfy latency. An API-compatible backend lacking required tool/schema features must fail the capability gate.

Guided and independent practice: Use the offline queue/readiness simulator. Independently test bursts, dependency outage, cancellation and compatible artifact/data rollback.

Answer criteria: Correct units and limits; distinct liveness/readiness; feature equivalence checked; local production permitted when its controls and ownership satisfy requirements. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

A production system serves a defined real task under operational ownership, access policy, reliability requirements and a recovery plan. It may run locally, on premises or through a hosted service; location alone does not distinguish a demo from production. This lesson covers the infrastructure that turns one into the other, without the over-engineering that wastes federal budget on cleverness no one needs.

Part 1 What 'production' actually means.

Before you pick infrastructure, agree on what production means for your system. The answer differs across federal contexts.

Three production tiers, in order of stakes:

• Internal staff augmentation. A tool a small number of cleared analysts or developers use. Hundreds of requests per day. Downtime is annoying but not catastrophic. Treat this as one workload example; impact and authorization depend on the actual task.
• Operational decision support. A tool that informs operational decisions but doesn’t make them. Thousands of requests per day. Downtime affects mission tempo; quality drops can affect decisions.
• Mission-critical inline. A tool whose output is consumed directly into mission systems. Many thousands to millions of requests per day. Downtime stops the mission. Wrong answers compound.

Each tier has different infrastructure requirements. Don’t build mission-critical infrastructure for an internal staff tool, and don’t ship a mission-critical system with internal-staff-tool infrastructure.

Part 2 The inference serving layer.

Three serving frameworks illustrate different implementation choices. This list is not a market-share ranking or a current support guarantee:

• vLLM. Originally from Berkeley. PagedAttention for memory-efficient KV cache management. An example serving engine with model- and hardware-specific support; benchmark the actual workload rather than assume a throughput ranking.
• TGI (Text Generation Inference). A historical serving-framework example. The official documentation, checked 2026-10-03, states maintenance mode. Assess support, security fixes and migration options before using an existing deployment; do not assume it receives the same ongoing development as an active candidate.
• SGLang. Strong on complex generation patterns (structured output, multi-turn, agent workflows) and now a mature, production-grade option in its own right, especially for agent-heavy workloads.

What you get from a serving framework:

• Batching across concurrent requests for higher throughput
• KV cache management that doesn’t fragment GPU memory
• Streaming responses (server-sent events)
• Health endpoints, metrics, basic observability hooks
• An API surface whose exact supported capabilities must be tested; OpenAI-compatible syntax does not establish feature or behavior equivalence

A working setup with vLLM:

pip install vllm
vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --tensor-parallel-size 1 \
  --gpu-memory-utilization 0.9

Keep this optional reference bound to loopback. Before any wider exposure, implement authentication, authorization, network/egress policy and rate limits in the approved boundary. It is not a deployment instruction for the required offline lab.

Hit a separately authorized local endpoint with a compatible client:

from openai import OpenAI
client = OpenAI(base_url="http://localhost:8000/v1", api_key="not-needed")
resp = client.chat.completions.create(
    model="meta-llama/Llama-3.1-8B-Instruct",
    messages=[{"role": "user", "content": "Summarize this..."}],
)

The reference bring-up demonstrates an endpoint shape. Switching targets requires capability/contract tests for model IDs, schemas, tools, streaming, state, usage and errors; familiar syntax alone does not keep behavior equivalent.

Part 3 Containers, orchestration, and the right level of complexity.

Two competing temptations to resist:

• Over-engineer. Kubernetes-everything, service-mesh-everything, ten layers of abstraction for a system that serves 200 users. Choose complexity from availability, rollout, scheduling, isolation and operational skills, not user count alone.
• Under-engineer. Run on a single host with no health checks, no restart policy, no logging. Works until the first incident.

One candidate for a small controlled deployment:

• Docker container per service.
• systemd or docker-compose to start things on the host and restart on failure.
• A reverse proxy (nginx, Caddy) in front for TLS, rate limiting, and access control.
• One log aggregator (journald, Vector, fluentd) shipping to a central store.
• Boring host monitoring (node_exporter + Prometheus + Grafana, or your agency’s standard stack).

You can serve real production workloads with that setup. The right time to introduce Kubernetes is when you have multiple services, multiple environments, and a real team that knows k8s. Not before.

When Kubernetes does become correct:

• Multi-tenant serving across many users
• Multiple models behind a single endpoint with rolling updates
• A platform team distinct from the application teams
• Existing agency standardization on it (often the deciding factor)

Part 4 GPU economics: the math that decides architecture.

Production AI inference is dominated by GPU cost. Two patterns worth burning in:

Pattern 1: Throughput-bound

You have a high enough request rate that the GPU is busy. Cost per request is roughly:

cost_per_request = (gpu_hourly_cost) / (requests_per_hour_at_full_utilization)

Batching, KV cache reuse, and prompt caching all shift this favorably. Higher utilization can reduce idle capacity but increase queues/tail latency. Choose a tested operating point from latency, throughput, quality and headroom requirements.

Pattern 2: Latency-bound

You need first-token latency below a threshold (e.g., 500ms). The GPU may be idle some of the time because you can’t batch as much. Cost per request is higher.

The trade is real. A chat-style UI needs low first-token latency; a batch summarization pipeline can tolerate seconds and run at higher utilization, lowering cost.

Quick math worked

Constructed arithmetic only: assume an accelerator costs 3 resource units/hour. This is not a current provider or government-cloud quote, and it excludes hardware, licensing and staffing costs.

Suppose a measured local trace reports 600 completed jobs/hour at the declared input/output mix. Dividing 3 units/hour by 600 jobs/hour gives 0.005 units/job before retries and overhead. Do not divide by concurrent users to estimate request cost: concurrency is work present at a time, not throughput. No model capacity or acceptable latency is asserted here.

3 units/hour / 600 completed jobs/hour = 0.005 units/completed job (constructed assumptions)

Multiply the illustrative rate by completed jobs per month, not user-hours. For a constructed demand of 20 users × 2 completed jobs/user/hour × 4 hours/day × 20 days/month, monthly demand is 3,200 jobs and the workload-scaled estimate is 16 resource units before retries and overhead. A continuously reserved accelerator still incurs reserved hours during idle time, so this workload estimate is not its monthly bill. Record provisioned hours, retries, tools, storage, staffing, hardware and licensing separately. Compare hosted and local totals under the same workload and authorized data boundary; neither is universally cheaper.

Part 5 Load balancing, autoscaling, and the patterns that actually matter.

Once you have multiple GPU servers, you need to route requests across them.

• Round-robin with sticky sessions. Simplest. Works when each request is independent. Doesn’t work well when context caching matters.
• Hash-based affinity. Route requests with the same conversation ID or user ID to the same server. May improve locality if the actual server/session caching mechanism supports it; verify isolation, failure redistribution and load imbalance.
• Least-busy routing. Pick the GPU with the most available capacity. Useful when request lengths vary widely.

Autoscaling is trickier for AI than for stateless web services:

• Spinning up a new GPU server takes minutes (load duration depends on artifact size, storage, network and implementation).
• You can’t burst into existing capacity instantly the way you can with web servers.
• Cold start during traffic spikes leads to timeout cascades.

Practical autoscaling pattern:

• Choose tested utilization/headroom from arrival bursts, queue/tail latency, failover and load time; no fixed percentage is required.
• Pre-warm models on standby servers; don’t spin from zero.
• Use queue depth, not just CPU, as the scaling signal: once requests start queuing, you need capacity immediately.

Part 6 The deployment pipeline.

A working deployment pipeline for federal AI:

• Code in version control. Every model artifact, every container, every config tied to a commit.
• Containers built in CI. Reproducible. Signed. Pushed to a registry your runtime can pull from.
• Models versioned separately. Model weights are huge; you don’t put them in your application container. Pull from object storage at startup; verify checksums.
• Configuration via environment. Different envs (dev, test, prod) differ by configuration, not by code.
• Health checks before traffic. Don’t accept traffic until the model is loaded and a smoke test passes.
• Canary or blue-green rollouts. Send a percentage of traffic to the new version; promote if eval and operational metrics hold.
• Rollback that’s been tested. Roll back in one command. Test the rollback in dev periodically; the night you actually need it is not when you discover it doesn’t work.

Validate the pipeline against actual change types and runtime dependencies. A documented pipeline without executed gates or a compatible rollback is incomplete evidence.

Interface/reference limits

Use the workflow reference to distinguish model capability, deployment and harness controls. Provider snippets in this lesson are illustrative, have not been executed against an actual provider in this revision, and require current SDK/model capability verification before separately authorized use.