Cost, latency, and trade-offs

Lesson preparation and outcomes

Prerequisites: Usage fields, arithmetic and basic percentile interpretation.

By the end, you can: Calculate task resource estimates and compare candidates without confounding model and harness.

Worked scenario and practice feedback

The declared invented rate card gives 0.0036 units for 1,000 input and 200 output tokens. A retry doubles performed work even if the first result was unusable. Ten timings cannot establish rare p99 behavior.

Guided and independent practice: Calculate the example by hand then with Python; independently add cache misses, retries and a fallback without required schema support.

Answer criteria: Units/rates and synthetic provenance declared; unknown usage not zero; cache authorization enforced; same harness used for model comparison. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Every API call you make has two numbers attached to it: how much it cost and how long it took. Both can grow without you noticing. This lesson teaches you to instrument them and respect them.

Part 1 Calculate from declared rates and measured usage.

Use the exact endpoint’s current rate card and billing definition. Parameter count, local versus cloud deployment and frontier capability are separate axes. Many hosted models do not disclose parameter counts. No universal ordering of quality, latency or price follows from size.

For an illustrative token-billed endpoint, the estimate is (uncached_input * input_rate + cached_input * cached_rate + output * output_rate) / 1_000_000, plus separately billed tools, cache writes, storage or other charges. Rates must share currency and billing units. Reasoning-token accounting is provider/model specific; record what usage includes and avoid double counting.

Constructed example, not a provider quote: 1,000 input tokens at 2 units/million plus 200 output at 8 units/million equals 0.0036 units per call. At 1,000 calls the estimate is 3.6 units, before retries and other charges. Output is four times input in this invented rate card; there is no universal five-times rule.

For a local deployment include memory/compute capacity, utilization, operator time, backups and maintenance. A zero API invoice does not establish zero cost. Budget by task, set usage limits and expose budget exhaustion before starting another call.

Part 2 Measure cache hits and eligibility.

Prompt caching reuses eligible repeated input portions at the provider or serving layer. Eligibility, prefix matching, minimum size, lifetime, read/write price and availability vary by endpoint and model. Consult the actual documentation and record cache-hit usage. Do not assume a five-minute lifetime or a fixed savings percentage.

Distinguish prompt caching from response caching. Prompt caching can reduce work while still generating a fresh output. Response caching reuses an earlier answer and therefore needs freshness and authorization checks. Put stable shared prefix content before variable content only where this matches the endpoint’s caching mechanism; do not add irrelevant context just to qualify for a cache.

In a constructed case, 800 eligible tokens at 0.5 units/million and 200 uncached tokens at 2 units/million cost 0.0008 input units versus 0.002 with no cache. This excludes output, cache writes and misses. Calculate the hit-rate break-even point using the chosen endpoint, rather than describing this example as measured savings.

Part 3 Streaming vs batching.

Streaming can improve time to first useful content without lowering total completion time. Treat emitted chunks as provisional: cancellation, a refusal, tool request, truncation or schema error can occur later. Do not execute actions from unfinished output.

Batch processing fits work with no interactive deadline. Discounts, turnaround bounds, compatible models and cancellation rules depend on the endpoint. Offline, simulate jobs with IDs and shuffled results: join on IDs, handle partial failure, and avoid treating submission as completion. A batch is not permission to process a larger data boundary.

Part 4 Compare candidates under a controlled harness.

Select candidates from the task: input modalities, context size, required output/tool features, data boundary and acceptable error consequence. Then hold the harness, cases, tool permissions and grading protocol constant when comparing models. Record model revision and supported reasoning budget; unsupported controls must be rejected or disclosed, not silently ignored.

Predeclare slice-specific acceptance rules. A lower aggregate score might be acceptable for a low-consequence draft but a single critical disclosure failure can block a route. Compare quality with median/tail latency, refusal/partial rates and total task cost including retries and human correction.

Routing is an additional system to evaluate. Its classifier can send a hard or sensitive task to an unsuitable route; uncertain cases need a evaluated fallback or human review. Any savings are measured results of your workload, not a promised 50–80%. To compare harnesses, keep the model constant.

Part 5 Caching responses and dedup.

A response-cache key must include the factors that can change authorized meaning: tenant or access scope, task/prompt version, model revision, relevant parameters, corpus revision and input. A hash of text alone is not authorization. Do not share one user’s private response with another user who asks the same words.

Revalidate permission on read. Invalidate on source deletion, access-policy change, model/prompt change or known staleness. Define maximum retention even for apparently stable facts. Idempotency is different: it deduplicates an action by identity and recorded outcome, rather than treating similar prompts as the same action.

Interface/reference limits

Use the workflow reference to connect task, model, deployment, harness/tools, evidence and the responsible human decision. Synthetic demonstrations establish only the stated tested conditions.

Primary source checked October 2, 2026: OpenAI prompt caching.