Evaluation and observability

Lesson preparation and outcomes

Prerequisites: Input/result contracts, loops and metadata logging.

By the end, you can: Design task-relevant evals and minimize observability content.

Worked scenario and practice feedback

A keyword test passes the text no logistics experience because it contains logistics. Negation and unsupported achievements reveal why keyword checks are only a smoke test. Use a claim rubric and human adjudication for ambiguous cases; test the logger independently.

Guided and independent practice: Create supported, unsupported, abstention and denied-access fixtures; independently seed secrets in normal and exception paths.

Answer criteria: Preserved case IDs, separate critical gates, calibrated judgments and allowlisted telemetry; no score claimed as a live-model result. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Without task-relevant evaluation, a team cannot reliably tell whether a change improved its system. This lesson is about the discipline that prevents that.

Part 1 The vibe-check trap.

Vibe-checking is when you tweak a prompt, run one or two examples, and decide it’s better. It offers weak evidence because remembered examples may not represent the task or its critical failures.

Three problems with vibe-checking:

• You only test what you remember to test. A new prompt may be better at the case you ran and worse at the four you forgot.
• You can’t tell when the model regresses. When a provider updates a model silently, your output quality changes overnight.
• You can’t onboard anyone. “Quality is whatever I think looks good” doesn’t scale past one person.

The fix is an eval set: a fixed collection of test cases with expected behavior. You run the eval set after every change. The score moves up or down.

Part 2 What an eval set looks like.

A minimum-viable eval set has three fields per case:

EVAL_CASES = [
    {
        "id": "vet-translator-army-logistics",
        "input": "I was a 88M, E-6 in the Army, for 11 years. Drove convoys.",
        "must_contain": ["supply chain", "logistics"],
        "must_not_contain": ["88M", "Hooah"],
    },
    {
        "id": "vet-translator-marines-comms",
        "input": "I was a 0651, E-5 in the Marines, for 6 years. Field radios.",
        "must_contain": ["communications", "network"],
        "must_not_contain": ["0651", "Oorah"],
    },
    # ... 10 to 50 more cases ...
]

Then a runner:

def run_eval(test_fn):
    results = []
    for case in EVAL_CASES:
        output = test_fn(case["input"])
        passed = (
            all(s.lower() in output.lower() for s in case["must_contain"]) and
            not any(s.lower() in output.lower() for s in case["must_not_contain"])
        )
        results.append({"id": case["id"], "passed": passed, "output": output})
    score = sum(1 for r in results if r["passed"]) / len(results)
    return score, results

Twenty cases is a fine starting eval set. Once you have it, every prompt change runs through it. Use predeclared quality and critical-failure gates. One aggregate score can hide a harmful regression on a rare slice.

Part 3 Three kinds of evals you'll need.

• Deterministic checks. Does the output contain X? Does it parse as valid JSON? Does the number match? Cheap, fast, accurate when applicable.
• Reference comparison. Does the output match a known-good answer? Useful for translation, summarization, structured extraction. Cheap to implement, can be brittle.
• LLM-as-judge. A second LLM grades the output against a rubric. Expensive but flexible. Use it for subjective qualities (tone, helpfulness, structure).

Layer them. Deterministic first: anything that fails the cheap check shouldn’t go to the expensive one. LLM-as-judge only for the cases you can’t grade mechanically.

A judge is an instrument with its own failure modes. Blind model/version labels where feasible, randomize presentation order, anchor score meanings with examples, calibrate against qualified human ratings and treat candidate text as untrusted data. Record disagreement and adjudication. Refusal, partial output, invalid score and missing evidence are distinct errors.

def validate_judge(result):
    if result.get("status") != "complete":
        return "ungraded"
    grade = result.get("grade", {})
    if set(grade) != {"score", "evidence_id"} or type(grade["score"]) is not int:
        return "contract_error"
    if not 1 <= grade["score"] <= 5 or grade["evidence_id"] not in {"case-1"}:
        return "contract_error"
    return "reviewable"
assert validate_judge({"status": "refusal"}) == "ungraded"
assert validate_judge({"status": "complete", "grade": {"score": 9, "evidence_id": "case-1"}}) == "contract_error"
print("judge boundary checks passed")

These mocks test grade handling, not grader accuracy. A live model judge needs separately authorized use and its own evidence; it cannot alone pass a critical release gate.

Part 4 Observability: knowing what happened in production.

Evals tell you whether the system is good. Observability tells you what it actually did, request by request.

For each call, begin with allowlisted metadata:

• Request ID, task class and prompt/template version
• Result status and approved evidence identifiers
• Token counts (input, output)
• Latency
• Resource estimate from declared rate/usage fields, including retries and any separately billed tools or storage; label unknown fields
• Outcome: did the user keep the result or reject it?

Two patterns:

Log to a file or table

import json, time
from pathlib import Path

def log_call(request_id, model, status, latency_ms, usage):
    record = {
        "request_id": request_id,
        "ts": time.time(),
        "model": model,
        "status": status,
        "latency_ms": latency_ms,
        "input_tokens": getattr(usage, "input_tokens", None),
        "output_tokens": getattr(usage, "output_tokens", None),
    }
    Path("logs").mkdir(exist_ok=True)
    with open("logs/calls.jsonl", "a") as f:
        f.write(json.dumps(record) + "\n")

JSONL (one JSON object per line) is the right format for log files. You can cat and grep them. You can also pd.read_json("logs/calls.jsonl", lines=True) and analyze in pandas.

Use a tracing tool

For production work, look at tools like Langfuse, Helicone, or OpenLLMetry. They handle the storage, give you a dashboard, and support traces under configured capture rules. Disable content capture unless a documented diagnostic purpose, access policy, retention and deletion route permit it.

Don’t reach for them on day one. Select storage from access control, retention, durability and diagnostic needs; user count alone is not a security design.

Interface/reference limits

Use the workflow reference to connect task, model, deployment, harness/tools, evidence and the responsible human decision. Synthetic demonstrations establish only the stated tested conditions.