Workflow evaluation and operational metrics

Test a workflow’s important failures, not just its average score.

Visual lessons: 1 · 1:41 total Read estimate: 3 min Practice estimate: 4 hr Updated: 2026-10-03

In this lesson · A6 of 8

Progress for this track is saved in this browser only. No completion badge is issued. View your learning.

Before you practice

Choose the model, environment and tools for this task. Record the actual configuration and test its limits.

Start with the lesson’s synthetic or offline exercise. A paid model, account or live tool is not required to learn the decision process. Optional live experiments require your own authorized setup and budget.

Configuration and evidence checklist

Task and standard: The artifact you need, its audience, and the checks it must pass.
Model and capability: Exact model identifier or visible product label; version if available. Record unknowns. Check supported modalities, tools, schemas, context and reasoning settings.
Application and harness: Product/runtime and version; how it selects context, routes tools, enforces approvals, stores state and resumes work.
Execution and data path: Where inference, files, tools, retrieval, embeddings, logs and backups run. Identify every remote destination.
Inputs and permissions: Use supplied synthetic data for training. Record allowed files, tools, network access and actions; enforce boundaries outside the prompt.
Limits and fallback: Set deadline, output size, step/retry limits and approved spend ceiling before live work. Offline exercises need no paid calls. State what happens on timeout, refusal or unavailable features.
Evidence and review: Source versions, observed outputs, tests, corrections, unresolved issues and the human responsible for consequential decisions.

These fields are a worksheet. They are not a claim that a particular provider or configuration has been tested by VetsLearn.

Download a session record template

Choose a model, environment and workflow →

Start with the idea

Test a workflow’s important failures, not just its average score.

Decide what counts as success before testing. Include cases with missing evidence and blocked actions. Report those results separately from routine cases.

Release gate: A required check that must pass before a workflow is allowed to move forward.

See the example and check your understanding

A made-up teaching example

The made-up workflow answers eight routine cases correctly. It guesses on two missing-evidence cases and tries to send on two prohibited-action cases. Faster drafting does not cancel those failures.

Ask yourself

Can many correct routine answers outweigh an unauthorized send?

Show the explanation

No. Apply the critical rule separately. Stop or revise the workflow rather than hiding the failure inside an average.

Start with these checks

Set expected behavior before the test.
Report counts for each case type.
Include checking time in effort measurements.

Use the different practice case below and check every requirement in its full instructions.

Go to the practice task ↓

The video explains the main idea. Use the sources, instructions and checks below to complete the practice.

Visual lesson1:41

Captions appear here during playback.

Workflow evaluation and operational metrics

Test a workflow’s important failures, not just its average score.

Playback speed Download transcript Open video file

Source-based teaching example · Fictional data · Reviewed 2026-10-03. Microsoft Mark · locally synthesized. On a small screen, use fullscreen for diagram details or read the transcript.

Read the descriptive transcript

0:00 · What you will learn

Test a workflow’s important failures, not just its average score.

What appears on screen

Test a workflow’s important failures, not just its average score.

0:07 · The idea

Decide what counts as success before testing. Include cases with missing evidence and blocked actions. Report those results separately from routine cases.

What appears on screen

Decide what counts as success before testing. Include cases with missing evidence and blocked actions. Report those results separately from routine cases.

0:20 · One useful term

Release gate. A required check that must pass before a workflow is allowed to move forward.

What appears on screen

A required check that must pass before a workflow is allowed to move forward.

0:29 · A worked example

The made-up workflow answers eight routine cases correctly. It guesses on two missing-evidence cases and tries to send on two prohibited-action cases. Faster drafting does not cancel those failures.

What appears on screen

The made-up workflow answers eight routine cases correctly. It guesses on two missing-evidence cases and tries to send on two prohibited-action cases. Faster drafting does not cancel those failures.

0:45 · Your turn to think

Can many correct routine answers outweigh an unauthorized send? Pause and decide before continuing.

What appears on screen

Can many correct routine answers outweigh an unauthorized send? Pause the video if you need more time.

1:02 · Why this answer works

No. Apply the critical rule separately. Stop or revise the workflow rather than hiding the failure inside an average.

What appears on screen

No. Apply the critical rule separately. Stop or revise the workflow rather than hiding the failure inside an average.

1:13 · Check 1

Set expected behavior before the test.

What appears on screen

Set expected behavior before the test.

1:18 · Check 2

Report counts for each case type.

What appears on screen

Report counts for each case type.

1:24 · Check 3

Include checking time in effort measurements.

What appears on screen

Include checking time in effort measurements.

1:29 · Now try a different case

Open the lesson and try its different practice case. Use every check in the full task. Watching alone does not mark the lesson complete.

What appears on screen

Open the lesson Try the new case Check and revise The full task and its requirements are in the linked lesson.

Watching explains the example. Complete the lesson practice to check what you can do. Explore all visual lessons →

Before you start

Prerequisite: A1 baseline and A2 declared acceptance criteria. Offline/no-cost route: paper, calculator and the synthetic facts below. Optional tools must already be authorized. No paid API, private workplace record, account creation or external action is required.

What you will be able to do

Build routine, edge, abstention, adversarial, and prohibited cases.
Measure quality, time, burden, severity, and acceptance.
Separate model, process, data, and test failure.

Part 1 A favorable average can hide unacceptable failure

Construct routine, edge, missing-evidence, adversarial and prohibited-action cases before evaluating. Define expected behavior and severity in advance. Score quality, total effort, review burden, refusals and critical errors separately; averaging them can hide unacceptable failures. Compare baseline and candidate on the same task mix and review standards. Repeat or vary runs when stochastic behavior matters and retain version/configuration. Small synthetic samples reveal defects but cannot establish broad reliability. Identify model, retrieval/data, harness/process and test-design failures so the repair targets the cause. Release, constrain, revise and stop are distinct recommendations with named owners and residual limits.

Part 2 Worked synthetic example

Synthetic evaluation: 8 routine cases all correct, 2 missing-evidence cases guessed instead of abstaining, 2 prohibited-action cases attempted a send. Aggregate factual success 8/12 is incomplete: abstention=0/2 and unauthorized-action avoidance=0/2. Even 100 routine successes would not excuse those critical failures. Baseline total effort 150 minutes, candidate 130, but critical gate fails. Decision: stop external action capability and revise; no deployment recommendation follows from 20 minutes less effort.

Part 3 Guided practice and feedback

Set gates before scoring: zero unauthorized actions; all missing-evidence cases explicitly unresolved; at least 7/8 routine cases correct in this training set. Given the results, the candidate fails first two gates. These are exercise criteria, not universal production thresholds. If a gold answer itself omits a source conflict, repair the test and rescore transparently.

Practice exercise · 240 MIN

Independent practice

Design a 30-case evaluation with expected behavior and severity: 12 routine, 6 edge, 6 missing/conflicting evidence, 3 injections and 3 prohibited actions. The lesson pack supplies literal inputs, complete constructed candidate drafts/traces and timing records for this mix. Declare your criteria from the inputs before opening candidates or review keys. You can adapt cases, but record changed versions and update their expected answers; old candidate results apply only to unchanged inputs. Compare the supplied manual baseline with candidate effort including correction, compute slice metrics and propose a bounded gate decision. All supplied timings/results are fictional, not provider measurements. No paid inference is needed.

Acceptance and feedback: Case expectations declared first; denominator/effort reproducible; critical failures not averaged away; data/model/process/test errors distinguished; no claim of population reliability from 30 invented cases; owner and fallback named.

Lesson pack: a6-learner-assets.zip. Current versioned synthetic sources, literal exercise inputs, worksheets and review keys are included. Read the sources and declare your checks before opening feedback. These are formative training assets, not a credential or operational approval.

Review and next step

Self-review is learning evidence, not independent certification or operational approval. A7 turns failures and configuration changes into change/incident procedures.

Substantive review: 2026-10-02. See the configuration guide and the NIST AI RMF, a voluntary framework rather than a course authorization or credential.

Lesson status:PRACTICE REVIEW REQUIRED

Review your practice