Valid JSON can still be wrong
Validate output structure, factual values and failure states separately.
Reviewed: 2026-10-03
Illustrated teaching example • Fictional data
Narration: Microsoft Mark · locally synthesized

00:00:00.000 — Structure is only the first check.

Structured output can make integration easier, but valid JSON does not establish truth. A response can satisfy a schema while inventing a value. We will separate parsing, schema validation, factual checks, and the response states that require different handling.

Visual description:
Parse → Shape → Facts → Outcome

00:00:21.292 — A structurally valid invented value.

This constructed object contains a team size and savings percentage in the expected types. It parses and may satisfy a basic schema. But the fictional source says six people and unknown savings. The object’s structure passes while its content fails.

Visual description:
{
  "team_size": 20,
  "savings_percent": 40
}
Constructed response · schema validity is not evidence

00:00:41.042 — Validate in layers.

First determine whether there is a complete usable response. Then parse it, validate its shape and allowed values, and check the facts against the source or calculation. Apply business rules and authorization separately. A single success flag should not collapse all of these decisions.

Visual description:
Response state
Parse + schema
Source + rules
Accept or reject
Each layer answers a different question.

00:01:02.250 — Reject the unsupported content.

Check team size against the source and reject twenty because the source says six. Treat savings as unknown rather than accepting forty. Design the data contract to represent an unknown where appropriate. A stricter schema still cannot manufacture missing evidence.

Visual description:
Schema passes. Source check fails.
team_size: 20: Expected 6 (fail)
savings_percent: 40: No measured value (fail)
Repair: Use supported value and explicit unknown (pass)

00:01:21.917 — Handle distinct response states.

Complete output, incomplete output, refusal, and transport or tool errors need different paths. Do not parse a refusal as the requested artifact or treat a timeout as an empty success. Record a safe status, preserve the useful partial state, and use bounded retries only when appropriate.

Visual description:
Complete: Validate shape and facts
Incomplete: Do not claim finished work (warning)
Refusal: Handle as its own state (warning)
Error: Recover or report failure (fail)

00:01:44.542 — Try a valid but false fixture.

In the lesson, supply one structurally valid but factually wrong object and one incomplete or refusal case. Show which validation catches each case and how the application responds. Passing means neither missing evidence nor a failure state becomes a misleading success.

Visual description:
Which check catches each failure?
Shape mismatch → schema.
Invented value → source check.
Refusal or timeout → state handling.
Your evidence: fixtures, validation results and safe responses.