Capstone: production AI in a federal context

Lesson preparation and outcomes

Prerequisites: All prior Expert evidence and operations concepts.

By the end, you can: Defend task/model/deployment/harness/evidence choices and rehearse recovery.

Worked scenario and practice feedback

A synthetic documentation assistant passes routine answers but fails cross-tenant access after a configuration change. The package is no-go for its claimed scope: preserve the case, contain, repair, roll back and verify. A polished briefing cannot replace missing evidence.

Guided and independent practice: Complete the offline capstone artifacts; independently run a change/incident exercise and technical board with a no-go outcome available.

Answer criteria: Reproducible normal, denial and rollback results; mocked/unexecuted components explicit; no ATO, clearance, job readiness or real federal shipment claimed. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Tier 02 had a capstone. This one is bigger. Tier 02 practiced a reproducible build. This capstone integrates engineering and federal-context decision evidence using a synthetic sandbox. It does not prove authorization to ship into a mission environment, independent expertise or job readiness.

Part 1 What makes a good Expert capstone.

The Tier 02 filters still apply: it solves your own problem, it justifies only the components needed for the task, and it is small enough to ship. Expert adds three more:

• It demonstrates production-related controls offline. Use a synthetic simulator with monitoring signals, scoped logging, evals, declared SLOs and verified recovery; actual service operation is a separate authorized decision.
• It carries an illustrative documentation package. System card, governance, eval framework, risk register and security artifacts for this synthetic case. Actual program requirements depend on the system, mission and responsible authorities; this package is not an authorization submission.
• It demonstrates depth. Not just “I built a RAG.” The capstone shows judgment on at least one hard engineering or compliance choice you can defend.

Three project shapes that work well for an Expert capstone:

• A synthetic public-document AI tool. On-prem open-weights model, RAG over a public-document corpus (FAR, DFARS, federal regulations, AI policy memos), structured-output API, eval set with bias measurement, SBOM, runbook.
• A federal-context fine-tuned model. LoRA fine-tune of a small open-weights model for a specific federal task: translating military experience for civilian resumes, summarizing federal solicitations, extracting structured data from acquisition documents.
• An agentic system with strict scope. An agent that helps a specific federal user role (acquisition analyst, transition counselor, security officer) complete a defined workflow. Tight scope, narrow tools, demonstrable safety controls.

What doesn’t work:

• A clone of an existing commercial product without a federal-specific angle.
• A demo with no reproducible evaluation or recovery evidence. Actual deployment is not required by this classroom capstone.
• A project that’s mostly UI polish and minimal engineering depth.

Part 2 The deliverable list.

By the end of the capstone build, you have all of these:

1. A local repository or private archive. Clear README and setup; document rights for included synthetic/source material. Public hosting and a particular license are optional decisions.
2. A reproducible offline system or simulator. Exact commands, synthetic inputs and explicit mocked components; no paid service or actual deployment required.
3. A complete eval framework. 50+ test cases. Automated runner. Executed results from the actual build period; never fabricate historical runs.
4. A NIST AI RMF-aligned documentation package from Lesson 07. System card, governance, risk register.
5. Security artifacts. Threat model, SBOM, dependency vulnerability scan results, the security findings from Lesson 06’s drill.
6. A draft compliance scaffolding from Lesson 08. System Security Plan skeleton, three proposed/tested sandbox controls whose status is explicitly recorded, STIG status notes.
7. Operational documentation. Runbook, SLOs, incident response procedures.
8. A capability statement featuring the capstone (Lesson 09).
9. An optional 90-second local walkthrough demonstrating the system at work.
10. A 5-page case study: the engineering decisions you made and why, written for a federal evaluator audience.

This bundle is what you bring to interviews. It is what you reference in proposals. It is the answer when someone asks “have you actually built something at this level?”

Part 3 The architecture decision record (ADR) habit.

For at least three meaningful technical decisions in your capstone, write an ADR, a short document explaining what you decided, why, and what alternatives you rejected.

A working ADR template:

# ADR 001: Evaluate a local serving candidate

## Status
Proposed; no live benchmark or deployment executed

## Context
Fictional task target: sub-second first useful content for a declared synthetic workload.
No hardware capacity has been demonstrated. Operational data and authorization remain out of scope.

## Decision
Use an offline serving simulator for course evidence. Record a local engine as a candidate
for future separately authorized benchmarking, with required schema/tool/state capabilities.

## Alternatives considered
- Another local serving engine: compare on the same cases and workload; no throughput ranking assumed.
- Direct runtime: evaluate batching, access and observability requirements; no unsupported failure claim.
- Hosted endpoint: evaluate exact approved data boundary, capabilities and total task cost.

## Consequences
The simulation demonstrates admission/readiness/recovery rules, not accelerator capacity.
A future operational route requires measured quality/latency, security review and approval.

ADRs accomplish three things:

• They force you to be specific about your decisions.
• They expose your rationale and alternatives for an evaluator to inspect.
• They make your work portable: anyone joining the project later has a record of why things are the way they are.

ADRs make alternatives and consequences reviewable. Tie each decision to requirements and evidence rather than assuming reviewer preferences.

Part 4 Honest scoping at the Expert tier.

The hard part of an Expert capstone is keeping it small enough to actually finish while still showing depth.

Discipline that pays off:

• Pick one capability done well, not three done okay. “An on-prem RAG with proper evals and a runbook” beats “A RAG plus an agent plus a fine-tune all half-built.”
• Pick one customer. Yourself, a specific veteran cohort, a specific federal user role. Generalizing comes later.
• Pick one reproducible environment. Define the local synthetic build or simulator first. Actual deployment is a separate authorized activity.
• Cut the UI. A working CLI or basic API is fine. Prioritize accessibility and task usability while keeping the interface within scope.

When you feel the urge to add a feature, write it down in a FUTURE_WORK.md instead. The list itself is a strong signal to evaluators that you have judgment about scope.

Part 5 The presentation that lands.

You will present this capstone in interviews. Sometimes formally, sometimes informally. The presentation that lands is short and specific.

A working structure for a 10-minute capstone presentation:

1. The problem. One slide. Plain English. Who has it. Why it matters.
2. The system. One slide. Architecture diagram. What’s running where.
3. Demonstration. Two minutes using supplied synthetic inputs. Run it locally, show a recording, or trace it on paper if execution is unavailable. Identify mocks and distinguish observed results from expected ones; include a failure or denied action.
4. The hard choice. One slide. The ADR you’d defend. “We chose X over Y because…”
5. What it gets wrong. One slide. Honest. You measured this; here’s the gap.
6. Compliance posture. One slide. NIST AI RMF alignment, eval framework, security testing.
7. What I’d do next. One slide. The honest roadmap.

What evaluators take away:

• You can describe the problem in plain English (Foundations 03 still applies).
• You built it. The demo works.
• You can defend a hard choice with engineering reasoning.
• You measured what you built, and you’re honest about gaps.
• You understand the federal compliance frame.
• You have judgment about what comes next.

What they downvote:

• “Let me tell you about my architecture” without explaining the problem.
• Recorded demos that gloss over real behavior.
• Refusing to acknowledge limitations.
• A roadmap that sounds like an MBA case study, not engineering.

Part 6 What the evidence establishes.

A completed capstone provides a bounded demonstration of implementation, tests, documentation and a recovery exercise under declared conditions. State executed versus mocked components, dataset provenance, model/SDK/runtime revisions, missing cases and unresolved decisions. Course completion does not confer an ATO, clearance, contracting authority, independent certification or employment qualification.

A reviewer should reproduce a normal case, a denied action and a rollback. Explain who would need to approve the actual data, deployment, permissions and residual risk before any operational transfer. Keep the synthetic demo private or share only reviewed artifacts voluntarily.

Interface/reference limits

Use the workflow reference to connect task, model, deployment, harness/tools, evidence and the responsible human decision. Synthetic demonstrations establish only the stated tested conditions.