Leader's model of AI capability and accountability

Prerequisites: distinguish output from evidence; review the AI workflow reference. Public self-study is not official agency training, accreditation or a credential.

What you will be able to do

This lesson develops the ability to describe the sociotechnical system and assign accountable owners. The setting is NORTHSTAR Mission Portfolio, a fictional public-service portfolio containing ten proposed AI and non-AI interventions, three synthetic vendors, constrained staff capacity, and a staged incident. Every person, organization, system record, metric, and outcome in the case is fictional. Do not replace the case with operational, classified, CUI, export-controlled, proprietary, employer-confidential, or third-party personal information.

By the end of the lesson, you will be able to:

• Demonstrate how to describe the sociotechnical system and assign accountable owners.
• Apply that model to the synthetic NORTHSTAR Mission Portfolio environment.
• Produce a reviewable artifact that separates evidence, assumption, uncertainty, and authority.
• Recognize the critical failure and select a safe remediation path.

The performance standard is not confident language. It is a traceable decision, an artifact another qualified reviewer can inspect, a clear boundary around the learner’s authority, and a safe response to uncertainty. Knowledge checks are formative and do not award a certification or endorsement.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 System accountability

Leaders ask what work completes, what evidence supports it, what fails and who decides. Separate model, product interface, harness and compute. Harness permissions can turn an answer into action; local inference can use hosted tools. Model reputation does not establish system containment or reliability. Human review needs sources, expertise, time, visible uncertainty and rejection authority. Assign mission, data, engineering, operations and risk decisions separately. A responsibility map documents authority; it cannot grant it.

Diagnose whether a human control works

State the consequence first: a wrong opening time inconveniences a visitor; an invented benefit deadline can affect access to a service. Give reviewers the original source, the generated claim and a way to reject or escalate. Observe completion time and error detection instead of counting approvals.

Fictional test: ten drafts contain three planted wrong dates. Reviewer V1 receives only polished summaries and detects 0/3. Reviewer V2 receives source spans, a conflict flag and six minutes per draft and detects 3/3. This supports the V2 design under those ten cases, not a guarantee of all future detection. Add a missing-source case to see whether V2 abstains.

Decision	Responsible role in case	Evidence requested
Public draft quality	Mission owner	Source and reviewer results
Retrieval access	Data/security owners	Positive and negative role tests
Release change	Engineering/change process	Versioned regression results
Official benefit decision	Authorized official, unresolved	Applicable authority and domain review

Feedback: the leader cannot infer that the person called “reviewer” has legal or mission authority. Coverage and legitimacy are distinct questions.

Supplied planted-date review

Source notice P1: “Applications are accepted through 17:00 on 12 November. This notice does not state individual eligibility.” Bad draft D1: “You are eligible and must apply by 17:00 on 10 November.” Reviewer role: the fictional Notice Reviewer may check wording and sources, reject a draft and refer eligibility questions to the responsible human official. The reviewer may not decide eligibility.

Expected review: flag 10 November as contradicted by P1; replace it with 12 November only with the correct cited span. Remove the unsupported person-specific eligibility claim and provide a human contact route. Record both defects independently. A reviewer who fixes the date but leaves “you are eligible” has only partially completed the control.

Guided practice: mark each claim as supported, contradicted or absent from P1. Then compare a source-free review screen against one showing P1 and a reject button. Explain how the missing source or inability to reject prevents meaningful review. This one complete fixture is sufficient for the core exercise; the ten-draft detection figures above are invented aggregate evidence for the extended case.

Part 2 Synthetic worked example

NORTHSTAR calls a summary “reviewed,” but the reviewer has thirty seconds and no source access. The leader narrows output to a draft, restores source links and measures whether the reviewer detects a planted wrong date before allowing use.

Guided practice: identify the unsupported conclusion, name missing evidence and rewrite its scope. Change one assumption and identify which decision reopens.

Part 3 Feedback and repair

Pass when questions seek artifacts, model and harness are distinct and review capacity is measured. A committee or model is not the owner of an official decision.

Retain the failed version, explain the criterion, correct the artifact and test a different invented case.

Next step: carry the artifact and failed cases into the next module lesson or capstone. Retain limitations when explaining your work.