Data engineering for AI applications

What you will be able to do

1. Validate mixed-format inputs against explicit schemas.
2. Preserve provenance, permission, and deletion state.
3. Separate training, tuning, and evaluation records.

Prerequisites: dictionaries, loops, functions, exceptions and installed Python or paper tracing. Reuse earlier Builder artifacts only when the task requires them.

This curriculum runs self-study on VetsLearn. Where the text calls for an instructor, peer or formal review, use the stated acceptance criteria for self-review. Reflection questions support reflection; they do not replace independent assessment. A peer review is optional unless your actual organization requires it.

Part 1 Technical instruction

Data accessibility is not authorization. An ingestion pipeline should carry a record identifier, source identifier, owner, authorized purpose, access label, acquisition time, transformation version, and deletion state alongside content. Validation distinguishes structural errors from policy errors. Idempotency means rerunning the same input does not duplicate or mutate records unexpectedly. Quarantine corrupt records; do not silently coerce values that change meaning. Split data before fitting transformations and preserve a manifest that lets a reviewer reconstruct every output.

Mechanism and implementation

Stable identity precedes transformation. Preserve source and revision, parse dates explicitly, quarantine invalid records and define duplicate rules. Split by source/entity/time before fitting transformations. Deletion must propagate to derived chunks, indexes and caches; removing only a source file leaves copies.

Constructed offline example

Save as topic_example.py and run python topic_example.py with an installed Python 3.11+ interpreter. This standard-library construction uses synthetic values; its checks are not provider measurements.

rows = [{"id": "a", "owner": "one", "value": 3},
        {"id": "a", "owner": "one", "value": 3}, {"id": "b", "value": 7}]
accepted, quarantine = {}, []
for row in rows:
    if not row.get("owner"):
        quarantine.append(row["id"])
    else:
        accepted.setdefault(row["id"], dict(row))
assert list(accepted) == ["a"]
assert quarantine == ["b"]
print("one accepted ID, one quarantined ID")

Engineering decisions

• Define reject, quarantine, and repair rules.
• Select stable identifiers before transformation.
• Decide which metadata must survive every derived artifact.

Worked example

A synthetic intake combines CSV and JSONL. The pipeline rejects invalid dates, quarantines missing owner fields, deduplicates on a stable case ID, and produces a lineage manifest with input hashes. Reprocessing yields byte-equivalent normalized output.

Diagnose the failure

Identical reprocessing is idempotent; conflicting duplicates need an explicit rule. If a second run changes counts, inspect state and identifiers before blaming model quality.

Part 2 Guided practice

Predict counts and run. Add a conflicting duplicate with changed value; quarantine it rather than silently choosing the last record. Independently delete a source and prove its derived IDs disappear.

Answer criteria and feedback: Identical reprocessing is idempotent; conflicting duplicates need an explicit rule. If a second run changes counts, inspect state and identifiers before blaming model quality.

Continue with the downloadable pack below. If the pack or interpreter is unavailable, trace this self-contained example by hand in an input/expected-status/controlling-rule table and label execution unverified. Do not invent passing output. No account, credential, paid call, purchase or actual deployment is required.

Part 3 Remediation

Inspect the validation report before changing code. Repair one error class at a time and keep rejected source rows unchanged for comparison. After repair, reassess with the alternate scenario in scenario_alt.json. A critical authorization, data-handling, evidence-integrity, or prohibited-action failure cannot be averaged into successful practice; it requires targeted remediation.

Part 4 Sources and currency

• https://airc.nist.gov/airmf-resources/airmf/
• https://www.rfc-editor.org/rfc/rfc4180

Use the workflow decision reference for model/deployment/harness distinctions and controlled comparisons. Source currency must be checked before each cohort. Provider/product behavior is illustrative where mocked; resolve current interfaces only from official documentation in an authorized environment.