Fine-tuning, post-training, and alignment

Lesson preparation and outcomes

Prerequisites: Held-out splits, baseline evals and model/data manifests.

By the end, you can: Choose prompting, lookup, tools or adaptation from the defect and evaluate changes.

Worked scenario and practice feedback

Stale policy needs current evidence, arithmetic needs a deterministic tool, and stable style may justify prompting or adaptation. SFT learns targets; DPO learns from preference pairs; LoRA changes effective updates. They are different choices, not one mandatory pipeline.

Guided and independent practice: Create the offline failure-routing and split manifest. Independently add an adapted output with better format but a retained-capability or critical disclosure regression.

Answer criteria: No mandatory RAG-first rule; data rights/coverage and rollback documented; held-out slice results govern selection, not training loss. If the local pack or interpreter is unavailable, trace the synthetic input, expected output and failure guard on paper; label execution unverified. Required practice uses no paid API calls, accounts, purchase, operational data or actual deployment.

Choose adaptation from task evidence. Prompting, retrieval, deterministic tools and fine-tuning address different failure mechanisms; they are not a mandatory staircase. This lesson covers when each one wins and how to actually fine-tune when the case calls for it.

Part 1 The decision tree before you fine-tune.

Diagnose the mechanism before choosing an intervention:

• Unclear instructions or unsupported output shape: revise prompts, examples and supported schemas, then evaluate.
• Missing changing evidence: supply current context, direct lookup or retrieval with source/version checks.
• Arithmetic or exact database answers: use a validated deterministic tool or query.
• Stable domain behavior, style or narrow task efficiency: evaluate supervised or preference adaptation against a baseline, subject to data rights and operational constraints.

Retrieval is not a prerequisite to tuning when it does not address the failure. There is no universal minimum dataset count: use held-out learning curves, coverage and uncertainty to decide whether available data is sufficient.

When fine-tuning genuinely wins:

• Style and tone matching. You need every output to read in a specific voice, and that voice is hard to specify in a prompt.
• Specialized format. You need consistent structured output the model struggles to produce reliably even with schema-mode tools.
• Latency or cost reduction. A smaller, fine-tuned model can match a bigger general model on a narrow task at lower inference cost.
• On-device or air-gapped deployment. You need a model small enough to run somewhere a large model can’t.

When fine-tuning loses:

• The domain knowledge is changing faster than you can re-train.
• Data coverage, rights or held-out evidence are insufficient for the intended claim.
• The base model is already at the quality bar.
• You don’t have the operations capability to maintain a fine-tuned model in production.

Part 2 The post-training stack: what each step does.

Modern foundation models are pre-trained, then post-trained. As an applied engineer you usually don’t pre-train, but you may do post-training. Know the steps.

• Pre-training. Self-supervised training on a massive corpus. Develops next-token capabilities under the corpus/objective; instruction following and refusal behavior must be evaluated rather than assumed absent or guaranteed. Disclosed pipelines differ and can include continued training; do not infer proprietary steps.
• Supervised fine-tuning (SFT) / instruction tuning. Training the base model on examples of (prompt, ideal-response) pairs. Makes it follow instructions. This is the step you are most likely to do yourself when fine-tuning for a specific task.
• Reinforcement learning from human feedback (RLHF). Training a separate reward model on human preference data, then using reinforcement learning to make the model produce responses humans prefer. Heavy machinery. Mostly done by frontier-model providers.
• Direct preference optimization (DPO). A preference-learning objective using chosen/rejected pairs without the same online reward-model/rollout procedure as common RLHF approaches. Outcomes depend on data and setup; no universal equivalence or improvement is promised.
• Constitutional AI / RLAIF. Variants that replace human feedback with rules or AI judgments. Anthropic uses constitutional AI; others have adopted similar approaches.

Choose an objective from task evidence, data type, supported training setup and operations constraints; do not infer which proprietary post-training steps were used.

Part 3 LoRA: low-rank adaptation and its trade-offs.

Full fine-tuning of a modern open-weights model updates every parameter. For a 7B model, that’s billions of parameters and requires serious GPU memory and time.

Low-Rank Adaptation (LoRA) is the trick that makes this tractable for applied teams. Instead of updating the original weights, LoRA adds small low-rank matrices to a few layers and only trains those. The original model stays frozen.

What you get:

• Fewer trainable parameters for a low-rank configuration; the ratio depends on rank and targeted modules
• May fit a local device for some model/configuration choices; measure memory and runtime for the chosen setup
• Easy to swap LoRA adapters: same base model, different fine-tuned behaviors
• Combines well with quantization (QLoRA) for even less memory

# Conceptual: real code uses peft / transformers
from peft import LoraConfig, get_peft_model

config = LoraConfig(
    r=16,                  # rank
    lora_alpha=32,         # scaling
    target_modules=["q_proj", "v_proj"],  # which attention projections to adapt
    lora_dropout=0.05,
)
model = get_peft_model(base_model, config)

LoRA can reduce trainable-parameter and optimizer memory; quantized-base adaptation can further reduce memory with compatibility and quality trade-offs. Check target modules and runtime support for the actual architecture. Neither technique is universally best.

Two practical notes:

• Quality ceiling. Compare base and adapted models on fixed held-out cases; parameter count does not establish a universal quality ordering.
• Inference still uses the adapter. Some runtimes load base and adapter separately; others support a merged artifact. Track base/adapter or merged revision, compatibility and rollback in either path.

Part 4 The dataset matters more than the technique.

A working SFT or LoRA pipeline is straightforward. Building a good dataset is the actual work.

Five disciplines that pay off:

• Curate every example. Poor labels and duplicated sources can overwhelm a larger dataset. Evaluate both quality and coverage with held-out learning curves instead of relying on a universal sample-count claim.
• Cover the failure modes. Include examples of the cases the base model gets wrong, with the right answers. Include corrected failures while preserving distribution coverage and retained capabilities.
• Vary the prompt formats. If you only train on one prompt structure, the model overfits to that structure and breaks on slight variations.
• Hold out a real test set. Same eval discipline as Tier 02 Lesson 08, applied to fine-tuning. Don’t trust loss; trust the eval.
• Version the dataset. Treat the training dataset like code. Git for the manifest, content-addressable storage for the data, a clear lineage from each model checkpoint back to the data that produced it.

Two failure modes to expect:

• Overfitting. Eval performance peaks early in training, then drops as the model memorizes the training set. Use the validation set to pick the right checkpoint, not the last one.
• Catastrophic forgetting. Fine-tuned model gets very good at the new task and loses general capability the base model had. Mitigations: include some general data in the SFT mix; LoRA preserves stored base weights but adapted behavior can still regress; test on capabilities you care about retaining.

Part 5 Operations for a fine-tuned model in production.

If you fine-tune and ship it, you own a new set of operational responsibilities.

• Versioning. Every fine-tuned model needs a version. Tie it to the base model version, the LoRA adapter version, the dataset version, and the training run identifier. When the eval moves, you need to be able to trace it.
• Re-evaluation cadence. Decide how often you re-run the full eval set. Choose cadence and change triggers from risk, task drift and controlling requirements; do not assume a universal monthly schedule.
• Rollback plan. What happens if a new fine-tune ships and quality drops? Have the rollback ready before the deploy.
• Provenance documentation. For federal work, document the training data sources, the licenses, and the consent posture. This shows up in ATO conversations and in NIST AI RMF audits.

Two questions to ask before you go to production with a fine-tuned model:

• What is the maintenance budget over the next 12 months? (Compute, engineering time, on-call.)
• Who owns the rollback decision when something goes wrong at 0200?

Interface/reference limits

Use the workflow reference to distinguish model capability, deployment and harness controls. Provider snippets in this lesson are illustrative, have not been executed against an actual provider in this revision, and require current SDK/model capability verification before separately authorized use.

Primary source checked October 2, 2026: OpenAI direct preference optimization.