# Vendor Bake-Off Plan
Pick a vendor by running a real test, not a slide deck.

_A two-week plan to evaluate 2-3 AI vendors against your actual use case. Forces real test data, a weighted rubric, and a signed decision._

Used in: Lesson leadership/02 — https://vetslearn.com/lessons/leadership/02
Source: https://vetslearn.com/resources/templates/engineering#t-bakeoff-plan

---

# Bake-Off Plan: [USE CASE]

**Vendors under test:** [VENDORS]
**Owner:** [OWNER]
**Budget for the bake-off:** [BUDGET]
**Decision date:** [DECISION DATE]

---

## 1. Use case

*Fill this in: one paragraph. What the system needs to do, for whom, at what volume. Be specific.*

## 2. Success criteria

*Fill this in: 3-5 measurable criteria. Each one must have a number or a pass/fail bar.*

- Quality: [metric and target, e.g. 90% pass rate on the eval set]
- Latency: p95 under [N] seconds
- Cost: under [$X] per 1,000 calls
- Security: meets [SOC2 / FedRAMP / on-prem requirement]
- Support: response time under [N] hours

## 3. Test data

*Fill this in: the actual eval set. Pulled from real tickets, real docs, real user inputs. No synthetic-only data.*

- Size: [N] cases
- Source: [text]
- Tags: happy / edge / adversarial
- Owner: [name]
- Location: [repo or shared drive]

## 4. Evaluation rubric (weighted)

| Criterion | Weight | How scored |
|---|---|---|
| Quality on eval set | 40% | Pass rate, judged by rubric + 20% human review |
| Latency | 15% | p95 under load test |
| Cost at projected volume | 15% | Vendor pricing applied to forecast |
| Security and compliance | 15% | Checklist pass/fail |
| Developer experience | 10% | 1-5 score from 2 engineers |
| Support and roadmap | 5% | 1-5 score from sales calls |

## 5. Hands-on test plan

### Week 1 — Setup and run

- Day 1: kickoff with each vendor, get API keys, get access
- Day 2-3: build the same thin client against each vendor
- Day 4: run the eval set against all vendors
- Day 5: load test for latency and cost at projected volume

### Week 2 — Score and decide

- Day 1-2: human review of 20% sample per vendor
- Day 3: security and compliance checklist per vendor
- Day 4: score the rubric, write the decision memo
- Day 5: present to stakeholders, decision

## 6. Cost model

*Fill this in: projected monthly cost at expected volume for each vendor. Include hidden costs — egress, fine-tune, support tier.*

| Vendor | Cost per call | Monthly volume | Monthly cost | Annual cost |
|---|---|---|---|---|
| [A] | [$] | [N] | [$] | [$] |
| [B] | [$] | [N] | [$] | [$] |

## 7. Final decision matrix

| Criterion | Weight | Vendor A | Vendor B | Vendor C |
|---|---|---|---|---|
| Quality | 40% | | | |
| Latency | 15% | | | |
| Cost | 15% | | | |
| Security | 15% | | | |
| DX | 10% | | | |
| Support | 5% | | | |
| **Total** | 100% | | | |

## 8. Sign-off

- Engineering lead: ____________ Date: ______
- Product lead: ____________ Date: ______
- Security: ____________ Date: ______
- Finance: ____________ Date: ______

---

## How to use

Do not skip the real-data step. Vendor demos are tuned to vendor-friendly examples — your eval set is the only fair test. Two weeks is the right size. Longer drags. Shorter cuts corners. If a vendor will not give you API access for the bake-off, that is a finding.

## Check before you use it

- Test data comes from real tickets and real documents. Vendor demos are tuned for synthetic.
- The same thin client runs against every vendor. Different harnesses invalidate the scores.
- The rubric weights are agreed before any scores come in.
- The cost model includes the hidden lines: egress, fine-tune, support tier.