Most AI vendor evaluations go wrong at the same point. The team runs a proof of concept, the model performs well on a handful of impressive tasks, procurement asks for a security questionnaire, and the contract gets signed with a marketing page as the main evidence of what was purchased. Six months later the model behind the alias has been swapped, the price has stepped up at a usage tier nobody modeled, and the data processing terms turn out to be less specific than the slide deck.
The fix is to treat the purchase as two decisions. The first is whether the capability is good enough for your task. The second is whether the dependency is acceptable. The twenty-five questions below are organized to answer both, and they are written to be answerable in writing, because an answer that only exists in a sales call is not an answer.
Start with the decision, not the questionnaire
Decide before you send anything whether you are buying a capability or a relationship. A capability can be replaced in a sprint if the evaluations are honest and the integration is thin. A relationship, meaning a fine-tuned model on your data with your prompts embedded in product logic, is a multi-year commitment. Most teams accidentally buy the second while telling themselves they bought the first. The questions that follow get harder as you move down the list, so use the early ones to disqualify fast.
Write a two-page memo before the first call. It should state the task, the measurable definition of good enough, the volume you expect over twelve months, and what happens to the product if the vendor disappears in a year. That last question is the one that changes the shape of the deal, because it decides whether you need export rights and a thin integration or whether a deep dependency is acceptable. Get the memo signed by the person who owns the budget, not just the engineer who will carry the pager.
Model provenance and change control
- Which base models do you use, and can you pin an exact version rather than a floating alias?
- What notice do you give before changing, upgrading, or retiring a model in production?
- Will you disclose the model providers and any subprocessors involved in serving our requests?
- Are you fine-tuning, distilling, or routing between providers, and can that routing change without a contract amendment?
- When a model is deprecated, who pays for the evaluation and re-integration work?
The first question matters more than it looks. A version alias means the vendor decides when your behavior changes. If the answer is that only an alias is available, ask for a written commitment that the aliased model will remain available for the contract term and that changes carry at least thirty days notice. Then build the evaluation suite so a swap is a test run, not an incident.
Data handling: retention, training, residency
- Is our content used to train or improve any model, and is that opt-out contractual or merely a dashboard setting?
- What retention applies to prompts, outputs, embeddings, and logs, and can we set it to zero for sensitive workloads?
- Where is data processed and stored, and can we pin residency to a region per workload rather than per account?
- Which of your staff can read our content, under what approval, and is that access logged?
- What is your subprocessor list and change notification process?
- How does deletion propagate to backups, caches, and vector indexes, and within what window?
Read the answer to question 6 against the contract, not the trust center. The major cloud platforms publish data protection terms that state customer content is not used to train foundation models without permission, but third-party applications built on top of those platforms set their own terms. NIST's Generative AI Profile is a useful checklist for the governance side of this, though it will not resolve a contractual ambiguity. When the answer is ambiguous, ask for the sentence you can put in the contract.
Evaluation and quality evidence
- Can we run our own evaluation set against the pinned version before signature, and after every model or prompt change?
- What accuracy or error rate do you observe on tasks like ours, and how was it measured?
- How do you detect quality regressions, and what is the rollback path if a release degrades?
- What is the failure behavior when your upstream provider is degraded: fallback model, refusal, or latency spike?
- Do we get per-call traces and logs for debugging, in a format we can ingest, and for how long?
Vendors will quote benchmark scores. Benchmarks measure the model, not your task. Ask instead for the size and composition of their internal evaluation set and whether they will accept yours. A vendor that runs your eval before signature and reports the failures is a better partner than one that reports a composite score.
Reliability, SLA, and incident terms
- What availability and latency commitments do you offer, with what credits, and are upstream outages excluded from the calculation?
- Which security attestations do you hold, what is the scope of the report, and can we see the current one under NDA?
- How do you isolate our data and model artifacts from other tenants, and what happens to them at termination?
- What is your incident notification window, root-cause commitment, and escalation path for a severity-one event?
An SLA measured monthly against 99.9 percent availability tells you almost nothing about a service whose failure mode is intermittent quality collapse. Ask for the error budget policy and for the historical incident list with durations. If the vendor will not share incident history before signature, that is your answer about what post-signature support looks like.
Commercial terms: pricing cliffs and escape hatches
- What are the pricing units, and which of them grows faster than usage: tokens, seats, ingested documents, or egress?
- Show the price at three times and ten times current volume, including steps, minimums, overage rates, and annual true-ups.
- What are our rights if you are acquired, if the underlying model is discontinued, or if a model provider changes its terms?
- What is the exit: export format, ownership of fine-tuned artifacts, termination assistance, and wind-down period?
- What counts as a material change requiring renegotiation, and how much notice must you give?
Pricing cliffs are the most common source of post-signature surprise, because AI pricing often has a committed-spend tier whose overage rate is punitive and a per-seat component that scales with headcount rather than value. Model three scenarios: usage doubles, usage falls by half, and one business unit is sold off. The third one is where seat-based contracts hurt.
How to score the answers
| Area | Good answer looks like | Walk away when |
|---|---|---|
| Provenance | Exact version pin, named providers, 30-day change notice | Floating alias with no deprecation commitment |
| Data | No training on customer content, contractual, zero retention option, deletion SLA | Training opt-out only in a settings panel |
| Evaluation | Runs your eval set, shares failures, rollback path documented | Benchmark scores as the main evidence |
| SLA | Incident history shared, credits defined, root cause committed | Availability number only, no incident list |
| Commercial | Priced at 3x and 10x, export rights, material-change clause | Overage rates withheld until signature |
How to read an evasive answer
Vendors are not obliged to be evasive, and the pattern is usually visible in how a question is answered rather than whether it is. An answer that restates a commitment without a specific mechanism is a signal: the vendor says it never trains on customer data but cannot point to the contractual sentence, or says it holds a security certification but will not share the report scope. Ask the follow-up question once, in writing. If the second answer is still general, treat it as a no and plan accordingly, because the same ambiguity will be there when you need it resolved during an incident.
Two answers deserve special weight. A vendor who will run your evaluation set and report the failures is telling you how it operates internally. A vendor who will put the export format and wind-down assistance in the contract is telling you it expects to keep enterprise customers. Neither promise costs much to make and both are hard to fake in the middle of a renewal.
What to do next
- Write the version-pin and change-notice requirements into the contract before the security review starts.
- Build a 30 to 50 item evaluation set from your own tasks and make re-running it a condition of every release.
- Model total cost at three times and ten times current volume, including overage and seat growth.
- Negotiate export format, fine-tuned artifact ownership, and wind-down assistance while you still have leverage.
- Put the due diligence answers in a file next to the contract so the next team inherits evidence instead of opinions.