A compelling AI demonstration can make a purchasing decision look easy. The system summarizes a document, answers a difficult question, or completes a workflow in seconds. What the demonstration rarely reveals is where your data goes, how the system handles failure, or what happens when the underlying model changes.
Consider an AI assistant that drafts customer-service responses. Producing a polished answer is useful. Inventing a refund policy, exposing another customer’s information, or sending an unauthorized response is not. Vendor selection must distinguish between impressive output and dependable operation.
Choose an AI vendor based on evidence of business value, data protection, operational controls, and an achievable exit—not the quality of its demonstration. The following checklist turns that principle into a practical procurement process.
Start With Evidence and Shared Accountability
AI procurement should involve the business owner, IT, security, privacy, legal, and procurement—not become an isolated technology purchase. Assign a lead who can consolidate findings and resolve conflicting requirements.
For every checklist item, record four things:
- The vendor’s answer: What capability, commitment, or limitation is being represented?
- Supporting evidence: Which document, test, demonstration, or contractual provision supports it?
- Internal reviewer: Who is qualified and accountable to evaluate the answer?
- Unresolved conditions: What must change, who owns that change, and when must it be completed?
This checklist is a practical synthesis informed by sources including NIST’s AI Risk Management Framework. The framework is voluntary guidance, not a certification or a guarantee of legal compliance. Apply scrutiny proportionate to the intended use: an internal drafting tool and an autonomous payment agent require different controls.
The Eight-Point AI Vendor Selection Checklist
1. Define the Job Before Evaluating the Vendor
Start with a workflow, not a product category. “We need AI” is not a purchasing requirement. “We need to reduce the time employees spend finding approved policy information” is a testable objective.
- Identify the workflow, users, business owner, and intended benefits.
- Document current performance and measurable acceptance criteria.
- Specify prohibited uses and the consequences of incorrect results.
- Determine which decisions or actions require human approval.
Request: A use-case-specific implementation proposal, known limitations, and a written division of responsibilities.
For the customer-service assistant, separate drafting from sending. Define whether it may suggest refunds, access account information, or communicate externally. NIST’s AI RMF core guidance emphasizes intended use, business value, risk, and human oversight.
2. Map Where Your Data Goes
“We do not train on your data” is only the beginning of the conversation. It does not explain retention, human access, logging, or other secondary uses.
- Inventory prompts, uploaded files, outputs, logs, feedback, and any stored representations of your data.
- Identify hosting locations, subprocessors, model providers, and personnel with access.
- Confirm retention periods, deletion procedures, and backup exceptions.
- Clarify whether data supports training, fine-tuning, evaluation, or other secondary purposes.
Request: A data-flow diagram, subprocessor list, applicable privacy terms, and contractual data-use commitments.
Include support tickets and diagnostic logs in the review; sensitive information can leave the main application through troubleshooting. The FTC warns AI providers to honor privacy and confidentiality commitments. Ensure the contract reflects the promises made during sales discussions.
3. Investigate the Whole Supply Chain
You may be buying an interface built on someone else’s model, cloud platform, retrieval service, and external APIs. Those dependencies affect availability, security, and your ability to adapt.
- Identify underlying models, external APIs, retrieval services, and critical dependencies.
- Distinguish components the vendor controls from those it purchases.
- Review how suppliers and model changes are assessed.
- Identify fallback arrangements for dependency outages or withdrawal.
Request: An architecture diagram, component inventory, and supplier-risk process.
Ask what happens if an upstream provider becomes unavailable or changes its terms. Can your workflow continue safely? The NCSC’s secure-development guidance calls for assessing AI supply chains and maintaining secure, documented components.
4. Test Security—Not Just Paperwork
Independent assurance reports matter, but their scope matters more. Determine whether they cover the proposed service, relevant infrastructure, and controls you will actually use.
- Review identity management, access restrictions, encryption, tenant separation, and audit logging.
- Examine relevant independent assessments and remediation evidence.
- Test prompt injection and sensitive-data disclosure in generative AI systems.
- Test authorization boundaries and approval controls wherever AI can take actions.
Request: Security architecture, assessment scope, recent findings, and demonstrations of controls.
For example, place malicious instructions inside a test document and assess whether the assistant follows them. Test whether an action-taking system can exceed the user’s permissions. Conduct testing in an authorized environment. NIST’s Generative AI Profile recommends adversarial testing, including prompt-injection testing.
5. Run a Representative, Bounded Pilot
A pilot should answer a purchasing question, not simply generate enthusiasm. Define its scope, approved data, duration, and pass/fail criteria before results arrive.
- Use examples drawn from the intended workflow, with appropriate data approval.
- Include ambiguous inputs, edge cases, and consequential failure scenarios.
- Measure correctness, unsupported claims, consistency, latency, and human-review effort.
- Where relevant, evaluate performance across affected user groups.
- Document failures and limitations—not just successful outputs.
Request: Reproducible evaluation methods, test results, and documented limitations.
In the customer-service example, include incomplete account histories and conflicting policy documents. Measure how much correction employees perform. A fast draft that requires extensive checking may not improve the workflow. Record the model version and configuration so later comparisons remain meaningful.
6. Plan for Changes and Incidents
The system you approve may not behave identically after a model update, configuration change, or new data connection. Selection must include ongoing operating responsibilities.
- Assign responsibility for monitoring performance and security.
- Require notice of material model or service changes.
- Establish retesting triggers, pause controls, and fallback procedures.
- Agree on incident notification, support escalation, and response commitments.
- Protect monitoring logs according to their sensitivity.
Request: Monitoring demonstrations, incident procedures, release policies, and service-level commitments.
A fallback might mean disabling automated sending while preserving human-reviewed drafting. The NCSC’s operation and maintenance guidance recommends monitoring behavior and supporting customers’ evaluation of model changes. Clarify both vendor obligations and your internal response duties.
7. Negotiate Rights and Exit Conditions
A workable exit is part of a responsible purchase. Address it before your workflows and data become difficult to move.
- Clarify rights to inputs and outputs, including relevant usage restrictions.
- Have counsel review liability, indemnities, and applicable sector requirements.
- Specify export formats, deletion verification, transition support, and termination charges.
- Test whether another tool or a manual process can replace the service.
Request: Proposed contract language and a demonstrated export process.
Exporting documents alone may be insufficient. Depending on the workflow, you may also need configurations, evaluation records, prompts, and audit history. Verify that exports are usable, not merely downloadable. Do not assume that contractual output rights resolve every intellectual-property issue; counsel should assess the intended use.
8. Calculate the Full Cost
Advertised subscription or usage rates rarely capture the complete operating expense. Compare vendors against the same workload and control requirements.
- Include subscription fees and usage charges.
- Add integration, storage, support, and required security features.
- Estimate evaluation, monitoring, employee training, and human-review costs.
- Account for overages, renewal increases, and eventual migration.
Request: An itemized commercial proposal with pricing assumptions and relevant contract terms.
Use pilot measurements to build expected-use and high-use scenarios. Include repeated requests, document processing, and correction effort. Compare the cost of completing an acceptable task—not simply the cost of generating an answer. Keep projected benefits separate from benefits actually demonstrated.
Use Mandatory Gates Before Weighted Scoring
A weighted score can conceal an unacceptable risk. Strong functionality and attractive pricing should not compensate for unresolved critical data, security, authorization, or contractual requirements.
Define mandatory gates before comparing proposals. These might include acceptable data-use commitments, enforceable access controls, required human approvals, and workable deletion and exit terms. A vendor that fails a mandatory gate should not advance to production approval.
Then score acceptable vendors on workflow performance, operational fit, total cost, support, and portability. Set weights before reviewing final results so procurement does not become an exercise in justifying a favorite.
Record the decision as approve, approve with conditions, or reject. Conditional approval must specify an owner, deadline, verification method, and permitted scope. It must not become a workaround for unresolved mandatory requirements.
Make the Decision Defensible
The best AI vendor is not necessarily the one with the most impressive demonstration. It is the one that can deliver measurable value within boundaries your organization understands and can enforce.
Before your next vendor meeting, define the workflow, appoint the reviewers, and establish the mandatory gates. Ask for evidence, test representative failures, and verify the exit. Those steps turn an exciting technology purchase into a defensible business decision.