An AI assistant answers a customer’s question clearly, cites a policy, and recommends a refund. The response looks useful. But the policy is outdated, the citation does not support the recommendation, and the assistant has permission to issue the refund without approval.
This hypothetical scenario illustrates a central problem: a convincing answer is not necessarily a reliable outcome. Reliability depends on the entire workflow—the information available, the task assigned, the permissions granted, and the checks applied before anything consequential happens.
For business leaders, the question is not simply, “Can we trust AI?” It is: “For this task, under these conditions, what evidence and controls justify using it?”
The checklist below translates principles from NIST’s AI Risk Management Framework 1.0, its Generative AI Profile, and OWASP guidance into practical operating steps. It is a working checklist, not a certification or a guarantee.
AI reliability is an operating discipline—not a promise made by a model.
Before Deployment: Define What Reliable Means
☐ Define the job, its limits, and its owner
“Help the customer support team” is too broad to test. “Draft responses to billing questions using approved policy documents, without issuing refunds or changing accounts” establishes a usable boundary.
For each workflow, document:
- Allowed tasks: What the system should produce or accomplish.
- Prohibited uses: Decisions, topics, and actions outside its authority.
- Users and consequences: Who relies on it and what an incorrect result could cause.
- Accountability: A named owner authorized to approve changes or suspend use.
Set acceptance criteria before testing. Specify which factual errors are unacceptable, which outputs require review, and which failures block deployment. A brainstorming assistant and an account-access assistant should not face identical requirements: their consequences differ.
☐ Establish data boundaries
Inventory what the system can access through prompts, uploaded files, connected applications, and document retrieval. Identify personal, confidential, proprietary, and privileged information before connecting it.
Verify the applicable vendor terms and configuration, including retention, use of submitted data, access controls, and deletion options. Do not assume that a familiar product name means every subscription or deployment has identical protections.
Keep passwords and API keys out of prompts. Where integrations need credentials, use controlled authentication mechanisms rather than exposing secrets to the model.
For example, an internal policy assistant may need approved employee guidance. That does not mean it needs access to individual personnel files. Reduce access to what the task actually requires.
☐ Build realistic tests
A polished demonstration shows that a system can succeed. Testing should reveal where it fails.
Create a repeatable evaluation set with expected answers or scoring criteria. Include ordinary requests, ambiguous wording, missing information, conflicting documents, outdated policies, and adversarial inputs. Where relevant, test differences across languages, accessibility needs, and user groups.
Evaluate the complete workflow: retrieval, generated responses, reviewer decisions, and connected tools—not just the model’s text.
For a policy assistant, test what happens when an obsolete document contradicts the current policy. Does it identify the conflict, choose the authoritative source, or confidently blend both? Record results by failure type and severity. A single average score can hide a critical weakness.
Before Trusting an Answer: Check Evidence and Uncertainty
☐ Verify consequential claims
A citation is a starting point for verification, not proof. NIST’s Generative AI Profile identifies fabricated citations as a risk and recommends checking sources.
Before relying on an answer that could affect customers, finances, legal obligations, security, or reputation:
- Open cited sources and confirm that they exist and support the specific claim.
- Check dates, names, quotations, statistics, and policy versions.
- Recalculate important figures with an appropriate independent tool.
- Distinguish what the source states from what the model inferred.
Suppose an AI-generated proposal claims that a contract allows early termination. The reviewer should inspect the relevant clause, its conditions, and any amendments—not approve the statement because the answer includes a document title.
Keep verification proportionate to consequences. An internal brainstorming draft does not need the same scrutiny as contractual advice.
☐ Make uncertainty actionable
“Be accurate” is not an operating procedure. Define what the system should do when it lacks enough information or encounters a question outside its scope.
Useful rules include:
- Ask for clarification when missing details could change the answer.
- Decline to conclude when approved sources do not establish the answer.
- Escalate when the request requires professional judgment or exceeds authorized limits.
For example, if a refund depends on the purchase date and that date is unavailable, the system should request it rather than invent an assumption.
Test these behaviors explicitly. A model’s confident tone or self-reported confidence is not independent evidence of correctness. Reviewers also need clear instructions, access to evidence, and enough time to challenge an output; human review alone does not guarantee accuracy.
Before Allowing Actions: Protect the Workflow
☐ Treat external content as untrusted
Documents, emails, websites, and retrieved passages may contain instructions intended to redirect an AI system. A malicious document might tell an assistant to ignore its rules, disclose information, or send data elsewhere.
OWASP’s prompt-injection guidance explains this risk, including indirect attacks delivered through external content. Retrieval-augmented generation can provide relevant context, but it does not eliminate prompt injection or guarantee factual accuracy.
Separate trusted instructions from retrieved material, and test whether hostile content can change the assistant’s behavior. Do not rely on a prompt telling the model to ignore attacks as the sole defense. Limit what the surrounding application permits, even if the model follows a malicious instruction.
☐ Restrict permissions and validate outputs
Give connected tools only the access required for the approved task. An assistant that drafts a customer response does not automatically need permission to send it, issue a payment, or modify an account.
Enforce authorization outside the model. Application code and access controls should determine which users can access records and which actions are permitted.
Before another system consumes generated output, validate it. Check structured data against the expected schema and business rules. Treat generated commands and queries as untrusted input, and prevent them from bypassing application safeguards.
Require explicit approval for high-risk actions. That approval should show the proposed action, affected records, supporting evidence, and likely consequences—not merely ask someone to click “Confirm.” Test attempts to access unauthorized accounts or perform prohibited actions before deployment.
After Launch: Monitor, Retest, and Prepare to Stop
☐ Operate a feedback and shutdown process
Deployment is the beginning of ongoing evaluation. Models change, documents become stale, integrations evolve, and users discover unexpected ways to use a system.
Track factual errors, unsupported claims, reviewer corrections, escalations, complaints, unauthorized-action attempts, and incidents. Protect monitoring records: logs can contain sensitive prompts, retrieved information, and outputs. Limit access and retention accordingly.
Repeat relevant evaluations after changes to models, prompts, retrieval sources, permissions, or tools. Compare results with the approved baseline before expanding use.
Document who can disable the workflow, how to revoke tool access, and how work will continue safely. For a customer service assistant, the fallback might be routing requests to staff using approved templates. Test that process before an incident makes it urgent.
Keep a One-Page Reliability Record
Make the checklist operational with a short record for each AI workflow:
- Owner: Accountable person and escalation contact.
- Allowed task: Intended users, permitted outputs, and prohibited actions.
- Data access: Approved sources, sensitive-data restrictions, and tool permissions.
- Test results: Evaluation date, system version, acceptance criteria, and unresolved failures.
- Required review: What must be checked, by whom, and before which actions.
- Known limitations: Conditions where performance is inadequate or untested.
- Stop procedure: Shutdown authority, technical steps, and fallback workflow.
This worksheet makes assumptions and responsibilities visible. It does not establish that the system is universally safe or reliable beyond the conditions evaluated.
Start With One Workflow
Reliable AI adoption requires more than better prompts. It requires clear boundaries, realistic testing, verified evidence, controlled permissions, and a practical response when something goes wrong.
Choose one AI workflow your organization already uses. Assign its owner, complete the reliability record, and test a consequential failure before expanding deployment. Close the most serious gap first.
The goal is not to eliminate every possible mistake. It is to build a process that detects important failures, limits their consequences, and keeps people accountable for the decisions AI helps them make.