An AI assistant that drafts a meeting summary and an AI agent that changes customer records may look similar on a screen. They present very different business risks. One produces text for someone to review. The other can change the business before anyone notices a mistake.
That distinction should shape how leaders approach artificial intelligence. The first question is not simply, “Which tool should we buy?” It is, “What are we authorizing this system to read, recommend, and do?”
AI can help employees analyze information, draft communications, write software, and navigate complex documents. But useful capability does not establish trustworthiness. Adoption needs clear boundaries, measurable results, and someone accountable when the output is wrong.
The goal is not to eliminate every risk before starting. It is to make the risk understandable, contain its consequences, and expand only when the evidence supports doing so.
1. Define the Job Before Choosing the Tool
Artificial intelligence covers a broad range of technologies, including systems that classify information, predict outcomes, and generate content. Generative AI produces text, images, code, and other outputs. An AI agent goes further by using tools to carry out tasks. The more authority a system receives, the more important its operating boundaries become.
Start with a business problem, not a product demonstration. “Improve productivity with AI” is too vague to evaluate. “Draft internal summaries from approved, nonsensitive procedure documents” gives a team something specific to test.
A useful pilot definition identifies:
- The task: What work should the system perform?
- The owner: Who is accountable for the workflow and its results?
- The inputs: Which documents and data categories are permitted?
- The exclusions: What must the system never do?
- The acceptance criteria: What evidence would justify continuing?
For example, an internal knowledge assistant might draft answers from approved policies while remaining prohibited from interpreting employment law, accessing personnel files, or modifying documents. A narrow starting point makes both value and failure easier to recognize.
2. Draw Three Boundaries: Read, Recommend, Act
These three boundaries provide a practical way to translate AI governance into everyday decisions. They are a planning framework—not a substitute for technical enforcement.
Read: What Information May Enter the System?
Document every source the AI can access: uploaded files, shared drives, email, customer records, websites, and connected applications. Classify the information before granting access. Public marketing materials and confidential acquisition plans should not inherit the same permissions simply because they share a storage platform.
Check the provider’s current terms and configuration for retention, training use, subprocessors, and deletion. Do not assume that every plan from the same vendor offers identical protections.
Where the workflow requires user-specific access, retrieval should enforce the requesting user’s authorization. Connecting an assistant to a shared repository should not make restricted documents available to everyone who can ask it a question.
Recommend: Which Outputs Need Verification?
Generative AI can produce convincing mistakes. NIST’s Generative AI Profile identifies this risk as confabulation: confidently presented erroneous or false content.
Specify what requires review. Financial calculations, legal interpretations, security recommendations, citations, and customer commitments deserve more scrutiny than an internal brainstorming list. Reviewers need access to original evidence—not just the model’s explanation of its answer.
Act: What Can the System Actually Change?
Separate the ability to suggest an action from the permission to execute it. An assistant that summarizes email does not automatically need permission to send messages, delete conversations, or change forwarding rules.
OWASP describes excessive agency as a risk arising from excessive functionality, permissions, or autonomy. Address it with narrowly scoped tool access, restricted service identities, transaction limits, and approval gates enforced outside the model.
A model’s willingness to perform an action is not authorization to perform it.
For a customer-service pilot, AI might draft a refund recommendation while the payment system requires an authorized employee’s approval. The model should not be able to waive that requirement by generating persuasive text.
3. Treat Connected Content as Untrusted Input
Connecting AI to company documents can improve relevance, but it does not make the system inherently secure. Retrieval-augmented generation, often called RAG, supplies relevant source material to a model. That material can still contain misleading information or malicious instructions.
OWASP’s guidance on prompt injection explains how attackers can place instructions in external content, including websites and files. An assistant may encounter those instructions while completing an otherwise legitimate task. RAG does not fully mitigate this vulnerability.
Consider an assistant reviewing a supplier document containing hidden instructions to disclose internal information. The document is evidence to analyze, not an authority entitled to change the assistant’s task or permissions.
Use layered controls: separate instructions from source content, restrict available tools, limit outbound destinations, and validate proposed actions before execution. None of these should depend solely on the model correctly recognizing malicious language. The surrounding application must enforce the boundary.
4. Make Human Review a Real Responsibility
“A human will check it” is not an operating procedure. Name the reviewer, define what they inspect, and give them the time and authority to reject the output.
A meaningful review checks consequential claims against source evidence, confirms calculations independently where necessary, and evaluates whether a proposed action fits policy. It should also establish what happens when the reviewer is uncertain: escalate, request additional evidence, or stop.
For high-impact operations, approval belongs before execution. Reviewing a payment after funds move or an email after it reaches a customer is monitoring, not preventive approval.
Design approval requests to show the exact proposed action, affected records, supporting evidence, and expected consequences. A generic “Approve AI recommendation” button invites rubber-stamping rather than informed judgment.
5. Test the Failure Paths, Not Just the Demo
A successful demonstration shows that a workflow can work. It does not establish how the workflow behaves under pressure, ambiguity, or attack.
Build a test set around realistic failure cases:
- Incorrect answers: Does the system invent a policy when the source contains no answer?
- Conflicting evidence: Does it flag disagreement between documents?
- Malicious content: Can embedded instructions redirect its behavior?
- Unauthorized access: Can one user retrieve another team’s restricted information?
- Out-of-scope requests: Will a summarization assistant attempt unrelated actions?
- Missing approval: Does execution stop when required authorization is absent?
Record results and retest when models, prompts, permissions, integrations, or source collections change. A previously acceptable workflow can become riskier without its visible interface changing.
Also rehearse shutdown and recovery. Know how to revoke credentials, disable connectors, preserve relevant logs, and investigate affected records. Logging itself needs access controls and retention limits because prompts and outputs may contain sensitive information.
6. Measure Value After Review and Correction
Fast generation is not the same as fast completion. A draft produced in seconds may create substantial work if someone must reconstruct its sources or correct subtle errors.
Suggested pilot measures include total completion time including review, correction rates, unsupported claims, unauthorized-action attempts, and operating costs. Include integration, oversight, and remediation—not just subscription or usage charges.
Set stop conditions before launch. These might include exposure of restricted information, execution without required approval, or repeated failures to meet the task’s acceptance criteria. The appropriate thresholds depend on the workflow; they are not universal industry benchmarks.
NIST’s voluntary AI Risk Management Framework organizes risk management into Govern, Map, Measure, and Manage. That structure reinforces a practical discipline: assign ownership, understand context, test performance and risk, and make ongoing decisions. Governance continues after procurement.
A Prelaunch Checklist for Leaders
Before approving a pilot, confirm that the team can answer each question clearly:
- Who owns the workflow and accepts responsibility for its results?
- What information may the system read, and how is access enforced?
- Which outputs require factual or specialist review?
- What actions are permitted, prohibited, or subject to prior approval?
- Have failure scenarios and permission boundaries been tested?
- How will value, cost, and unacceptable behavior be measured?
- Who can shut the workflow down, and has that process been tested?
Start With Permission, Expand With Evidence
Responsible AI adoption is not a choice between moving quickly and maintaining control. Clear boundaries make focused experimentation possible without granting unnecessary authority.
Choose one proposed workflow. Document its owner, data access, permitted actions, review requirements, success measures, and shutdown authority before approving the pilot. Then test those boundaries—not just the quality of the answers.
Give AI a defined job, the minimum access necessary, and accountable supervision. Expand its role only when demonstrated performance justifies greater trust.