An AI assistant retrieves a confidential document and includes its contents in a customer-facing answer. An autonomous agent starts making unauthorized changes in a connected application. A support chatbot produces misleading guidance that customers act on. Each situation demands a response—but not necessarily the same response.
The challenge is not simply determining whether the model “went wrong.” It is understanding what happened across the entire workflow, stopping further harm, and deciding when the service is safe to restore.
You do not need a disconnected incident-response program for AI. You need to extend your existing program to account for model behavior, retrieved content, third-party services, and systems that can take action. NIST’s incident-response guidance places response within broader cybersecurity risk management. AI belongs inside that operating discipline, not outside it.
Use the following checklist to make that extension practical. It is an operational synthesis of the cited guidance, not a verbatim government checklist.
1. Prepare Before the First Alert
The worst time to discover that nobody can disable an AI agent’s purchasing permissions is after it starts placing unauthorized orders. Preparation must establish both decision-making authority and working technical controls.
- Name the response team. Identify the incident commander, application owner, security lead, privacy/legal contact, business owner, and provider support contact. Assign backups and document how to reach them.
- Map the workflow. Record the model and provider, retrieval sources, connected applications, tool permissions, service accounts, and external dependencies.
- Define stop conditions. Specify which behaviors trigger escalation, restricted access, or shutdown—and who can authorize each action.
- Prepare a fallback. Document how essential work will continue through a manual process or another approved service.
- Rehearse provider coordination. Confirm support channels, available evidence, responsibilities, and notification arrangements before an incident.
NIST’s Generative AI Profile highlights ownership, third-party response planning, and deactivation criteria. Translate those principles into something testable: can the on-call team actually stop the affected workflow without waiting for an unavailable administrator?
2. Recognize the Incident and Assess the Harm
Not every incorrect answer proves a cyberattack. A failure may involve unreliable output, configuration errors, unauthorized access, malicious manipulation, or a combination. Classify the behavior without prematurely declaring its cause.
- Record the observation. Describe what the system produced or did, who noticed it, and when it occurred.
- Separate facts from hypotheses. Label confirmed findings, suspected causes, unanswered questions, and confidence levels.
- Assess consequences. Identify affected people, data sensitivity, business context, and actions already completed.
- Assign severity and escalate. Use potential and observed harm, not merely the volume of problematic responses.
Consider a customer-service assistant that displays another customer’s contract. The initial fact is that restricted content appeared in an answer. Whether the cause was excessive retrieval permissions, an application defect, or prompt manipulation remains an investigative question. Containment should not wait for that distinction.
Microsoft’s AI incident-response guidance emphasizes context-dependent severity. A single unauthorized financial action can warrant greater urgency than numerous low-impact formatting errors.
3. Contain Active Harm Without Waiting for Certainty
Contain the capability causing harm—not just the interface where someone noticed it.
Closing a chat session may leave background jobs, tool credentials, and scheduled agent activity running. Containment must follow the system’s actual permissions and execution paths.
- Pause the affected workflow. Disable the relevant feature, agent, deployment, or integration at the narrowest effective boundary.
- Restrict access. Revoke or suspend tool permissions, limit network connections, and rotate credentials when exposure or misuse warrants it.
- Check pending actions. Identify queued jobs, retries, and downstream operations that could continue after the front end is disabled.
- Activate the fallback. Route affected work to approved human review or another established process.
The UK’s NCSC stresses the ability to halt autonomous agent activity immediately. Test that capability in advance; a documented shutdown procedure is not enough if it cannot interrupt execution.
Preserve evidence alongside containment where feasible, but do not allow evidence collection to prolong active harm. Record what you changed, when, and why.
4. Preserve an AI-Specific Evidence Package
Traditional logs remain essential, but they may not explain an AI failure on their own. Investigators also need the interaction context: what instructions the application supplied, what content it retrieved, and what actions it requested.
- Capture identifiers and configuration. Preserve the incident ID, timestamps with time zone, application, provider, model identifier or version where available, and relevant deployment settings.
- Preserve the interaction. Collect available prompts, application instructions, responses, retrieved material, conversation context, and tool or API requests and results.
- Map access and effects. Identify the users, records, connected systems, service accounts, and external services involved.
- Protect evidence integrity. Retain original records, document collection and access, and preserve relevant versions before routine changes overwrite them.
- Apply privacy safeguards. Restrict access, use approved storage, and follow applicable retention requirements.
The CISA AI incident information checklist identifies model details, attack artifacts, timestamps, affected systems, and impact as useful reporting information.
Do not create a second exposure while investigating the first. Sensitive prompts and outputs should not be copied into unrestricted tickets or general-purpose chat channels. Record evidence gaps explicitly rather than treating missing logs as proof that nothing happened.
5. Investigate the Workflow and Repair the Weakness
“The model did it” is not a root-cause analysis. AI applications combine models, instructions, identity controls, retrieval pipelines, and tools. The failure may sit at the boundary between those components.
- Trace the path. Reconstruct how the request moved through authentication, retrieval, model processing, tool selection, and execution.
- Establish scope. Search for related activity across relevant users, sessions, deployments, and time periods.
- Verify downstream consequences. Determine what information was disclosed and whether records, transactions, messages, or other systems changed.
- Fix the contributing weakness. Address authorization, excessive permissions, unsafe tool execution, retrieval controls, or other failures supported by the evidence.
In the contract-disclosure example, blocking the triggering phrase might suppress one symptom. If retrieval still permits cross-customer access, the underlying exposure remains. A durable repair would enforce appropriate document access before content reaches the model and test that boundary independently.
Similarly, an agent should not gain transaction authority merely because its generated text requests it. Sensitive actions need enforceable application-side permissions and, where appropriate, human approval. Prompt instructions can support those controls; they should not substitute for them.
6. Validate Recovery, Communicate, and Improve
Prove readiness before restoring autonomy
One successful replay does not establish that a probabilistic system is safe. Recovery testing must cover related failure paths and legitimate work.
- Test variants. Include rephrased requests, different user permissions, altered retrieved content, and relevant tool combinations.
- Test normal operations. Confirm that remediation has not broken essential tasks or encouraged unsafe workarounds.
- Restore in stages. Begin with limited access or reduced capabilities, monitor closely, and keep rollback available.
- Require authorization. Have the designated recovery authority review the evidence, remaining risks, and monitoring plan before broader restoration.
Keep communication useful and factual
Communication starts during response, not after recovery. Tell appropriate stakeholders what is confirmed, what remains uncertain, what containment is in place, and when to expect the next update. Avoid declaring that “no data was exposed” before the investigation supports that conclusion.
Legal and privacy staff should assess contractual and legal notification obligations. Voluntary information sharing under CISA’s AI playbook is separate from legally required reporting. Neither should be assumed to satisfy the other.
Close with an after-action review. Assign an owner and deadline to each improvement, update the playbook, and rehearse the revised process. Include business disruption and the effectiveness of the fallback—not just technical findings.
Make Response Readiness a Condition of Expansion
AI incident response is ultimately about operational control. Your organization must be able to stop unsafe behavior, preserve trustworthy evidence, determine consequences, and justify restoration.
Before expanding an AI system’s autonomy, ask four questions: Who can stop it? What evidence will remain? How will we determine the impact? Who authorizes its return?
Bring the application owner, security team, operations lead, and privacy/legal representative together for a tabletop exercise. Walk through a disclosure or unauthorized-action scenario using this checklist. Any answer that depends on an undocumented assumption becomes a task to complete before the system receives more authority.