An AI assistant summarizes a suspicious email in seconds. Another reads an invoice and prepares a payment request. Both save time—but the second can create far greater consequences if it follows malicious instructions hidden in a document.
That distinction is central to AI cybersecurity. The question is not simply whether artificial intelligence can improve security. It is whether an organization can capture its benefits while controlling the information it accesses, the decisions it influences, and the actions it takes.
For business leaders and security teams, a sound strategy addresses three priorities: using AI for cyber defense, defending against AI-enabled attacks, and securing AI systems themselves. These areas are also reflected in NIST’s preliminary Cyber AI Profile.
How AI Can Improve Cybersecurity
AI cybersecurity is broader than a chatbot for security analysts. Machine-learning systems can help identify unusual activity and classify threats. Generative AI can summarize evidence, explain technical material, and assist investigations. Neither should be treated as an independent authority.
Start with bounded, evidence-based tasks
Useful initial applications include:
- Alert triage: Group related alerts and summarize the evidence for analyst review.
- Incident documentation: Draft timelines from approved logs and investigation notes.
- Technical analysis: Explain a suspicious script or detection rule without executing it.
- Security communication: Translate technical findings into clear remediation instructions.
For example, an assistant could assemble an account-compromise timeline from identity and email logs. It should identify the underlying records, flag missing evidence, and distinguish observations from interpretations. An analyst—not the model’s confidence—determines whether the account was compromised.
The Australian Cyber Security Centre’s guidance on AI in cyber defense emphasizes verifying outputs and preserving human accountability. AI cannot compensate for missing logs, inaccurate asset inventories, or poorly understood permissions.
Measure security outcomes, not demonstration quality
Before a pilot, establish a baseline. Compare investigation time, summary accuracy, missed findings, unsupported recommendations, review effort, and operating cost. Include ambiguous cases and incomplete records. A tool that produces answers faster but requires substantial correction may not improve the workflow.
How Attackers Use AI—and What Changes for Defenders
Attackers can use AI to make familiar scams more convincing. The FBI warns about personalized phishing and AI-generated voice or video impersonation. Polished writing and a recognizable voice are no longer sufficient reasons to trust a request.
Consider an employee receiving an urgent voice message that appears to come from an executive requesting a bank-account change. The safest response is not to debate whether the voice sounds artificial. It is to follow an established verification process.
- Verify independently: Contact the requester using a previously established number or trusted channel—not contact details supplied in the suspicious message.
- Separate sensitive approvals: Require independent approval for payment changes, privileged access, and consequential credential resets.
- Train for behavior: Focus on urgency, secrecy, and pressure to bypass procedures rather than spelling mistakes.
- Strengthen authentication: Prioritize phishing-resistant multifactor authentication for high-risk accounts.
Deepfake detection can support an investigation, but it should not be the only barrier protecting money, credentials, or sensitive information.
The Main Security Risks in AI Applications
Securing AI requires protecting the entire application: models, data sources, connectors, credentials, tools, and infrastructure. Joint government guidance on secure AI deployment combines conventional security controls with AI-specific protections.
1. Prompt injection
Prompt injection attempts to redirect a model through malicious instructions. An indirect injection arrives through material the AI reads, such as a webpage, email, or document.
Imagine a support assistant retrieving a document that instructs it to send customer records to an external address. The document is evidence to process—not an authority to obey. Retrieval-augmented generation, which supplies relevant documents to a model, does not eliminate this risk.
Practical controls: Treat external content as untrusted, separate it from application instructions, limit available tools, and test malicious-document scenarios. As OWASP’s prompt-injection guidance explains, mitigation requires multiple controls; a system prompt is not a reliable security boundary.
2. Sensitive-data exposure
An AI application may reveal information through submitted prompts, retrieved documents, generated answers, or operational logs. Broad access can turn a useful assistant into an unintended route to confidential material.
Practical controls: Minimize submitted data, enforce access permissions before retrieval, and review provider terms covering retention, training use, deletion, and incident responsibilities. The NIST Generative AI Profile provides a governance foundation. An “enterprise” product label does not replace reviewing the actual configuration and contract.
3. Excessive agency
Risk increases when an assistant can send messages, execute commands, modify records, or change permissions. A mistaken answer becomes more dangerous when it triggers an operational action.
Practical controls: Begin with read-only access, narrowly scope tool credentials, and require explicit approval for high-impact actions. Enforce authorization in the application or downstream service—not through the model’s interpretation of policy. OWASP’s excessive-agency guidance explains why functionality, permissions, and autonomy all matter.
4. Unsafe output handling
Generated text is not trusted code. Passing it directly into a browser, database, or command interpreter can introduce vulnerabilities.
Practical controls: Validate structured outputs, use parameterized database queries, apply context-appropriate output encoding, and prevent unrestricted command execution. Human review supports these controls; it does not replace secure engineering. See OWASP’s improper-output-handling guidance.
5. Poisoning and supply-chain compromise
Attackers can target datasets, models, dependencies, and other AI lifecycle components. Unverified artifacts can compromise an application before a user submits a prompt.
Practical controls: Use approved sources, verify provenance and integrity, version datasets and model artifacts, inspect unfamiliar components in isolation, and retain known-good releases. NIST’s adversarial-machine-learning guidance describes poisoning and other attacks against AI systems.
A Practical AI Security Adoption Plan
Step 1: Inventory use and assign ownership
Identify approved applications, embedded AI features, employee-selected tools, models, and integrations. Give each deployment an accountable owner and documented purpose. Establish acceptable-use rules that explain what information employees may submit and how exceptions are approved.
Step 2: Map data access and action boundaries
Document what enters the system, what it retrieves, which identities it uses, and which actions it can initiate. Ask what happens if an answer is wrong or an external document is malicious. Apply existing security controls—including patching, secrets management, and access reviews—alongside AI-specific defenses.
A model’s willingness to refuse a request is not a substitute for an application’s ability to prevent it.
Step 3: Test boundaries before expanding access
Test legitimate tasks, unauthorized-data requests, malicious documents, and prohibited tool actions. Confirm that one user cannot retrieve another user’s restricted information. Success means the application’s controls hold, not merely that the model gives a reassuring response. Repeat testing when models, permissions, connectors, or data sources change.
Step 4: Monitor and prepare for failure
Record relevant retrieval and tool activity while protecting sensitive log content. Set usage limits to constrain runaway requests and unexpected costs. Establish a procedure to disable the capability, revoke credentials, preserve evidence, restore a known-good configuration, and continue through a manual workflow.
Understand the Tradeoffs
More data can improve context but increase exposure. More automation can reduce repetitive work but amplify mistakes. More logging supports investigations but creates additional sensitive records. Self-hosting offers deployment control while transferring maintenance and security responsibilities to your team.
Match controls to consequences. A drafting assistant and an agent with production-administrator access should not receive the same risk treatment. Use NIST’s AI Risk Management Framework to structure governance, and evaluate systems against realistic business needs—not pressure to automate.
AI Cybersecurity Checklist
Review these items before launch and after significant changes:
- Every AI system has an accountable owner and defined purpose.
- Tools, models, data sources, and integrations are inventoried.
- Sensitive-data uses and provider terms have been reviewed.
- Human accounts have strong authentication.
- Tool identities use narrowly scoped permissions.
- Retrieved content is treated as untrusted input.
- Access authorization is enforced outside the model.
- High-impact actions require explicit approval.
- Outputs are validated before downstream use or execution.
- Adversarial tests cover data access and tool actions.
- Performance, security events, costs, and errors are monitored.
- Shutdown, credential revocation, and fallback procedures are tested.
Build Trust Through Controls, Not Confidence
AI can strengthen cybersecurity, but confident answers and impressive demonstrations are not proof of safety. Dependable outcomes require reliable evidence, limited permissions, verified outputs, and accountable people.
Start with one bounded use case. Assign an owner, establish a baseline, map its access, test its failure modes, and rehearse recovery. Expand only when both the business value and the surrounding security controls have proved dependable.