Current as of October 6, 2026.
Consider a payment request that arrives from a familiar account, references a real transaction, and matches the sender’s writing style. The message is convincing because an attacker has access to the conversation—and AI can help turn that access into targeted fraud. Better grammar is not the central threat. Better use of stolen context is.
At the same time, defenders can use AI to investigate suspicious activity, identify software weaknesses, and reduce repetitive work. The technology creates opportunity on both sides while introducing systems that need protection themselves.
The question is not whether AI favors attackers or defenders in the abstract. It is who can deploy it effectively, what authority it receives, and whether controls keep pace. For organizations and individuals, preparation starts with understanding that asymmetry.
How AI Is Changing Cyberattacks
Familiar attacks become faster and more targeted
Microsoft’s 2026 Digital Defense Report summary describes threat actors using AI for reconnaissance, social engineering, malware and exploit development, and post-compromise activity. Much of the observed use supports particular stages of existing workflows rather than replacing an entire attack with autonomous execution.
That distinction matters. Organizations should not become so focused on hypothetical AI superattacks that they neglect compromised accounts, exposed systems, or abused trusted access. AI can accelerate those familiar routes without changing their underlying mechanics.
Compromised information becomes more useful
In September 2026, Microsoft reported disrupting EvilTokens, a cybercrime service whose AI tools analyzed compromised inboxes, identified trusted relationships and payment conversations, and helped prepare impersonation-based fraud. This is a vendor’s account of a specific investigation—not a measurement of all AI-enabled crime.
The practical implication extends beyond email security. A finance team needs an independent process for verifying bank-account changes even when a request comes from a legitimate account. Authentic access does not guarantee an authentic instruction.
Vulnerability management faces greater pressure
The UK National Cyber Security Centre’s assessment through 2027 anticipates that AI-assisted vulnerability research and exploitation will increase pressure to remediate weaknesses quickly. This is a forecast, not a guaranteed outcome. Nevertheless, accurate inventories, visibility into internet-facing systems, and timely mitigation remain sensible priorities.
Where AI Can Strengthen Defense
AI is most useful when it helps security teams work with evidence. It becomes dangerous when its output is mistaken for evidence.
NIST’s Cyber AI Profile, still an initial preliminary draft, organizes the challenge around securing AI, conducting AI-enabled defense, and thwarting AI-enabled attacks. Its proposed defensive applications include incident triage and containment recommendations with human review.
Useful starting points include:
- Summarizing an incident while linking conclusions to original records.
- Suggesting investigation steps for an analyst to evaluate.
- Flagging possible code weaknesses for developer review.
- Drafting containment options without executing them.
Google’s Chrome Security Team has described AI-assisted vulnerability discovery, triage, and patching in its development workflow. That demonstrates a real application, not a guarantee that every team will achieve similar results.
NIST’s Generative AI Profile warns about confidently incorrect output. A persuasive incident summary still needs validation against logs, code, configuration, or other independently verifiable material.
The Asymmetry: Equal Access Does Not Mean Equal Advantage
The following is a strategic interpretation of the evidence, not a quantified prediction.
Attackers can adopt assistance faster than defenders can operationalize it
An attacker may use a model to improve one narrow task without building an enterprise-grade system. A defender must consider data confidentiality, integrations, permissions, reliability, auditability, and accountability. Access to the same technology therefore does not imply equal implementation burdens.
Defenders have advantages—but must activate them
Organizations control their own identities, systems, telemetry, and business processes. Those assets can give defensive AI valuable context. But incomplete inventories, fragmented logs, and unclear ownership reduce that advantage.
Smaller organizations may benefit from managed services rather than building custom AI workflows. The tradeoff is dependence on provider controls, visibility, and incident-response support—not an automatic improvement in security.
The gap between prepared and unprepared organizations can widen
The NCSC anticipates a divide between systems that keep pace with AI-enabled threats and those that become more vulnerable. The advantage is not permanently assigned to attackers. It depends partly on whether defenders improve fundamentals while adopting useful automation.
AI adoption is not the outcome. Faster, better-validated security decisions are the outcome.
AI Creates an Attack Surface of Its Own
An AI application includes more than a model. Its exposure includes connected documents, tools, identities, permissions, infrastructure, and external services.
Prompt injection: untrusted content becomes an instruction
Imagine an assistant retrieving a document that contains malicious directions to disclose confidential information. The risk is that the model treats those directions as instructions rather than untrusted document content.
OWASP’s prompt-injection guidance explains that retrieval-augmented generation and fine-tuning do not fully mitigate this problem. A stronger system prompt is not a complete security boundary. Authorization must also be enforced by application code and downstream systems.
Excessive agency: a mistake becomes an action
A recommendation-only assistant can produce a bad suggestion. An agent with broad permissions might also delete records, publish information, or change access. OWASP recommends limiting functionality, permissions, and autonomy, with approval for consequential actions.
Data integrity: unreliable inputs undermine outputs
Manipulated source material can influence an AI system’s behavior. Joint NSA-led guidance emphasizes data provenance, authenticated revisions, and lifecycle protection. Organizations need to know where important AI inputs originate and who can change them.
A Practical Plan for Organizations
1. Strengthen identity and business verification
Prioritize phishing-resistant authentication for sensitive access. CISA identifies FIDO/WebAuthn and enterprise PKI-based options, while acknowledging legacy-system constraints. Plan recovery and fallback procedures alongside deployment.
For compromised accounts, follow platform-specific response instructions, including session and token revocation where appropriate. A password reset alone may not terminate existing access. Independently verify payment changes through a trusted second channel.
2. Inventory AI systems and their connections
Record each application’s owner, data sources, integrations, permissions, and permitted actions. Include internally developed agents and employee-adopted services. CISA’s agentic-AI guidance emphasizes layered defenses, identity management, monitoring, and limits on autonomy.
Ask vendors about retention, training use, administrative logs, incident notification, and access revocation. Do not assume enterprise controls exist merely because a product includes an AI feature.
3. Start with bounded assistance
Choose a narrow, read-only pilot. For example, let an assistant draft an investigation summary, but require an analyst to verify its claims before approving the record. Read-only access still creates confidentiality risk, so restrict the information available to the task.
If execution authority is later justified, use narrowly scoped tools and identities. Require meaningful approval outside the model for high-impact actions, and establish a tested way to disable the agent.
4. Measure total performance before expanding
Compare the pilot with the existing workflow. Measure time to a validated conclusion, review effort, incorrect recommendations, missed findings, unauthorized-action attempts, and total operating cost. Define stop conditions in advance.
The tradeoff is straightforward: automation may reduce drafting time while increasing verification work. A faster response is not an improvement if it is less reliable.
What Individuals Should Do
Do not rely on spotting an artificial voice or an unusually polished message. The FBI’s impersonation warning notes that generated content can be difficult to identify.
For urgent requests involving money or account access, stop and contact the person using a number or channel you already trust—not details supplied in the message. Never share authentication codes. Establish a family verification phrase, use phishing-resistant authentication where available, and report suspected fraud through the FBI’s Internet Crime Complaint Center.
An AI-Security Readiness Checklist
Use this as a planning aid, not a certification:
- Ownership: Every AI application and agent has an accountable owner.
- Access: Data, tool, and execution permissions are documented and minimized.
- Verification: Important recommendations are checked against original evidence.
- Testing: Evaluations include prompt injection and unauthorized-action scenarios.
- Authorization: High-impact actions require controls outside the model.
- Monitoring: Activity is auditable, and access can be revoked promptly.
- Recovery: Teams can disable a compromised agent and preserve relevant records.
- People: Staff independently verify payment changes and unusual requests.
- Value: Success is measured through validated outcomes and total cost.
The Advantage Goes to Disciplined Deployment
AI can accelerate both useful security work and criminal activity. It also introduces risks through the data and authority connected to it. None of those realities makes maximum autonomy the right objective.
Start with one bounded defensive use case, one clear owner, and measurable acceptance criteria. In parallel, strengthen identity, patching, verification, and recovery. Expand only when evidence supports doing so. The organizations best positioned to benefit will not simply use more AI—they will control it more deliberately.