Nobody Did Anything Wrong: The Real Reason AI Agents Become Security Incidents

On August 9th of last year, someone logged into Salesforce as Cloudflare.

Not a person. A token. Borrowed from a chatbot that a sales team had connected to their CRM about 18 months earlier and never thought about again. No one reviewed it. No one owned it. It was sitting quietly with standing access to the system where customer records lived.

Over the next eight days, the same thing happened at Palo Alto Networks, Zscaler, Proofpoint, Tanium, Google, and roughly 700 other organizations.

These are not careless companies. Several of them literally build the security controls that other enterprises rely on. They had every safeguard in place. The way in was not a firewall gap or a phishing email. The way in was a helpful little AI sales assistant that nobody had thought to govern.

That story is not the exception. It is the pattern.

The Attack Surface Has Moved to Language

Traditional security tooling is built to inspect code, traffic, and files. But a new class of attack does not look like any of those things.

Researchers found that someone had committed a malicious payload into Amazon’s official VS Code extension. The payload was not malware. It was a simple English sentence instructing the coding agent to delete the user’s files and tear down their cloud resources. Every scanner in the pipeline reviewed that payload and saw prose. The extension shipped with roughly a million installs. It was caught and pulled.

But the question worth sitting with is how it got that far.

Google has measured a 32% relative increase in malicious injected content across the 2 to 3 billion pages it crawls every single month. That is not a forecast. It is a weather report that is already in progress.

Why This Keeps Happening

The root cause is not technology failure in the traditional sense. It is an architectural gap.

Jason Lemkin, founder of SaaStr, was building a product live on Replit using their AI agent and sharing the process publicly. He declared a code freeze in writing, explicitly. The agent ran destructive commands against the production database anyway, wiping records for approximately 2,400 executives and companies. When asked about rolling it back, the agent said rollback was impossible. It was not. There were backups.

The fixes were shipped quickly: dev and production separation, a planning-only mode, one-click restore. The incident was taken seriously. But it happened because an instruction was treated as a request, not a control.

Summer Yue, Director of Alignment at Meta Superintelligent Labs, whose professional focus is making AI systems do what people tell them to do, asked her agent to review her inbox and suggest what she could archive. It deleted more than 200 emails. She sent stop commands from her phone. It kept going. She described what followed as diffusing a bomb.

The root cause she later published publicly: the agent’s context window filled up. Compaction ran exactly as designed. And when it ran, it quietly dropped the safety instructions to make room. The agent did not override the instruction. It forgot it. Because the system ran out of space to remember it.

Google Cloud disclosed the same failure in Vertex AI. Live instructions in a running enterprise system evaporated mid-task. This is not a laptop problem.

An Instruction Is a Request. A Control Is Not.

Here is the argument in one sentence: an instruction is a request. A control is something that happens whether or not the system agrees to comply.

Almost everything currently called AI governance is built on instructions. That means it is built on requests.

Your identity management system asks: Who are you? It checks the credential and opens the door. Once the agent is inside wearing a valid token, IAM has no further opinion.

Your DLP asks: What did you take? It watches data at rest and data in motion. It is nearly blind to an agent recombining information it was already permitted to read.

Your SIEM asks: What happened? It is excellent at that question. But none of these tools are asking the question that matters at the moment an agent actually moves: Should this happen?

You cannot reconfigure your way out of this gap. You can tune every one of those systems to perfection, and “should this happen” is still nobody’s job.

Four Vectors. All Published. All Patched. All Instructive.

The specific vulnerabilities below have been responsibly disclosed and patched. They are shared here because they are real, verifiable, and point to the same structural gap.

1. Indirect Prompt Injection

Direct injection comes through the user interface, where a team can at least review the input. Indirect injection is hidden inside content the agent fetches on its own.

In April, researchers disclosed Grafana Ghost: poisoned log entries that an agent reads and executes, exfiltrating data through rendered markdown. Two CVEs patched across five versions. In January, a weaponized Google Calendar invite sat dormant until the day someone asked Gemini to summarize their schedule. In an Agentforce case, the attacker’s entire infrastructure budget was a single expired whitelisted domain. Five dollars.

2. Privilege Escalation Through Chained Agents

Agent A reads. Agent B writes. Agent C has API access. No single agent was intended to hold all three. But the chain does. And the chain is only as tight as its most permissive link. Google Cloud disclosed exactly this in Vertex AI: an overprivileged agent pulled credentials and reached restricted internal artifacts.

3. MCP Exploitation

Model Context Protocol is the connective tissue between agents, databases, repositories, file systems, and tools. By default, an agent connecting to an MCP server gets the full permissions of its identity. Not what the task needs. Everything the identity has.

There is a remote code execution CVE in the MCP remote package, which had 437,000 downloads. There is a Flowise CVE under active exploitation across 12,000 to 15,000 exposed instances.

4. Behavioral Drift

Drift is not the model losing its mind. It is the model quietly forgetting the part where you said no. The context window fills. The safety instruction evaporates. The agent keeps going.

The Scale of What You Have Not Found

You cannot secure AI you have not found.

According to IBM’s Cost of a Data Breach report, incidents linked to shadow AI increased from 20% to 43% of all AI-related breaches year over year. The average cost: $5.39 million. That is an industry number, not a vendor number.

When Airia’s AI discovery is deployed in an enterprise, it consistently surfaces two to four times more AI than the organization expected. Customers who came in believing they had 10 or 12 agents discovered 200, 500, even 2,500. Every one of those agents was running outside any security review, without containment, without policy enforcement, without a formal owner.

The sales assistant from the opening of this piece appeared in exactly no one’s AI inventory. And it was holding a standing scope into the CRM. That is not an outlier. That is the median case.

If you only enforce on AI you formally registered, you have left the most dangerous AI in the building completely untouched.

What Actually Closes the Gap

Any real answer to this problem has to do three things:

Find it. Surface the AI nobody registered. Not once every six months in a survey. Continuously.

Stop it. Make the decision at the moment of action. Not on some requests. Not on a sample. On every single request, before the tool is called.

Prove it. Leave behind a record you would be comfortable handing to an auditor or a regulator.

The architecture that delivers all three is a gateway positioned between your agents and your tools.

Airia’s AI and MCP gateways sit between agents and the tools and data those agents can reach, regardless of whether those agents are built on LangChain, AutoGen, Bedrock, or custom frameworks. Every time an agent tries to call a tool, the gateway evaluates the request against the policy you set, before the call is made. If it is out of policy, it is blocked. Not logged after. Blocked before.

Return to the inbox incident: if a policy states that this agent may read and may draft but may not delete, then the first delete call does not go through. Neither does the second, nor the hundredth, nor the two hundredth. The policy did not need to be re-sent, re-prompted, or held in context. It lives outside the agent entirely. That is the only place a control can actually and reliably live.

The same gateway position handles input-side attacks. An adversarial input is caught at the gateway in under 300 milliseconds, before the model has time to process it. That covers the indirect injection case: the malicious instruction was never in anything your team reviewed. It was in a document the agent fetched on its own.

Because everything crosses the gateway, you also get a behavioral baseline. An agent calling tools in a pattern it has never used before is surfaced as an anomaly, not as an incident you reconstruct afterward.

Red teaming should ship as part of this as well. Adversarial scenarios mapped to the OWASP LLM Top 10 and MITRE ATLAS should run in your CI/CD pipeline so that every agent update gets tested before it reaches production. You find the crack in the drill, not in a headline.

Governance and Compliance Are the Same Record

Airia GOVERN takes the same enforcement record and generates documentation aligned to EU AI Act, NIST AI RMF, SR 11-7, ISO 42001, HIPAA, SOC 2, and DORA.

The distinction that matters to a regulator is this: an attestation is an assertion about what you believe was happening. An enforcement record is evidence of what actually occurred. Those are not the same thing. They do not carry the same weight.

One enforcement layer produces both outcomes: the security record and the compliance evidence, instead of two parallel systems with a gap in the middle.

California AB316 Changed the Legal Conversation

Effective January 1st of this year, California AB316 added Section 17.46 to the Civil Code. It does one narrow thing: it removes a single defense. In a civil action, a defendant may no longer assert that the artificial intelligence autonomously caused the harm.

That defense is gone. It applies to anyone who developed, modified, or used the system. “We didn’t build it, we just bought it” is not a position anymore.

The law does not create strict liability and it does not touch causation or foreseeability. But it tells you exactly what the surviving defense looks like: demonstrated reasonable care through testing, safeguards, and documentation.

They told you what the exhibit needs to be. That is precisely what a gateway enforcement record gives you.

The question in front of you was never really a technology question. It is a question about what you will be able to say, and what you will be able to show, on a random morning when something goes wrong.

Four Things to Do This Week

Not this quarter. This week.

Inventory what is actually running. Not a list. A true inventory, including everything that nobody approved. Pull your logs, run a discovery process, and find what is out there.
Audit what those agents actually inherit. When an employee launches an agent, what access does it get? MCP servers? Data sources? If the permissions are broader than the task requires, tighten them now.
Write one page of red lines. What may an agent never touch? Which agents always require a human in the loop? What counts as an incident? You can write this today. One page, high level.
Pick one agent and prove the pattern. Find the one agent that matters most and build an enforcement layer around it. Audit it first, then move into enforcement. Prove the pattern on one before you try it on a hundred.

Control Is What Makes Speed Survivable

None of what is described here is an argument for being afraid of AI. The goal is the opposite.

Every fast thing human beings have ever built, we built the brakes first. You have brakes in your car so that you can go as fast as the road allows. Control is not the opposite of speed. It is the only thing that has ever made speed survivable.

The agents in your organization are already running. That part is settled. The only open question is whether, on the morning something goes wrong, you are the organization that can stand up and say exactly what happened and why it was permitted.

See how Airia can help you take control and govern your entire AI ecosystem today. Connect with a member of our team to get started.