The recent OpenAI-Hugging Face incident, where hundreds ofOpenAI’s autonomous AI agents hackedthrough container constraints for machine learning platform Hugging Face’s external infrastructure, is a notable example of agentic risk.
But even more mundane situations, like when an agent incorrectly interprets instructions and acts based on that mistake, can cause problems.
When incidents like these occur, finding the problem and disabling the agent are clearly important steps in mitigating the harm. But the work doesn’t end there.
In traditional incident response, the focus is broadly on identifying what was compromised, containing the harm, and restoring the affected systems. In incident response involving AI agents, it’s also necessary to reconstruct, and potentially reverse, a chain of actions and decisions made by the agent that may otherwise be legitimate.
AI agents can act in ways they’re authorized to, with no hacking involved, and still act inappropriately. An appropriate incident response must consider this.
Investigating the trail of AI agent incidents
There are several reasons why traditional incident response processes might not translate neatly into scenarios with AI agents in the mix.
For example, AI agents may be acting with legitimate credentials and using those credentials to take multiple permissible actions – often more than a single employee would likely be responsible for – while still acting incorrectly based on factors like poor instructions or data. Agents might make intermediate or phased decisions based on contexts that are changing, which can compromise the actions to come.
AI agents also often interact with multiple business units. For example, they might call APIs or use multiple tools. They can also hand work off to another agent, or even a human. And they might change or generate data that can trigger conventional or AI-enabled automations, leading to downstream effects separate from the agent’s direct responsibilities.
Considering all of this, it’s important to understand that when AI agents are involved, the incident trail is both technical and decisional.
Incident investigators need to determine not just what happened but also which information the agent had and when, the instruction or goal the agent pursued, the tool or API calls the agent made, and why one agent action led to others that followed.
The ability to reconstruct the trail depends on decisions made before the incident. Pavel Bantsevich, head of partnerships and project manager at Pynest, recommends giving each AI agent a scoped identity, then requiring it to operate through interfaces logging its actions.
Similarly, Srinivas Chippagiri, a senior member of the technical staff at Salesforce, argues against allowing agents to act through shared service accounts, because legitimate actions can then become difficult to attribute to the correct agent.
“When an agent uses shared or human credentials, your logs say a person did it, and the trail is gone,” Bantsevich said.
At Pynest, agent actions like pull requests, tool calls, and comments are tied to the identity of the responsible agent, making it possible to replay an event sequence, he said. Then, individual actions can be connected to the inputs that prompted them.
Priority one: preserve the agent’s context
During an incident involving AI agents, some of the early response steps are straightforward: stop the agent, revoke the agent’s credentials and permissions, and roll back the affected systems. These actions are often necessary to contain the agent and stop its unwanted actions.
But in a truly robust response, preserving enough evidence to reconstruct the AI agent’s behavior should be an early priority.
To identify the context surrounding the agent’s unwanted actions, an incident snapshot should include the agent’s identity, model and version. It’s also important to identify the system instructions involved, along with the user or delegated instructions.
Any available context should be preserved, including the data retrieved by the AI agent, its outputs, and the subsequent actions – including all the relevant timestamps. Finally, the snapshot should include the permissions given to the agent and any tool or API calls made.
It’s especially important to preserve not only a record of the resulting action, but also the information available to an AI agent when a decision was made, Bantsevich said. “Most teams only keep the outputs, and without the state the agent decided on, you cannot tell whether the agent was wrong or the data was.”
That information should also persist after the interaction ends. Treat agent decision logs as forensic evidence rather than relying on information that only exists temporarily in a model’s context window, Chippagiri recommends. Along with what happened, those records should preserve the agent’s goal or prompt, tools and other agents called, intermediate decisions, and outputs acted upon.
“If that reasoning trail is ephemeral or lives only in the model provider’s context window, you cannot reconstruct the chain after the fact,” he said.
In the process of putting together a response involving AI agents, organizations might discover they don’t actually collect or preserve some of these records. In these cases, procedures for collecting all the information needed to build out the context of an incident should be developed to ensure future incidents don’t run into the same issue.
Next, reconstruct the chain of events
Once an incident’s evidence and context are preserved, the next step is to reconstruct the causal chain. This goes beyond compiling a log of events and requires asking some questions about why things happened and what triggered it.
For example, how do you determine an agent’s root action from its downstream effects, or trace a workflow across agents from different vendors? If Agent B acts incorrectly based on bad information from Agent A, what happens and how can it be prevented? And which information is needed to reconstruct the sequence of events?
Correlation IDs that persist across agents and tools can help with that reconstruction process. Without them, investigators can be left trying to reconcile several separate logs rather than following one transaction or workflow across the involved systems. When correlation IDs are used, activity involving multiple agents and tools can be assembled into a single incident account, Bantsevich said.
Rolling things back and returning the agent to service
Once incident investigators are clear on what happened, the next step is to decide which of the agent’s actions should be reversed, and how.
Some actions may be easily reversed. For example, scheduled jobs can be cancelled or edited, permissions can be revoked or reinstated, and database changes can be reversed.
For others, reversing them could be a bad option if later work that relied on those actions was valuable. And other actions still could be impossible to reverse, or too onerous to make it worthwhile.
Emails can’t be unsent once read, for example. Employee actions that relied on information from an agent may have already been taken, and information already disclosed can’t be unlearned.
Ideally, organizations should determine how reversible an AI agent’s actions are before giving that agent the authority to take those actions. Deterministic actions involving structured inputs can often be replayed and reversed, Bantsevich says – and free-form actions that produce external effects might not be. Also, workflows that require human approval can provide a defined rollback point if something goes awry.
“Reversibility is a design decision. You cannot improvise it mid-incident,” Bantsevich said.
In those situations where actions are irreversible, or where reversing them is a bad option, a business rollback could be a better approach than a technical rollback.
To determine the best course of action, consider classifying AI agent actions according to whether or not it is possible or desirable to reverse them when they are incorrect, with consideration for what is required to mitigate or roll back the impact of harmful agent-led actions in each case.
Finally, when the AI agent has been halted, and the problems resulting from its actions have been identified and, where possible, fixed, incident responders must determine if and how it should resume operations.
Simply modifying the agent and observing that it now works correctly isn’t enough. A common mistake is returning an agent to production without first reproducing the failure that caused the incident, Chippagiri says. “If you cannot reproduce it in a contained environment, you have not understood it, you have only paused it.”
Similarly, Bantsevich recommends identifying the step that failed, fixing it, and rerunning the failing case through an auditor and the full test suite. Investigators should also consider if permissions, prompts, and other aspects of the agent should change, and if other agents are similarly vulnerable.
If an AI agent returns to service, it can first operate with reduced autonomy or more human oversight in order to confirm the problem is solved and catch any new issues as quickly as possible.
Design agents for both operation and investigation
As organizations incorporate a growing number of AI agents into their operations, they are understandably focusing on deciding what agents should be allowed to do. But effective incident response also depends on decisions made before an agent’s deployment, including how the agent’s actions are attributed, which context is preserved, and which actions are reversible if necessary.
To build that kind of forensic readiness into AI agent environments, their architecture must answer two questions: what is the agent allowed to do, and can we reconstruct what it did?
