A friend of mine works in compliance at a mid-sized bank. Last quarter, she got pulled into an incident that I keep thinking about, though not for the reasons she does.
An automated underwriting system had denied a loan application. The applicant complained. The complaint moved through the usual channels and eventually landed on someone’s desk with the question every regulator eventually asks in these situations: who made this decision.
My friend went to answer the question, which she had answered a thousand times before, and realized she couldn’t. The vendor had shipped a decision engine that pulled in a scoring model, a rules layer, a risk overlay, and a fraud filter, and each of those had been touched by three teams and two other vendors, and the chain of “who signed off on this specific outcome” had, somewhere in the last eighteen months, evaporated.
Everyone was still technically responsible. Which meant, in the practical sense that matters when a regulator is asking, nobody was.
I’ve been thinking about this incident for weeks because it points at something the AI industry has not addressed and is barely starting to notice. We got extraordinarily good at automating decisions. Cheap models, fast inference, easy tool integration, an entire ecosystem for shipping autonomous behavior.
What we didn’t automate, in the same period, is any of the accountability infrastructure that made human decisions defensible. Those two clocks are running at wildly different speeds now, and the gap between them is where a lot of the next few years of pain is going to live.
What we actually automated.
Take a minute and think about what the phrase “automated decision” now covers.
An LLM-based agent that decides whether to refund a customer. A recommendation system that decides what a user sees. An underwriting model that decides who gets credit. A code review bot that decides whether a pull request merits a human look. A hiring screener that decides which resumes advance. A pricing algorithm that decides what to charge you today for the same seat you would have paid less for yesterday. A moderation model that decides your post violated the rules. A vendor risk score that decides whether your company gets onboarded as a supplier.
Each of these was, ten years ago, the output of a human. Sometimes, many humans, sometimes one, but there was always a name attached. If the decision was wrong, you could go find that person, ask them why, and hold them to some standard. The standard might have been generous or harsh, but it existed. There was a face on the other end.
None of that infrastructure automated the decisions. The models moved to production. The names came off the outputs. And the systems we use to hold decisions accountable, from performance reviews to disciplinary processes to litigation to regulatory audits, still assume that when something goes wrong there is a person to point at.
Which brings us to the interesting question. What actually happens when there isn’t?
The Four Unsatisfying Answers
If you ask around, you’ll find four approaches to answering the accountability question for automated decisions. None of them work well, and understanding why is where the real problem lives.
The first answer is “the model is accountable.” This is a rhetorical move that lawyers, regulators, and philosophers all agree is nonsense. A model is a mathematical object. It cannot be sued, disciplined, or reasoned with. Nobody who has actually had to defend a decision in front of a regulator has ever tried this answer, because it does not survive contact with the question.
The second answer is “the vendor is accountable.” This works occasionally, until you read the vendor contract. Every serious AI vendor contract in 2026 disclaims responsibility for decisions made using the model, and courts have been consistent in enforcing those disclaimers. The vendor is responsible for the model behaving as documented. What you do with the outputs is your problem. This has been the standard software liability posture since roughly 1978, and it did not change when the software started making decisions.
The third answer is “the deploying team is accountable.” Legally, this is often correct, and it is where most regulatory pressure lands. But operationally it is a fiction. The team that deployed the model may consist of three engineers who fine-tuned a foundation model, integrated a vendor tool, added a rules layer someone in compliance wrote, and shipped the whole thing behind an API that a business unit calls without ever reading what it does.
Asking that team to defend a specific decision is like asking a plumbing crew to explain why a specific customer’s shower ran cold. They can describe the plumbing. They cannot tell you what happened to that specific customer.
The fourth answer, and the one everyone is quietly settling on, is “the executive is accountable.” Which is true in the sense that when things go badly enough, somebody senior gets asked to resign. It is also completely useless as a governance mechanism, because executives can neither prevent nor explain the individual decisions their systems make. The accountability lands somewhere but does not connect to anything that could have changed the outcome.
Four answers, none of which actually solve the problem. That is where we are.
The specific way AI makes this worse.
I want to be careful here because “automation reduces accountability” is not a new observation. It has been the subtext of every debate about algorithmic decision-making since at least the early ATM era. What is new is a specific mechanism that makes the AI version of the problem structurally worse than what came before.
Rule-based automated decisions, however hard to explain, can be reconstructed. If a claims system denied a claim in 2015, you could pull the code, trace the execution path, and produce a defensible narrative that ended in “the system correctly applied policy X.” The narrative was true whether or not policy X was a good policy. The point is that a specific decision led to a specific chain of reasoning, and that chain could be recovered.
Modern AI decisions cannot be reconstructed in that way. If an LLM agent refunds a customer, you can log the input, the tool call, and the output. What you cannot recover is why the model decided to invoke that tool with those arguments. The reasoning lives inside a matrix of billions of parameters that produced a specific output for a specific input under specific sampling conditions, and asking the model to explain itself afterward produces a plausible narrative generated by the same process that made the original decision. As several practitioners have pointed out over the last year, that post-hoc explanation has no privileged access to the actual cause. It’s fiction with metadata.
Which means the traditional recourse when a decision is challenged- produce the reasoning, defend it or correct it- becomes structurally impossible for a growing class of auto.
mated decisions. You have logs of what happened. You do not have a defensible account of why.
Regulators are starting to notice. The EU AI Act’s provisions on high-risk systems presuppose explainability infrastructure that most enterprises do not have.
Sector regulators in finance are asking questions that assume a chain of human sign-off that has largely disappeared. The gap between “accountability as regulators imagine it” and “accountability as the systems are actually structured” widens every quarter.
Where the failure modes are landing.
You don’t need to squint hard to see where this is going. A few patterns are already visible in the public record and the industry gossip.
There’s the quiet decision drift pattern. A model in production shifts subtly over months, either through retraining or through changes in the input distribution, and its decisions on borderline cases move in ways nobody notices until a regulator or a plaintiff’s lawyer runs a comparison. The team that deployed the model can produce logs of every decision. They cannot explain why the pattern of decisions has changed, because the pattern was never a designed behavior, just an emergent one. Every organization running non-trivial ML in production has had some version of this incident. Most have kept it quiet.
There’s the bystander effect pattern. When a decision is generated by an automated system that pulls from four vendor components, three internal services, and one policy layer, and the outcome is bad, every party in the chain has a legitimate reason to argue that the fault lies elsewhere. The vendor points to how the model was used. The integrator points to the vendor documentation. The policy team points to the integrator’s implementation. The business unit points to the policy. And the customer, meanwhile, is still owed an explanation nobody can produce.
There’s the accountability laundering pattern. Organizations increasingly present automated decisions as if they were the outputs of some neutral computational process, distinct from human choice. This is a rhetorical move rather than a factual one. Every automated decision reflects choices made by humans about what to model, what data to use, what thresholds to set, and what tradeoffs to accept. Presenting those choices as if the machine made them by itself is a way of shedding responsibility for decisions humans in fact made. Courts have started pushing back on this in specific cases. The pattern persists.
And there’s the scale asymmetry pattern. A human decision-maker who denies a hundred loans a year produces a hundred potential grievances. An automated system denying twenty thousand a year produces twenty thousand. The accountability infrastructure (appeals processes, ombudspersons, legal review capacity) was scaled for the human volume. When the volume goes up by two orders of magnitude, the infrastructure doesn’t. Which means individual grievances get less attention than they would have gotten from a human decision-maker, even as the total volume of grievances explodes. This gets called “efficiency” in vendor pitches. In practice, it’s the elimination of recourse for most affected parties.
None of these patterns are speculative. All of them have publicly reported precedents. What they have in common is that they don’t get solved by better models. They get solved, if at all, by rebuilding accountability infrastructure to match the scale and opacity of the systems we now deploy. Which is expensive, uncomfortable, and mostly not happening.
The Uncomfortable Question
Here’s the question I keep coming back to, and I think it’s the one the industry has been ducking.
Every serious AI product now ships with what the vendor calls a “human in the loop.” A person is somewhere in the chain, ostensibly reviewing, approving, or overriding the model’s outputs. This person is presented as the accountability anchor. Whatever the model does, the human signs off, so responsibility rests with the human.
The problem is that the humans in the loop, in most production deployments, are structurally incapable of exercising real accountability. They see the model’s output and, sometimes, a confidence score. They rarely see the reasoning. They almost never have time to reconstruct the input context in the depth that would be required to independently verify the decision. Their job, in operational terms, is to approve nearly everything the model produces, because if they didn’t, the throughput of the system would collapse and the automation’s business case would evaporate.
When something goes wrong, this human gets pointed at. This person “approved” the bad decision. In a formal sense, they did. In every practical sense, they were a rubber stamp attached to a decision that had already been made by a system they could not audit. Presenting that rubber stamp as accountability is, at best, an evasion. At worst, it’s a way of transferring blame from the organization to a low-level worker who never had the leverage to prevent the outcome.
We haven’t figured out what accountability looks like for automated systems without pretending. That’s the honest state of the field in 2026. Everyone is pretending, some more elegantly than others, but the pretending is universal, and the moments when the pretense breaks are exactly the moments the industry is trying not to think about.
What a serious response would involve.
I don’t have a clean answer. Nobody does. But if the industry were being serious about closing this gap, the response would involve some subset of the following, all of which are difficult and none of which are on most roadmaps.
Real evidence pipelines that capture not just what the model decided but everything about the context in which it decided, including the full input state, the retrieval sources, the tool schemas available, the sampling parameters, the model version, and the specific prompt configuration. Every enterprise AI deployment should be producing this evidence by default. Almost none are.
Formal delegation structures that name a specific human as accountable for each class of decision, with real authority to change the system when the pattern of decisions becomes problematic. Not a rubber stamp. A person with the standing to actually intervene, whose job includes reviewing patterns of outcomes and stopping the machine when it drifts.
Grievance and appeals infrastructure sized to the volume of automated decisions, not the historical human volume. If the machine makes twenty thousand denials a year, twenty thousand people can potentially appeal, and the process for those appeals cannot be another automated system, or you’ve just built an infinite loop.
Regulatory acceptance that automated decisions require different evidentiary standards than human ones. Most current frameworks still ask “who decided this” as if the answer were a single name. The frameworks need updating to ask “what system produced this, what evidence exists, and who has the authority to change the system.” A very different set of questions.
None of this happens without pressure. And pressure, historically, arrives after a specific and public failure, at which point the response is expensive and reactive rather than deliberate.
The bit worth taking seriously
The accountability gap in automated decisions is not a new problem, but it is entering a new phase. The volume of automated decisions is exploding. The opacity of those decisions is increasing. The regulatory infrastructure that governs them was designed for a world where a person could always be named. That world is ending.
What comes next is either a rebuilding of accountability infrastructure to match the systems we’ve deployed, or a slow drift into a state where automated decisions are effectively unaccountable, with occasional executives getting fired when the drift gets bad enough to make the news. The second option is where we’re currently heading. The first option is available, but it requires a conversation the industry has been avoiding.
Somebody’s going to have to have the conversation. Might as well be the people building the systems.
