By Robin Martherus
The easiest answer to agentic AI risk is also the most dangerous one: put a human in the loop.
You can hear the argument forming everywhere. Agents are becoming more autonomous. They can call tools, move data, write code, open tickets, send messages, provision infrastructure, and make decisions faster than the systems around them were designed to inspect. So the security answer seems obvious: add human approval. Put a person between the agent and the action. Create an escalation path. Add accountability. Make sure there is, as the old phrase goes, a neck to strangle when something goes wrong.
It sounds responsible. It is often the opposite.
Human-in-the-loop is useful at the beginning of a governance system. It is useful for training, calibration, high-risk exceptions, disputed decisions, and moral boundaries that have not yet been encoded. It is useful when a system is new, trust is unearned, and the organization does not yet know how the agent will behave under pressure.
But as the default operating model for agentic governance, it is a regression. It takes a technology whose value is autonomy and turns it into a notification generator. It recreates one of the oldest failures in security operations: drowning humans in decisions they cannot evaluate at the speed, volume, or context required.
Security teams already know how this story ends. It ends in alert fatigue.
And fatigue is only half the problem. Recent research shows the approval prompt itself can be attacked: the human approves a description of the action written by the agent, which means the agent can narrate its own permission slip. We will get to that.
The SOC Already Lived This Failure
Every security operations center has learned the same lesson the hard way: more alerts are not more security.
The first generation of monitoring tools assumed that visibility was the missing ingredient. If defenders could see more events, they could stop more attacks. So the industry gave them more events. Endpoint alerts. Network alerts. Identity alerts. Cloud alerts. SaaS alerts. DLP alerts. Vulnerability alerts. Anomaly alerts. Every tool became a sensor. Every sensor became a queue.
The result was not better judgment. It was exhaustion.
Analysts did not need more notifications. They needed fewer, better, higher-confidence decisions. They needed correlation. Suppression. Prioritization. Automation. Context. They needed systems that could handle routine decisions without dragging a human into every branch of the tree.
A 2026 survey by Ndichu and colleagues of AI-driven alert screening in SOCs describes the actual downstream work as filtering, triage, correlation, and contextualization before an alert deserves analyst attention. That is the lesson agent governance should import: the point is not to route more events to people. The point is to reduce the number of events that need people by making the machine layer more trustworthy.
Agentic governance is about to repeat the same mistake if we are not careful.
An autonomous agent can take hundreds or thousands of small actions in pursuit of a goal. Read this file. Query that database. Call this API. Summarize this thread. Create this branch. Invoke this tool. Ask another agent for help. Each action may be individually low risk. The risk often lives in the pattern, the purpose, the timing, the delegation chain, or the context around the action.
If the answer is to ask a human about every meaningful step, the system does not become governed. It becomes unusable. If the answer is to ask a human only when something looks risky, then the real question is not “is a human in the loop?” The real question is: what decided this was risky, and why do we trust that decision?
That is the governance problem. The human is not the governance layer. The human is one input to the governance system.
I Already Feel This in My Own Agent Workflows
This is not just a SOC problem. Anyone using modern coding agents has already felt the smaller version of it.
Every time I send Claude Code, Codex, or another agent to perform a task, the same pattern appears. The agent starts working. It reads files. It proposes edits. It wants to run a command. It wants to modify a file. It wants to continue after a tool call. Again and again, the system reaches back to me for permission.
The reason is understandable. These systems have guardrails. They are designed to keep a human driver in control before the agent takes actions that might matter. That is the right instinct. But the implementation often collapses into repeated low-value prompts.
It does not take long before I am wearing out the “2” key on my laptop, unconsciously choosing the approve-all path just to keep the work moving.
Then, at some point, I come back to attention and realize what happened. I approved a chain of actions because the first ten prompts were routine, obvious, and not worth interrupting me for. By the time the agent reached an action that actually changed the direction of the project, I had already been trained into approval mode.
That is not laziness. That is alert fatigue.
The system asked me too many questions that it should have been able to answer from intent. If I asked the agent to clean up a generated file, I should not need a separate permission prompt for the delete operation. If I asked it to update a draft, I should not need to approve every ordinary file write inside that declared scope. If I asked it to run the test suite, I should not be asked whether running the project’s normal test command is acceptable.
Those are not moral decisions. They are not governance decisions. They are scope-compliance decisions. The agent declared, or should have declared, what task it was performing. The governance layer should evaluate whether the requested action fits that task.
If it fits, approve it automatically and attest it.
If it does not fit, challenge it.
If it is destructive, ambiguous, outside the declared scope, or unusually high impact, escalate it with context.
That is how permission should work. Human attention should be conserved for the decisions that actually need a human. A system that spends human attention on obvious approvals is not being safer. It is training the human to stop reading.
The Industry Is Moving, But Not Far Enough
To be fair, the industry is not blind to this problem. The direction is already shifting away from a naive “humans approve everything” model.
Gartner, as reported by ITPro, is warning against one-size-fits-all agent governance and pushing proportional controls based on an agent’s autonomy level: observe, advise, act with approval, and fully autonomous operation inside stricter guardrails. That is the right shape. A read-only agent summarizing public documentation should not be governed like an agent that can move money, modify production systems, or email customers.
Richard Kang’s recent research on governed AI-assisted engineering makes the same point in a regulated software context. His proposed GAIE framework routes coding tasks through different oversight tiers based on regulatory impact, customer proximity, reversibility, and data sensitivity: human-in-the-loop for strategic functions, human-over-the-loop for customer-impacting work, and automated-with-monitoring for internal work. Again, the shape is right: oversight should be graduated, not universal.
The product market is moving in parallel. LangGraph ships human-in-the-loop interrupts, OpenAI’s Agents SDK supports resumable approval flows, and CrewAI lets tasks require human review. These are useful primitives. They make it easier to stop an agent before a tool call, API action, or final output crosses a boundary.
Enterprise platforms are forming around governance control planes: ServiceNow’s AI Control Tower and its Veza acquisition point toward access visibility across human, machine, and AI identities; Okta is pushing agent identity and shutdown controls; Noma Security is building monitoring, testing, and safeguards for autonomous agents.
Academic work is seeing the same gap from the other side. A 2026 survey by Wang, Li, and Tian of agent-human interaction security found that production agent systems heavily rely on human-centric mechanisms such as policy specification, runtime approval, and scope configuration, while research concepts such as intent anchoring and trust labeling have not meaningfully reached production. The paper names the core trade-off directly: current designs leave users caught between approval fatigue and uncontrolled autonomy.
This is all directionally correct.
But it is not the long-term answer if the industry stops there.
Approval primitives are not governance. Control towers are not governance. Identity inventories are not governance. They are pieces of the governance substrate. The danger is that the market takes the first visible artifacts of control — dashboards, approval prompts, human review queues, kill switches, agent registries — and mistakes them for the architecture.
That is exactly how we recreate the SOC problem in agentic form.
There is another distinction most products still blur. Some agents are cooperative: registered, instrumented, identity-bearing, and willing to present governance signals. Others are wild: browser-driven, locally installed, operating through a human session, hidden inside a workflow, or simply unknown to the enterprise. Most products are strongest on the cooperative path. They can govern agents that enroll, disclose themselves, and operate inside the vendor’s control plane. The harder problem is the wild path, where agentic behavior appears as ordinary browser traffic, ordinary API calls, ordinary configuration execution, or ordinary workflow automation.
That is where approval prompts and agent registries stop being enough. That split is a subject of its own; I wrote about it in Agents Are Not Agentic AI. This article stays on the cooperative path — because even there, where governance has every advantage, the industry’s default answer is failing.
The Approval Trap
The industry likes approval flows because they look accountable. Someone clicked yes. Someone accepted the risk. Someone can be named after the incident.
But an approval button is only as good as the context behind it. Most approval prompts fail that test.
They ask a human to decide under three bad conditions:
- Too little context: the reviewer sees the requested action, but not the agent’s full intent, history, lineage, or downstream plan.
- Too much volume: the reviewer sees so many prompts that approval becomes muscle memory.
- Too little time: the agent is operating at machine speed, while the human is being asked to make a judgment in seconds.
That is not governance. That is liability routing.
The human becomes a ceremonial checkpoint inserted into a system that still cannot answer the real questions. What is the agent trying to accomplish? Is this action inside the purpose it declared? Has this agent earned the right to act with less friction? Does the organizational context make this action inappropriate even if it is technically allowed?
If the system cannot answer those questions, asking a human to click approve does not fix the architecture. It hides the missing architecture behind a person.
Worse, it trains the organization to confuse accountability with control. After the incident, the audit trail will show that a human approved the action. That may satisfy a process requirement. It does not mean the human understood the action, the agent’s trajectory, or the consequences.
A neck to strangle is not the same thing as a control that works.
There is an even sharper problem — the one promised at the start. A June 2026 paper by Xiaoqi Weng on coding-agent approvals names a failure mode called “Lies-in-the-Loop”: the human approves a description of an action written by the agent, not necessarily the action that actually executes. The paper’s proposed property, consent integrity, says the approval surface must be bound to the real action at the execution boundary, rendered through a trusted path the agent cannot spoof.
That matters because it shows the approval prompt itself can be part of the attack surface. A human approval flow is not automatically safe. If the human is approving an agent-authored summary, the agent may be narrating its own permission slip. If the human sees too many prompts, they stop reading. If the prompt is not bound to the exact execution event, the approval record proves very little.
So the industry’s current direction is both promising and dangerous. Proportional oversight is replacing blanket human review. Control planes, identity graphs, guardrails, and audit trails are becoming real product categories. But approval workflows are easy to ship and easy to overuse. Human review can become compliance theater: someone clicked yes, therefore the system was governed. And many products still mostly answer identity, visibility, access, and workflow questions, not the deeper intent, trust, and should questions.
This is where the line has to be drawn. Human oversight is not the problem. Human oversight as the default runtime decision engine is the problem.
Autonomy Changes the Shape of Review
Human review made sense in older systems because the thing being reviewed was usually discrete and slow.
A firewall rule. A privileged access request. A production change. A vendor exception. A new OAuth application. A human could inspect the request, reason about the risk, and approve or deny it before the world changed too much.
Agents do not operate that way.
An agent is not a single request. It is a trajectory. Its risk is not always visible at the first action. It may begin with a legitimate purpose, discover new information, delegate to another agent, call a tool the operator did not anticipate, and shift tactics while still pursuing the same broad goal.
That means the review target is no longer a point-in-time approval. The review target is the agent’s evolving relationship to its declared purpose, earned trust, and governing constraints.
This is exactly where “human in the loop for everything” breaks down. A person cannot sit inside the trajectory at machine speed. They cannot manually evaluate every micro-decision. They cannot reconstruct composite intent across dozens of cooperating agents while the system is moving.
The governance layer has to do that work.
Humans still matter. They matter more than ever. But they belong in different places:
- At the beginning, defining the moral and organizational boundaries the system must enforce.
- During onboarding, calibrating new agents while trust is still unearned.
- At escalation points, resolving ambiguity the system cannot safely decide.
- In audit and feedback loops, correcting the governance system when its decisions drift.
- At high-consequence thresholds, where the organization deliberately requires human authority.
That is not human out of the loop. It is not quite the industry’s “human on the loop” either — a person watching runtime, ready to intervene. It is human above the loop: holding authority over the system rather than a seat inside it.
The Better Model: Intent, Trust, and Should
Human review is often used as a substitute for missing intent.
The system does not know what the agent is trying to do, so it asks a person whether the next action looks acceptable. That is backward. The agent should have declared its purpose before the action ever reached the reviewer.
Intent changes the control model.
If an agent declares, “I am summarizing customer support tickets to identify the top five product defects from the past week,” that declaration becomes a scope contract. The system can evaluate later actions against it. Reading support tickets may fit. Querying payroll does not. Exporting raw customer data to a public repository does not. Creating a new admin account does not.
The human should not be asked to approve each of those actions in isolation. The governance system should enforce the contract.
When the agent asks to amend the contract, then a human may belong in the loop. When the declared purpose is ambiguous, a human may need to clarify it. When the action is outside the declared scope but there is a plausible emergency, a human may need to arbitrate.
But that is a very different use of human judgment. The human is not a speed bump in front of every action. The human is the authority that shapes, amends, and adjudicates the purpose contract when the system cannot do so safely on its own.
Trust then determines how much friction the agent experiences inside that contract.
Trust is not a label assigned at deployment. It is not “this agent is approved” or “this agent is safe.” Trust has to be computed from behavior over time. It has to grow slowly, collapse quickly, and remain scoped to the purpose and environment where it was earned.
A new agent should start with friction. It has no history. It has no demonstrated alignment. It has not shown how it behaves under stress, ambiguity, failure, or adversarial input. Human oversight at this stage is not only reasonable; it is necessary.
But if the agent repeatedly declares clear intent, stays inside that intent, behaves consistently, accepts constraints, handles exceptions properly, and produces clean attestations, the system should learn from that. The agent should earn more autonomy. The governance system should require fewer human interruptions for routine actions inside the trusted envelope.
The inverse should also be true. If the agent drifts, changes behavior under pressure, touches unusual data, delegates unexpectedly, or approaches a normative boundary, trust should decay. Friction should increase. The system should challenge, constrain, or escalate.
The “should” layer sits above both intent and trust. This is where humans matter most, but not as per-action approvers. Humans create safety by defining what the system should value, what it must never do, what requires escalation, and whose interests count when values conflict. That is the normative layer.
A company may decide that no agent can move money above a threshold without signed human approval. A hospital may decide that an agent can break ordinary workflow constraints during a life-threatening emergency, but only with immediate physician notification and after-action review. A financial institution may decide that an agent cannot access certain deal materials when an employee is in a notice period, regardless of the employee’s credentials. A software company may decide that no coding agent can execute unpinned code from a repository configuration file, no matter who opened the project.
Those are human decisions. They are moral, legal, organizational, and contextual. They should be made by humans.
But once they are made, the system should enforce them at machine speed.
That means even the “should” layer should minimize human interruption during runtime. Most should decisions are not mysterious moral dilemmas. They are enforceable rules compiled from laws, regulations, compliance requirements, contracts, professional standards, organizational policies, and explicit human-defined values.
No agent should exfiltrate secrets to a public repository. No coding agent should execute unpinned setup code from an untrusted project configuration. No payroll agent should send salary data to a personal email account. No purchasing agent should split transactions to evade an approval threshold. No research agent should access deal materials during a trading blackout. These do not need a human approval prompt. They need a fast, typed, enforceable normative rule that denies the action and records why.
The human belongs in the grey areas.
This is not a new problem. Standards-based authorization systems such as XACML had to deal with conflicting policy results. One policy might permit an action while another denies it. The system needed a combining algorithm to decide what the final result meant.
The OASIS XACML 3.0 core specification defines rule- and policy-combining algorithms for exactly this purpose. deny-overrides gives priority to a deny decision when policies disagree. first-applicable takes the first determinate result in evaluation order. Four more variants cover the remaining cases, including policies that must always end in a definite permit or deny.
Those algorithms are crude compared with what agentic governance needs, but the core insight was right. When multiple rules speak at once, the system needs an explicit conflict-resolution model. It cannot pretend the conflict is not there.
Agentic AI makes this harder because agents can search the grey areas deliberately.
A capable agent can decompose a goal into actions that individually fall under permissive rules while avoiding the rule that would have blocked the composite purpose. One agent reads customer data. Another formats it. A third sends an “innocent” summary. Each local action may pass. The combined pattern may be exfiltration. That is the specification-gaming problem behind swarm attacks: the agentic system exploits the space between rules, scopes, and enforcement boundaries to get the answer it wants.
This is where human judgment still matters. Not because every should decision needs a person, but because unresolved conflicts, novel composites, and policy gaps need authoritative interpretation. Sometimes one human can decide. Sometimes a governance board, legal team, ethics committee, security owner, or affected business unit needs to agree on the rule. Once that decision is made, it should not remain a recurring human approval prompt. It should become part of the normative layer.
That is the difference between human governance and human interruption. The first scales. The second collapses.
The resulting model is human-calibrated autonomy:
- Humans define the boundaries: policies, forbidden actions, escalation thresholds, moral constraints, regulatory obligations, and risk appetite.
- Agents declare intent before acting: purpose, scope, resources, time bounds, delegation rights, and amendment rules.
- The system computes trust continuously: behavior, lineage, intent compliance, stress response, anomaly signals, and prior attestations.
- The “should” layer compiles human-defined rules into autonomous enforcement: not just whether the action is allowed, but whether it is appropriate now, here, by this agent, for this purpose.
- Conflict-resolution logic handles routine overlaps: deny-overrides, permit-overrides, escalation thresholds, risk-weighted composition, and other explicit combining rules.
- Humans handle the irreducible cases: unclear intent, high-consequence exceptions, unresolved policy conflicts, novel composites, and governance failures.
- The system learns from those interventions: human decisions become new rules, calibration data, or combining logic, not permanent manual bottlenecks.
This is the right use case for AI.
Not AI as an ungoverned actor. Not AI as a magic risk classifier. AI as part of a governance system that can reason over intent, behavior, trust, and context faster than a human SOC queue ever could — while still being bound by human-defined normative constraints.
There is a fair objection here. Earlier I asked what decided an action was risky, and why we should trust that decision. Doesn’t this model just move the same question into a machine? It does — and that is what makes the question answerable. A fatigued human clicking approve on prompt forty-seven leaves no reasoning behind. A governance layer leaves all of it. Its rules can be read before deployment. Its decisions are attested and can be replayed afterward. Its drift shows up in audit instead of in an incident. And it cannot be trained into approval mode. The trust question does not disappear. It moves from a psychology problem, which does not scale, to an engineering and audit problem, which does.
As agents become more autonomous, governance has to become more autonomous too. Otherwise the control plane becomes slower than the thing it is supposed to control.
The Bottom Line
“Human in the loop” sounds like safety because it puts a person near the decision. But proximity is not control.
The SOC learned this years ago. More alerts do not create better security. More approval prompts do not create better governance. They create queues, fatigue, and the appearance of accountability.
Agentic AI does not need humans clicking yes on every action. It needs humans defining the boundaries, calibrating the trust system, resolving the hard cases, and auditing the machinery. The runtime layer needs to be able to act autonomously because the agents it governs act autonomously.
The goal is not to remove humans from governance. The goal is to stop using humans as the missing architecture.
Leave a Reply