Human-in-the-Loop: How to Supervise AI Agents Without Slowing Automation Down

A distribution company set up its first AI agent to handle returns. It worked: it read the customer’s email, checked the order in the ERP, decided whether a credit note was due and issued it. Three happy weeks. Until a customer wrote «you sent me 400 units and I only ordered 40» and the agent, obediently, issued a €4,800 credit note for the customer’s own typo.

Nobody reviewed it because nobody was supposed to. The agent was designed to execute, not to ask. And that is the question almost no company asks before automating: it is not whether the AI makes mistakes —it does, like anyone— but what happens when it does and nobody is watching.

Human oversight of AI agents is exactly that: deciding where, when and on what criteria a person steps into the process. Designed well, it does not slow automation down. Designed badly —or skipped entirely— it is the reason so many AI projects end up switched off.

Why full autonomy is the most expensive mistake

Gartner predicts that over 40% of agentic AI projects will be cancelled before the end of 2027 — and not because of technical limits: the causes are escalating costs, unclear business value and inadequate risk controls (Gartner, 2025). In plain terms: the models are not what fail, process governance is.

What happens in practice is a pendulum. A company starts with an agent on a short leash, sees it get things right, and gradually lets it run end to end. Fine for months. Then the edge case arrives —the one that was not in the tests— and the agent acts with exactly the same confidence it shows when it is right: it issues the credit note, sends the email, approves the invoice.

The damage is rarely the mistake itself; it is that the mistake is irreversible and silent. A miscoded ledger entry can be fixed. An email already sent to a customer cannot. Nor can a transfer. That is why the design question is not «how often is the agent right?» but «can this specific action be undone?».

There is a further layer: since 2 August 2026, the EU AI Act’s obligations for high-risk systems apply, and its Article 14 requires effective human oversight — whoever oversees the system must be able to understand its limitations, interpret its output, disregard or reverse it, and stop the system through a stop button or equivalent procedure. If your automation touches recruitment, credit decisions or worker evaluation, this is no longer best practice: it is a legal requirement. We cover it in our EU AI Act guide for SMBs.

The three levels of human oversight for AI agents

There is no binary «supervise or don’t». There are three degrees, and a single process usually needs all three at different points.

• Human-in-the-loop (approves first): the agent prepares the action and waits for a person to sign off. Nothing leaves without a click. Reserved for the irreversible: payments, credit notes, contracts, customer communications, onboarding and offboarding.
• Human-on-the-loop (monitors and can stop it): the agent acts on its own but leaves a visible trail, and a person reviews by sampling or by exception, with the power to halt it. This is the sweet spot for medium volume: document classification, accounting coding, ticket assignment.
• Human-out-of-the-loop (autonomous within limits): the agent acts with no intervention inside a closed perimeter. Only for reversible, low-impact actions: tagging emails, moving files, updating an internal field, drafting text.

The classic mistake is applying one single level to the whole process. Put human approval on all 40 steps and you have built a bottleneck — people will end up bulk-approving without looking, which is worse than no oversight because it creates a false sense of control. Put it on none, and you get the €4,800 credit note.

How to design approval gates, step by step

This is the method we apply when deploying agents in companies that already have their systems in place:

1. Inventory the actions, not the tasks. List everything the agent can do to the outside world: send, pay, delete, publish, change a master record. Those are the only ones that need a governance decision.
2. Rate each action by reversibility and impact. Two questions: can it be undone within an hour? What does it cost if it goes wrong? Irreversible and expensive goes in-the-loop; reversible and cheap goes autonomous.
3. Use thresholds, not binary rules. «Credit notes under €200 are automatic, above that they need approval» works far better than «all credit notes need approval». The threshold concentrates human attention where the money is.
4. Add a confidence threshold. When the agent is unsure of its own answer, it must escalate. An agent that says «I don’t know» and asks is worth more than one that is right 95% of the time and silently wrong the other 5%.
5. Make approving take five seconds. If the request arrives by WhatsApp or email with two buttons and the context summarised, it gets approved on the spot. If it means logging into a platform, requests pile up and the process stalls.
6. Log everything. Who approved what, when, with what information visible, and what the agent proposed. Without an audit trail there is no way to audit — or to tune the thresholds.
7. Review thresholds quarterly. Oversight is meant to be graduated: what needs approval today can move to sampling after three clean months. You let out the leash based on data, not intuition.

It is worth separating this from AI data governance, which answers «what information can the agent see». Here we are dealing with the other half: what we let it do. Two distinct controls, and you need both.

What you gain when oversight is designed properly

The goal is not «less automation with more control», but more automation precisely because control exists:

• You automate more, not less. With approval gates scoped tightly, management is willing to hand over processes it would never have trusted to an agent with no brakes.
• The cost of error stops being a lottery. Mistakes stop in the approval queue, where they cost a minute, instead of reaching the customer.
• You comply without rebuilding anything. Control points and a decision log are exactly what the regulation asks for in high-risk processes.
• The team trusts the system. An agent that asks about the three decisions that matter each day drives adoption; one that acts blindly drives resistance and eventually gets unplugged.

When does each level make sense?

Practical criteria to decide without overthinking it:

• Prior approval if the action moves money, leaves the company, affects people (hiring, evaluation, credit) or changes a master record.
• Sampling oversight if volume is high and errors are detectable and correctable afterwards: classification, coding, routing, prioritisation.
• Full autonomy if the action is internal, reversible and the worst outcome is having to repeat it.
• And one rule for launch: for the first 30 days, everything goes through prior approval. That period is not red tape — it is the dataset you will use to decide what to release. When a process involves several coordinated agents, this control layer becomes essential: see multi-agent systems.

Frequently asked questions

Doesn’t human oversight cancel out the benefit of automating?

Only if you supervise everything. In a well-designed process, somewhere between 3% and 10% of runs reach a human; the rest go through on their own. The person shifts from doing 200 tasks to deciding on 8 — and those 8 are the ones that matter.

What is the difference between human-in-the-loop and human-on-the-loop?

In-the-loop means the agent stops and waits for approval before acting. On-the-loop means it acts on its own while a person monitors the activity and can intervene or halt it. The first protects irreversible actions; the second scales with volume.

Does the EU AI Act require me to supervise every agent?

No. The effective human oversight obligation in Article 14 applies to systems classified as high-risk — CV screening, employee evaluation, credit scoring and biometrics, among others. An agent that classifies invoices or answers internal questions does not fall under it, although supervising it is still advisable.

How do I stop approval becoming a mindless «OK» click?

By reducing the number of approvals and improving their context. If someone receives 80 approval requests a day, they will stop reading them: raise the thresholds. Every request should show what the agent proposes, on what data, and what happens if it is rejected.

How much does it cost to add this layer to an existing automation?

It is the cheap part of the project: you are inserting decision and notification points into a flow you already have, not rebuilding it. It usually takes days, and it prevents the kind of incident that costs thousands of euros and a customer relationship.

An AI agent without oversight is not a more automated company: it is a company with more surface area for error and fewer people looking at it. The difference between automations that last and those that get switched off after six months almost never lies in the model — it lies in whether someone sat down and decided which actions the agent was allowed to take alone. If you already have agents running, or you are about to, that is the exercise still pending.

Level	How the agent acts	Human review
In-the-loop (approves first)	Stops and waits for sign-off	Before every sensitive action
On-the-loop (monitors and can stop it)	Acts alone, leaves a visible trail	By sampling or by exception
Out-of-the-loop (autonomous within limits)	Acts with no intervention	None (reversible actions only)

Contact us and we’ll analyse your case for free →