A reviewer is not a control until it is measured
"Human in the loop" has become the default answer to almost every question about AI risk. Security guidance prescribes it for agents that combine untrusted input, sensitive data and external actions. Regulation requires it for high-risk systems. Procurement checklists ask for it. Most designs satisfy the requirement by adding an approval step and a reviewer.
The research on how people actually behave beside automated systems says that is not enough. Ben Green's 2022 study of 41 policies requiring human oversight of government algorithms concluded that people often cannot perform the oversight these policies assume, and that the requirement can give false comfort that legitimizes flawed systems. A reviewer who approves everything the system proposes adds cost and a signature, but not safety.
The thesis of this framework: design human involvement around the information a person adds, not around the need for a signature. Put people where the system is weakest (exceptions, low confidence, high impact and novel cases), give them what they need to disagree, keep their workload sustainable, and measure their catch rate with known errors. If you cannot show that reviewers catch mistakes, you do not have oversight.
Why humans in the loop fail
Automation bias
People tend to defer to automated recommendations, including wrong ones, and expertise reduces but does not remove the effect.
Figure 1
A wrong AI suggestion pulls experts down too
Radiologists' accuracy (%) in assigning the correct BI-RADS category, when the purported AI suggestion was correct vs incorrect
The clearest controlled evidence comes from radiology. In a study published in Radiology in 2023, 27 radiologists read mammograms with support from what they were told was an AI system; on some cases the suggestions were deliberately wrong. When the AI suggestion was correct, readers of all experience levels were accurate around 80% of the time. When it was wrong, inexperienced readers' accuracy fell to 19.8%, moderately experienced readers' to 24.8% and very experienced readers' to 45.5%. Even the most experienced group lost more than a third of its accuracy.
Knowledge work shows the same pattern. In the 2023 Harvard Business School and BCG field experiment with 758 consultants, those using GPT-4 on tasks within its capabilities produced higher-quality work faster, but on a task deliberately chosen to fall outside its capabilities, consultants with AI were 19 percentage points less likely to reach a correct answer than those without it. The tool did not just fail to help; it pulled competent people toward the wrong answer.
Review fatigue
The second failure is volume. When reviewers see a stream of items that are almost always fine, approval becomes reflexive. Healthcare has measured this for decades: a 2006 systematic review in the Journal of the American Medical Informatics Association found clinicians overrode drug safety alerts in 49% to 96% of cases. Alerts that fire too often teach people to dismiss them, including the ones that matter.
An approval queue for an AI agent is an alert system by another name. If 99% of proposed actions are correct, the reviewer's experience is that clicking "approve" is always right, until it is not.
Responsibility without capacity
The third failure is organizational. Oversight gets assigned to people who lack the time, the context or the authority to disagree: a reviewer shown only the system's summary, with a target of a few seconds per case, and no route to stop the system. That arrangement places accountability on the person while leaving the decision with the machine.
Where to put the human
There are four distinct ways to involve people, with different costs and different failure modes. Most processes should combine them rather than defaulting to approval of everything.
Figure 2
Four places to put a human, each with its own failure mode
Choose the placement per step, by what the person adds that the system cannot
Pre-approval
- Use when
- High-impact, hard-to-reverse actions on a short explicit list
- Human adds
- Accountability and a final check on the exact action
- Failure mode
- Rubber-stamping when volume is high and errors are rare
Exception handling
- Use when
- Low confidence, rule violations, novel or flagged cases
- Human adds
- Judgment on genuinely hard cases
- Failure mode
- Thresholds drift; easy errors above the threshold go unseen
Sampling QA
- Use when
- After the system acts, on a random sample of automated cases
- Human adds
- The true error rate of the automated band
- Failure mode
- Errors found late; needs a remediation path
Monitoring and override
- Use when
- Continuously, by a named owner with authority to stop
- Human adds
- Detection of drift, incidents and systemic failure
- Failure mode
- Dashboards nobody watches; stop button never tested
Exception handling is highlighted because it usually gives the most information per reviewer minute. It only works when paired with sampling QA, which measures what the threshold lets through.
Pre-approval belongs on actions that are high-impact and hard to reverse: payments above a threshold, deletions, legal or medical decisions, external communications in the organization's name. It is the most expensive placement and the most prone to rubber-stamping, so it should be reserved for a short, explicit list.
Exception handling routes only the cases the system flags (low confidence, rule violations, novel inputs) to people, who then handle them fully. This is where humans add the most information per minute, because the cases are genuinely hard.
Sampling quality assurance lets the system act, then has people review a random sample after the fact. It is how you find out the system's real error rate on cases it was confident about, which neither of the first two placements reveals.
Monitoring and override gives a named person live visibility and the ability to pause or reverse the system. It is the "stop button" that regulators expect and the kill switch that agent guardrails need.
A useful default for customer-facing processes is to add a fifth element: a guaranteed route to a person on request. Klarna, after reporting in 2024 that its AI assistant handled two-thirds of customer service chats, said in May 2025 that it was hiring human agents again and that customers must always be able to reach a human if they want one.
Designing confidence thresholds
Exception routing depends on a confidence signal and two thresholds: above the upper threshold the system acts, below the lower threshold a person handles the case fully, and in between a person reviews the system's proposal.
Four design rules matter more than the exact numbers:
- Calibrate on labeled data, not on the model's own score. A raw confidence score, and especially a language model's statement that it is confident, is not a probability of being right. Fit thresholds on a labeled set of your own cases, and recheck them whenever the model, prompt or input mix changes.
- Set thresholds by the cost of each error type. A wrongly approved refund and a wrongly rejected one rarely cost the same. Thresholds should reflect that asymmetry, not a single accuracy target.
- Add rule-based overrides. Some cases go to a person regardless of confidence: high value, a new counterparty, a vulnerable customer, a regulatory flag.
- Sample above the threshold. Without a random sample of auto-approved cases, the error rate of the automated band is unknown.
Figure 3Illustrative
Route by calibrated confidence, sample what you automate
Worked example: 10,000 supplier invoices a month processed by an AI extraction and matching step
- Calibrated confidence 0.97 or higher, no policy flagAutomated; 2% random sample reviewed7,200 cases, 144 touched
- Confidence 0.85 to 0.97Person reviews the proposal with source evidence2,000 cases, 2,000 touched
- Below 0.85, or high value, new supplier, or other flagPerson handles the case fully800 cases, 800 touched
People touch 2,944 of 10,000 cases (about 29%), and every touch has a purpose: a hard case, a policy exception, or measurement of the automated band.
In the illustrative example, people touch about 29% of cases instead of all of them, but every touch has a purpose: hard cases, policy exceptions, or measurement of the automated band. That is the trade to aim for.
Two interface choices reduce automation bias in the review band. Show reviewers the source evidence, not only the system's conclusion. And for a portion of cases, have reviewers record their own judgment before the recommendation is revealed; the gap between blind and assisted decisions is a direct measure of how much the recommendation is steering them.
Measuring whether review works
Most organizations measure reviewer throughput. Few measure reviewer effectiveness. The most informative measure is the seeded-error catch rate: deliberately insert cases with known errors into the review queue and track how many reviewers catch. It is the only direct evidence that the oversight is working.
Figure 4
Measure whether reviewers catch errors, not how fast they clear queues
A review-effectiveness scorecard for any human oversight step
- Seeded-error catch rateShare of deliberately inserted errors that reviewers catchWarningFalling, or never measured
- Approval rateShare of proposals approved unchangedWarningNear 100% for months
- Time per reviewMedian seconds spent per caseWarningToo short to read the evidence
- Blind vs assisted agreementHow often reviewers' own judgment matches the AI before it is shownWarningAssisted decisions far more aligned with AI than blind ones
- Escaped errorsErrors found later in auto-approved or reviewed casesWarningFound by customers or auditors first
- Queue load per reviewerCases per reviewer per shiftWarningRising with no change in staffing
An approval rate near 100% is a warning sign, not a success metric. It means either the system almost never needs correcting, in which case the review step may not be worth its cost, or reviewers have stopped looking. Seeded errors tell you which. For how this connects to broader value tracking, see From AI Experimentation to Enterprise Value.
What regulators expect
EU AI Act. Article 14 requires high-risk AI systems to be designed so that people can oversee them effectively, with measures proportionate to the risks, level of autonomy and context. It is unusually specific about what the people doing the oversight must be able to do.
Figure 5
Article 14 turns oversight into design requirements
What people overseeing a high-risk AI system must be enabled to do, and design features that provide it
- Article 14(4)(a)Understand the system's capacities and limitations, and monitor its operationDesign featurePlain-language limits documentation; dashboards for drift and anomalies
- Article 14(4)(b)Remain aware of the tendency to over-rely on outputs (automation bias)Design featureSeeded-error tests; blind-then-assisted review on a sample
- Article 14(4)(c)Correctly interpret the system's outputDesign featureShow source evidence and reasons, not only a score
- Article 14(4)(d)Decide not to use, or disregard, override or reverse the outputDesign featureOverride is one click, logged, and never penalized in targets
- Article 14(4)(e)Intervene or interrupt the system through a stop button or similar procedureDesign featureTested stop and rollback procedure with a named owner
Article 14 names automation bias explicitly, which makes the evidence above a compliance matter, not only a design one. For certain remote biometric identification systems it goes further, requiring that a match be verified by at least two competent people before action is taken. The timing has moved: the Digital Omnibus on AI, Regulation (EU) 2026/1744, entered into force on 27 July 2026 and deferred the high-risk obligations to 2 December 2027 for Annex III systems and 2 August 2028 for AI in products covered by EU product-safety law. The design work takes longer than the deferral, so it is better started now.
United Arab Emirates. Article 18 of the UAE's federal Personal Data Protection Law gives individuals the right to object to decisions made by automated processing, including profiling, that have legal consequences or seriously affect them, subject to exceptions for contract, other legislation and prior consent. In the Dubai International Financial Centre, Data Protection Regulation 10 requires autonomous systems processing personal data to operate within human-defined or human-approved purposes, and for high-risk processing requires an Autonomous Systems Officer. For organizations in the UAE, a documented route to human review of significant automated decisions is the practical baseline.
What to do in the next 90 days
- List every human review step in your AI-enabled processes, with the volume reviewed, time per case and approval rate.
- Flag any step with an approval rate above 98% and either remove it, convert it to sampling, or test it with seeded errors.
- Assign each review step a placement (pre-approval, exception, sampling, monitoring) and a stated reason.
- Recalibrate confidence thresholds on your own labeled data, with rule-based overrides for high-value and sensitive cases.
- Start seeded-error testing in at least one high-impact review queue and report the catch rate monthly.
- Give reviewers authority and evidence: source documents, time targets that allow real review, and a working way to stop the system.
- Map high-risk systems against Article 14 and local rules, so oversight evidence exists before December 2027.
The decision about which systems need which level of oversight belongs in the risk-tiering process described in The Enterprise AI Operating Model; the task-level view in How to Select Enterprise AI Use Cases helps decide where verification is cheaper than production.
Frequently asked questions
What is human-in-the-loop in AI automation?
It is the design of where and how people participate in an automated process: approving actions before they happen, handling exceptions the system flags, reviewing samples after the fact, or monitoring and overriding the system. Effective designs choose the placement per step and measure whether reviewers catch errors.
What is automation bias?
The tendency to over-rely on automated recommendations, including incorrect ones. In a 2023 Radiology study, radiologists' accuracy dropped sharply when an AI suggestion was wrong, and even very experienced readers fell from about 82% to 46%. The EU AI Act names automation bias in its human oversight article.
What does EU AI Act Article 14 require?
High-risk AI systems must be designed for effective human oversight. People overseeing them must be able to understand the system's limits, stay aware of automation bias, interpret outputs, override or disregard them, and stop the system. The obligations apply from December 2027 for most standalone high-risk systems.
How do you measure whether human review is effective?
Insert cases with known errors into the review queue and track the catch rate. Also watch approval and override rates, time per case, agreement between blind and assisted decisions, and errors found later in auto-approved cases. A near-100% approval rate is a warning sign.
Sources and further reading
- Automation bias in mammography, Dratsch et al., Radiology, 2023.
- The flaws of policies requiring human oversight of government algorithms, Ben Green, 2022.
- Navigating the jagged technological frontier, Dell'Acqua et al., Harvard Business School working paper, September 2023.
- Overriding of drug safety alerts in computerized physician order entry, van der Sijs et al., JAMIA, 2006 (AHRQ summary).
- EU AI Act, Article 14: Human oversight; Digital Omnibus on AI enters into force, K&L Gates Cyber Law Watch, 31 July 2026.
- UAE data protection law overview, DLA Piper; AI regulation in the DIFC, Mayer Brown, 19 January 2026.
- Agents Rule of Two, Meta AI, 31 October 2025.
- Klarna CEO reverses course by hiring more humans, Entrepreneur, 9 May 2025.