Leash 1.0 is out. Free for individuals.
All articles ENGINEERING

Human-in-the-loop without approval fatigue

Share
LinkedIn

If every action asks for approval, people stop reviewing. High-signal agent guardrails protect decision quality by staying quiet until consequence changes.

Human oversight works when the system escalates the right action, with the right context, to the right person, and binds the answer to the exact pending request.

A flood of low-value alerts being filtered into one clear high-risk approval
human in the loop AIapproval fatigueAI agent guardrailsAI agent approvalshuman oversight

The approval paradox

Permission prompts are attractive because they keep a human nominally in control. If the system asks too often, the prompt becomes a reflex test. Anthropic reported that users approved roughly 93 percent of Claude Code permission prompts in its telemetry, a strong signal that volume can erode attention even among technical users.

The solution is not to remove people from every decision. It is to reserve human attention for moments when intent, consequence, or authority changes. Safe autonomy and meaningful oversight reinforce each other when the boundary is designed well.

Escalate consequences, not commands

A policy based only on tool names creates noise. The same shell can list files or erase them. The same database tool can query test data or delete production records. The same email tool can save a draft or send to thousands of customers. Evaluate arguments, target, data class, environment, scope, reversibility, and external visibility.

Use the narrowest useful escalation. A person may approve one action, edit the proposed arguments, reject with feedback, or approve a bounded class for a short period. Avoid approvals that silently grant permanent authority to a tool or agent.

Routine AI agent actions flowing quietly while consequential actions reach a human reviewer
Escalate changes in consequence and authority, not every harmless command.

What a useful approval contains

A reviewer should understand the decision in seconds. Show the agent and user identity, action, target system, affected resource, data sensitivity, scope, reason for escalation, and whether the operation can be undone. Include the exact command or structured arguments for technical reviewers, but lead with impact.

Do not bury the risk in raw logs. A message such as wants to run a tool is not enough. State that the action will delete customer records, publish a release, send an external message, expose a secret, or change production access.

  • Who or what is acting
  • What will change and where
  • How many objects or people are affected
  • Which policy triggered and why
  • Whether the action is reversible
  • Approve once, edit, reject, and safe alternative options
  • Expiration time and link to supporting evidence

Route to the right person

The person running the agent may not own the consequence. A developer can approve a test command but not a production access change. A sales operator may prepare a customer campaign while legal or communications owns the final publish decision. Build routing around accountability and risk tier.

Approvals need a recovery path when the reviewer is unavailable. Expire the request, fail closed for high-risk actions, and let the agent explain what it needs. Never auto-approve simply because a notification timed out.

Bind every answer to one request

A durable approval record should identify the exact action, correlation, arguments, policy version, reviewer, response, and time. If the action changes after approval, evaluate it again. A modified command or different recipient is a new request.

Using several devices must not create confusion. An answer should resolve only the exact request that created it and disappear everywhere else once decided. A generic approval sent to every session could allow the wrong conversation or agent to continue.

One human approval cryptographically bound to one exact AI agent request while altered requests are rejected
An approval should resolve one exact request; changed targets or arguments require a new decision.

How Leash protects attention

Leash keeps approvals, questions, plan reviews, blocked actions, and completed work together. Desktop, web, and optional mobile apps show each person the requests that need their decision. Every response applies only to the exact action that created it.

Safe work continues without interruption. High-impact actions arrive with enough context to understand the consequence. A user can briefly pause protection for one conversation, while the pause remains visible and automatically expires instead of becoming a hidden permanent bypass.

Measure the quality of the control

Track approval rate by risk class, review time, edit rate, rejection reason, expired requests, overrides, repeated prompts, and incidents caught or missed. A 99 percent approval rate on one policy may indicate a safe automatic rule, not successful oversight. A high rejection rate may reveal a dangerous agent or a poorly explained action.

Review the data with users. The best policy removes low-value friction every week while sharpening the handful of decisions that deserve human judgment.

Sources and further reading

Continue the research