Design the Queue Your Agent Escalates Into

What breaks when "a human approves this" meets production volume

The first version is always the same, and it is always fine. Somebody adds a check before the risky tool call. The agent pauses, a row lands in a table, a Slack message goes out, an ops person clicks approve, the agent carries on. It works in staging. It works in the pilot with forty actions a day. Everyone agrees that human oversight is important and the ticket closes.

What ships to production is not that. What ships is a system where the number of pauses per day is a function of traffic, and the number of people available to unpause them is a function of a headcount plan written before anyone knew what the escalation rate would be. Those two numbers were never reconciled, because nobody framed the problem as one where they would need to be. And so the failure arrives on a predictable schedule: three weeks in, the queue has a backlog; six weeks in, someone clears the backlog in one sitting by approving everything; three months in, the approval rate is above ninety-five per cent, the median time in review is eleven seconds, and the oversight function that appears on the architecture diagram has quietly stopped existing.

That is the failure mode worth naming up front, because everything else in this guide is a defence against it. A widely cited rule of thumb in human-in-the-loop deployments is that an approval rate above roughly ninety per cent is a leading indicator that your escalation triggers are too broad. It is a heuristic, not a law, and it will not hold for every workload. But the mechanism behind it is robust: when nineteen items in twenty are fine, a reviewer's prior becomes overwhelming, the queue rewards speed over scrutiny, and attention degrades to the level the incentives support. You have not built oversight. You have built a very expensive random number generator that outputs "approve".

The reframe that fixes this is small and load-bearing. Escalation is a throughput problem before it is a safety problem. The safety property you want — that a human meaningfully considers the risky actions — is only achievable if the arrival rate of risky actions stays inside the service rate of the people considering them, with headroom. Trigger design, routing, review UX, SLAs and staffing are not five separate workstreams. They are five dials on one queue, and turning any of them moves the others.

• The queue is a capacity system with a service level, not a checkbox in an architecture diagram.
• Approval rate above roughly 90% means the trigger is too broad — a heuristic, but a reliable early warning that the queue is drifting to a rubber stamp.
• Escalate on risk signals, not on action categories. "All refunds" is a category. "Refund over the threshold, or flagged account, or low confidence, or irreversible" is a risk trigger.
• The review card is the product. A clean diff and a plain-language summary get considered decisions; a raw JSON payload gets rubber-stamping or a very slow queue, and usually both.
• Decide the timeout default deliberately. Fail-open, fail-closed and escalate-up are all correct answers to different blast radii, and the wrong one is picking a single policy for the whole queue.
• Rejections are your next eval set. Capture the reason, or you have paid for the signal and thrown it away.

What should escalate, and what should not

Almost every review queue that is drowning is drowning because its trigger is a category rather than a risk signal. Category triggers are seductive because they are easy to write, easy to explain to a compliance reviewer, and easy to defend in a design document. "All refunds go to a human." "Any email to an external address goes to a human." "Every database write goes to a human." Each of those is one line of code and each of them, at production volume, produces a queue whose contents are overwhelmingly boring.

The distinction matters because a category is a proxy for risk, and it is a bad one. Refunds are not risky; large refunds to accounts with unusual activity are risky. Outbound email is not risky; outbound email to a customer whose account is in dispute is risky. When you escalate on the category you send a human every instance of a class that is ninety-eight per cent benign, and you have chosen your approval rate in advance without noticing.

The alternative is to escalate on a disjunction of risk signals evaluated per action. Instead of if action.type == "refund", you get something closer to if amount > threshold or account.flagged or confidence < cut or not reversible. The same refund tool now produces two very different paths depending on the instance, which is exactly what you want, because the instances genuinely are different.

Routing by uncertainty score is the backbone of this. High-confidence outputs auto-approve. Borderline cases are flagged for review. Low-confidence outputs escalate immediately rather than waiting in the general queue. The effect is that reviewer queues hold only the decisions where human judgement changes the outcome, which is the entire point of having reviewers at all.

A trigger taxonomy. Score each action against all six; escalate on the disjunction, not on any single one.

Signal	The question it asks	A cheap proxy you probably already have	Escalate when
Reversibility	Can we undo this in one action, within an hour, without telling anyone?	Whether a compensating action exists in the tool registry	No compensating action exists
Blast radius	How many records, accounts or customers does this touch?	Row count in the intended write; recipient count on a send	Above a tier boundary, e.g. one record versus five hundred
Confidence	How sure is the model, calibrated against outcomes?	Verifier score, ensemble disagreement, self-report plus a check	Below the calibrated cut for this action class
Novelty	Have we seen an action shaped like this before?	Embedding distance to the corpus of previously accepted actions	Outside the 95th percentile of the prior distribution
Monetary value	What does the worst case cost if this is wrong?	The amount field, converted to a single currency	Above tiered thresholds, e.g. £250 / ₹25,000 then £5,000 / ₹5,00,000
Regulatory sensitivity	Does this touch protected data or a regulated decision?	PII classifier output; account jurisdiction and sector flags	Any hit, regardless of the other five signals

Two signals deserve a warning. Novelty is powerful and it drifts. An embedding-distance trigger calibrated in March will fire on half your traffic in September if your product has changed, and it will fire on almost nothing if your accepted-action corpus has quietly absorbed a category of mistakes. Recalibrate it on a schedule, not when someone complains. And confidence is only useful if it is calibrated. A raw model self-report is not a probability; it is a token distribution that correlates loosely with correctness and confidently with fluency. Before you route on it, check it against outcomes on a labelled sample, which is the subject of the next section.

Calibrating the threshold from data rather than intuition

The threshold is usually picked in a meeting. Someone says 0.7 because 0.7 feels like a reasonable amount of confidence, and 0.7 ships. Nobody revisits it, partly because there is no obvious moment to revisit it and partly because there is no obvious thing to compare it against.

There is a better method and it is not expensive. Take a sample of a few hundred completed agent actions from production — enough to cover your traffic mix, drawn across a full week so you catch weekday and weekend shapes. Have a competent human label each one for the thing you actually care about: was this action correct and appropriate to ship? Now you have ground truth, and you can treat the escalation decision itself as a binary classifier, evaluated the way you would evaluate any other classifier.

That reframing is the whole trick. Your trigger is not a policy, it is a model, and it has precision and recall. Recall on the escalation decision is your catch rate: of the actions that genuinely needed a human, what fraction did the trigger route to one? Precision is the inverse of your reviewer's boredom: of the actions the trigger escalated, what fraction genuinely needed a human? Approval rate is precision wearing a different hat — if ninety-six per cent of escalated items are approved, precision on the escalation decision is around four per cent, and you have built a classifier that would be laughed out of any model review.

Sweep the threshold across the labelled sample and you get the trade-off curve directly. A tight threshold gives you high recall and dreadful precision: you catch nearly everything and flood the queue, and within a fortnight the flood destroys the catch rate anyway because nobody is really reading. A loose threshold gives you high precision and poor recall: a small, engaged queue and a class of harms sailing past unexamined. The right point is not the maximum of any single number. It is the loosest threshold whose recall you can live with, given a queue your team can actually staff — which means the answer depends on your headcount, which means you cannot pick it before you have done the capacity arithmetic further down this page.

Two practical notes. First, calibrate per action class rather than globally. The right confidence cut for "draft a reply" and for "issue a credit note" have no reason to be the same number, and forcing them to be is how a single threshold ends up wrong for both. Second, expect to move the threshold as the agent improves. That is the healthy case, and the section on closing the loop below is about making it happen on purpose. If you want the fuller treatment of measuring the agent itself rather than the queue, evaluating agents on trajectory, tool calls and outcome is the companion piece.

Routing: tiers, expertise and load

Once you have decided what escalates, you have to decide where it goes. A single flat queue is the default and it is wrong for any system with more than one kind of risk in it, because it forces one SLA, one skill profile and one priority order onto items that deserve three of each.

Tiered escalation is the standard answer: tier-1 actions go to front-line reviewers on fast SLAs, and tier-3 actions go to specialists. The tier is a function of the risk signals you already computed for the trigger, so it costs almost nothing to add once the trigger is right. What it buys is the ability to give the fast-moving majority a fifteen-minute service level without pretending that a five-figure payment approval can be done in fifteen minutes by whoever is nearest.

The second axis is expertise. Reviewer routing by expertise is one of the three design decisions that most reduces approval fatigue, and it works for an unglamorous reason: a reviewer who understands the domain reaches a confident decision faster and is less likely to default to approve out of uncertainty. Routing a chargeback dispute to someone who has seen four hundred chargebacks is not just more accurate, it is measurably quicker, which feeds straight back into your capacity arithmetic.

A four-tier routing design. Queue-share figures are illustrative starting points, not measured results — replace them with yours within the first month.

Tier	What lands here	Who reviews it	Target SLA	Share of escalations	Timeout default
Tier 0	High confidence, reversible, low value, familiar shape	Nobody — auto-approve, sampled for audit	n/a	Not escalated at all	n/a
Tier 1	Borderline confidence, reversible, below the value threshold	Front-line ops reviewer, any domain	15 minutes	~70%	Fail-open with mandatory audit sample
Tier 2	Domain-specific judgement: billing, fraud, clinical, contractual	Domain reviewer with the relevant queue skill	2 hours	~25%	Escalate up to tier 3 with a fresh clock
Tier 3	Irreversible, high value, novel, or regulatory-sensitive	Named specialist or duty manager	8 business hours	~5%	No timeout — page a human and keep paging

The third axis is load, and it is the one that gets skipped. Automated reviewer assignment — the thing mature annotation and review platforms do as a matter of course — routes cases to available reviewers, tracks progress, and measures inter-reviewer agreement so that review scales with production volume instead of relying on whoever happens to be watching the channel. Without it you get the pattern every ops lead recognises: three reviewers with nothing in front of them and one with forty items, because assignment is happening by proximity to the notification rather than by capacity.

One clarification worth making, because the two get conflated. This is agent-to-human escalation. Agent-to-agent transfer is a different problem with a different contract — what state crosses, who owns termination, how you avoid an infinite loop — and we covered it separately in agent handoff contracts and state transfer. The confusion matters because teams sometimes solve the agent-to-agent problem, declare human-in-the-loop done, and discover the queue only when it is already full. An agent-to-agent handoff needs a contract; an agent-to-human handoff needs a contract and a rota.

The reviewer's screen is the product

Everything above determines what reaches a human. This section determines whether the human's attention is worth anything when it gets there, and it is the part that engineering teams systematically underinvest in because it looks like an internal tool.

Review context matters more than almost anything else in the design. Give reviewers a clean diff and a plain-language summary and you get considered decisions in a couple of minutes. Dump the raw JSON payload the agent produced and you get one of two outcomes: rubber-stamping, because reading it properly is harder than clicking approve, or very slow approvals, because the conscientious reviewers do read it properly and it takes them nine minutes an item. Most teams get both at once, from different reviewers, which also destroys inter-reviewer agreement.

A good review card contains six things and almost nothing else.

1. The proposed action as a clean diff. Before and after, field by field, with the changed values highlighted. If the action is a message, show the message as it will appear, not as a serialised object.
2. A plain-language summary in one or two sentences. "Refund £180 to order 44812 because the customer reports the item arrived damaged and the courier record shows a failed delivery scan." A reviewer should be able to decide most items from this line alone and use the rest to confirm.
3. The evidence the agent used. The specific retrieved documents, records or tool outputs the decision rests on — linked, quoted at the relevant passage, and bounded. Not the full context window.
4. The confidence, and why this escalated. Name the trigger that fired. "Escalated because: amount above £150 threshold; novelty score 0.81." A reviewer who knows why an item is in front of them reviews the right thing.
5. A two-click decision. Approve, reject, or edit-and-approve. Anything requiring a third click at scale is a design defect.
6. A rejection reason, captured as structured data. A short picklist of causes plus a free-text field, mandatory on reject. This is the single highest-value field on the card and the one most often left out.

That last point deserves the emphasis. Your rejection reasons are your next eval set. Every reject is a labelled failure, produced by a domain expert, at the moment of maximum context, on real production traffic — which is precisely the data you would otherwise pay an annotation vendor to manufacture. Cluster a quarter's worth and you get a ranked list of your agent's actual failure modes, which beats any synthetic eval suite you could write from imagination. The methods for turning that pile into something usable are in error analysis on production logs, and the mechanics of running review as a repeatable labelling operation — assignment, agreement, golden sets — are in building a human annotation pipeline.

SLAs, timeouts, and what happens when nothing happens

Here is the hardest design question in the whole area, and the one most teams answer by accident: what happens when the SLA expires and nobody has decided?

The question cannot be dodged, because "the item stays in the queue forever" is itself an answer, and it is usually the worst one. Unresolved queues do not stay unresolved. They accumulate until someone with a Friday afternoon free clears two hundred items in twenty minutes, which converts your careful oversight design into a rubber stamp retroactively and in bulk. SLA-enforced timeouts are one of the three levers that most reduce approval fatigue precisely because they stop this happening — an item leaves the queue by policy, with a recorded reason, instead of ageing quietly into a batch nobody reads.

There are exactly three defensible behaviours on expiry, and the mistake is picking one of them for the whole queue.

Choosing the timeout default by blast radius. The wrong answer is a single policy applied to every item in the queue.

Action	Reversible?	Cost of delay	Correct default on expiry
Draft a reply for a human to send later	Yes, entirely	Low	Fail-open — auto-approve, sample for audit
Refund under £250 / ₹25,000 to a verified account	Yes, clawback exists	High — delay is a customer-experience defect	Fail-open with a hard cap and post-hoc audit
Close or re-categorise a support ticket	Yes	Moderate	Fail-open
Send an outbound message to a customer	No — you cannot unsend	Moderate	Fail-closed — auto-reject and re-queue for the next shift
Change a production configuration or deploy	Partially, with effort	Varies wildly	Escalate up — never auto-approve, never silently drop
Delete or export customer data	No	Low — waiting is nearly free	Fail-closed, with no override path in the tooling
Pay an invoice above £5,000 / ₹5,00,000	No, in practice	Low	No timeout at all — page a named approver until answered
Grant a permission, role or credential	Technically yes, practically no	Low	Fail-closed

Three rules make this work in practice. Set the default in configuration rather than in code, so that changing it is a reviewable act rather than a deploy. Log every timeout decision as a decision, with decided_by: timeout attached, so it appears in your audit trail as what it is. And count timeout decisions separately in every metric you report — an approval rate that silently includes auto-approvals on expiry is not measuring human judgement at all, and a team can look healthy on that number while its actual human review has stopped functioning.

Set the SLA itself from the cost of delay rather than from ambition. A fifteen-minute tier-1 SLA is meaningful if your rota can hold it and a fiction if it cannot, and a fiction is worse than a longer honest number, because it makes your breach-rate metric meaningless and trains everyone to ignore the alerts. When the queue does back up badly enough to affect customers, treat it as an incident with a runbook rather than as an ops annoyance — incident response runbooks for LLM systems covers the shape of that playbook.

Capacity planning, with the arithmetic

This is the section that justifies the title. Queue depth and tier boundaries have to be measured and adjusted as the backlog grows, which makes human-in-the-loop design capacity planning and not only safety design. The arithmetic is not hard. It is simply never done, because the person who writes the trigger and the person who owns the ops rota are usually not the same person and often not in the same meeting.

Work a concrete example. An agent handling customer operations for a mid-sized business takes 12,000 actions per day. Median hands-on review time for a tier-1 item is 4 minutes — that is time on the card, not elapsed time. A reviewer works an eight-hour shift, which is 480 minutes, but nobody delivers 480 minutes of sustained review; between breaks, handovers, meetings, context switching and the hard items that take four times the median, 60 per cent is an honest planning ceiling, giving 288 effective review minutes per reviewer per day. Use your own number once you have measured it; use this one until you have.

Reviewers required at 12,000 actions/day, 4 minutes per review, 288 effective review minutes per reviewer per day.

Escalation rate	Reviews per day	Demand (reviewer-minutes)	Reviewers (raw)	Reviewers (headcount)
3%	360	1,440	5.00	5
5%	600	2,400	8.33	9
8%	960	3,840	13.33	14
12%	1,440	5,760	20.00	20
15%	1,800	7,200	25.00	25
20%	2,400	9,600	33.33	34

Read the 5 per cent and 15 per cent rows together, because that comparison is the argument. A ten-point difference in escalation rate — the difference between a well-calibrated risk trigger and a lazy category trigger on the same agent doing the same work — is the difference between 9 reviewers and 25. At an illustrative fully-loaded cost of £45,000 per reviewer per year in the UK, that is £405,000 against £1,125,000: a £720,000 annual gap created by a threshold somebody chose in a meeting. Run the same arithmetic on an India-based ops team at an illustrative fully-loaded ₹9,00,000 per reviewer per year and it is ₹81 lakh against ₹2.25 crore, a gap of ₹1.44 crore. Substitute your own salary bands; the ratio between the rows is the durable part.

Now the part that surprises people. Staffing to the mean does not work, because queues are not linear near saturation. Suppose you staffed for 8 per cent: 14 reviewers, 4,032 reviewer-minutes a day, a service capacity of 1,008 reviews per day. Watch what a small drift in escalation rate does.

Backlog growth against a fixed team of 14 reviewers (capacity 1,008 reviews/day). A two-point drift in escalation rate is not a two-point problem.

Escalation rate	Reviews arriving per day	Utilisation (arrivals ÷ capacity)	Backlog after 20 working days	Stable?
6.0%	720	0.71	0	Yes — comfortable
7.0%	840	0.83	0	Yes
8.0%	960	0.95	0	Technically — but p95 wait is already unacceptable
8.5%	1,020	1.01	240	No
9.0%	1,080	1.07	1,440	No
10.0%	1,200	1.19	3,840	No

An escalation rate moving from 8 per cent to 10 per cent is a two-point change and a twenty-five per cent relative one. It takes the queue from a zero backlog to 3,840 items in a month — nearly four full days of the entire team's output, and still growing every day, because once arrivals exceed service the deficit compounds and never self-corrects. Nothing recovers a queue in that state except adding people, tightening the trigger, or timing items out. Hoping is not on the list.

The 8 per cent row carries the second lesson. At 0.95 utilisation the queue is stable in the textbook sense — arrivals are below capacity — and it is already a bad place to work. Waiting time grows with utilisation as roughly ρ / (1 − ρ), so a team at 95 per cent utilisation waits about nineteen times as long as the same team at 50 per cent. On a pooled tier-1 queue those absolute waits stay small enough to hide the effect, which is why the problem usually surfaces first on a tier-3 queue where one specialist is the whole server.

Steady-state queue wait for a single tier-3 specialist, 25 minutes per review. Standard M/M/1 approximation: real arrivals are burstier, so treat these as optimistic.

Utilisation	Wait multiplier ρ/(1−ρ)	Expected wait in queue	Can this hold an 8-hour SLA?
50%	1.0×	25 minutes	Comfortably
70%	2.3×	58 minutes	Yes
80%	4.0×	100 minutes	Yes, with little slack
90%	9.0×	3 hours 45 minutes	Only on a good day
95%	19.0×	7 hours 55 minutes	No — the mean already breaches
98%	49.0×	20 hours 25 minutes	No

Here is the whole calculation as runnable code. Change the four constants at the top and it answers both questions for your system.

import math

# --- Inputs: replace with your own measured numbers ---
ACTIONS_PER_DAY     = 12_000   # agent actions that could escalate
MINUTES_PER_REVIEW  = 4.0      # MEDIAN hands-on time for a tier-1 review
SHIFT_MINUTES       = 480      # an 8-hour shift
UTILISATION_CEILING = 0.60     # sustained review as a share of the shift

REVIEWER_MINUTES_PER_DAY = SHIFT_MINUTES * UTILISATION_CEILING   # = 288.0


def reviewers_required(escalation_rate,
                       actions=ACTIONS_PER_DAY,
                       minutes=MINUTES_PER_REVIEW,
                       capacity=REVIEWER_MINUTES_PER_DAY):
    """Headcount needed to clear one day of escalations in one day."""
    reviews = actions * escalation_rate
    demand_minutes = reviews * minutes
    raw = demand_minutes / capacity
    # round() first: 12000 * 0.12 is 1440.0000000000002 in binary floating
    # point, and ceil() on the resulting 20.000000000000004 would say 21.
    return reviews, demand_minutes, raw, math.ceil(round(raw, 6))


def steady_state_wait(service_minutes, utilisation):
    """M/M/1 approximation of time spent waiting. Diverges as rho -> 1."""
    if utilisation >= 1.0:
        return float("inf")          # unstable: the backlog grows forever
    return service_minutes * utilisation / (1.0 - utilisation)


print("rate   reviews/day    demand(min)     raw   headcount")
for rate in (0.03, 0.05, 0.08, 0.12, 0.15, 0.20):
    reviews, demand, raw, heads = reviewers_required(rate)
    print(f"{rate:5.0%}  {reviews:11,.0f}  {demand:13,.0f}  {raw:6.2f}  {heads:10d}")

print()
print("tier-3 specialist, 25 min per review")
for rho in (0.50, 0.70, 0.80, 0.90, 0.95, 0.98):
    wait = steady_state_wait(25.0, rho)
    print(f"  utilisation {rho:4.0%}  ->  {wait:8.1f} min in queue "
          f"({wait / 60:.1f} h)")

Two habits make this arithmetic useful rather than decorative. Recompute it monthly against measured escalation rate and measured median review time, both of which drift. And staff to your peak hour rather than your daily mean: if forty per cent of your actions arrive in a four-hour window, your effective utilisation during that window is roughly double the daily figure, and the daily figure is the one that looks fine on the dashboard.

Follow-the-sun across two markets

Reviewer economics differ enough between India and the UK that the arithmetic above often produces a different answer in each market, and the interesting design is not choosing one but combining them. A Bengaluru or Pune team working 09:00 to 18:00 IST covers 03:30 to 12:30 UTC. A London team working 09:00 to 17:00 covers roughly 08:00 to 16:00 UTC. Run both and you get 12.5 hours of continuous coverage with a 4.5-hour overlap — the overlap being where handover, calibration sessions and the genuinely ambiguous tier-2 items belong, because that is the only time both teams are awake.

The remaining 11.5 hours are where the timeout design earns its keep. That window is precisely when a blanket fail-open policy silently auto-approves a night's worth of actions, and when a blanket fail-closed policy builds a backlog that lands, entire, on the first shift to log in. Neither is acceptable as a default for everything, which is the case for the per-action table above. In practice most teams end up narrowing the trigger for the uncovered window — raising the auto-approve confidence bar so fewer items escalate overnight — rather than changing what happens to the items that do.

Two operational details that catch people out. Cost is not the only variable: an India-based tier-1 team is typically the right economic answer for high-volume reversible decisions, while tier-3 items that require a named accountable individual under a UK regulatory relationship usually need to sit with the UK entity regardless of what the spreadsheet says. And review tooling has a location. If the review card renders customer records, then a queue store in AWS Mumbai read by reviewers in London, or one in AWS London read by reviewers in Bengaluru, is a cross-border data question your ops rota just created without a design review. India's DPDP framework permits transfer except to countries the government specifically restricts — a negative list rather than a general bar, with its rules phasing in through to 2027 — so the binding constraint is usually your own customer contracts and sector regulator rather than the statute. Ask before you build the rota, not after.

The metrics that tell you whether review is working

A review queue that is not instrumented is a queue you will discover through a complaint. Eight numbers cover it, and only one of them tells you whether the review is catching anything.

Queue metrics, healthy ranges and what a bad value is telling you. Ranges are rules of thumb for a general-purpose ops queue, not measured benchmarks.

Metric	How to compute it	Healthy range	A bad value means
Escalation rate	Escalated ÷ total agent actions	2–10%, and falling quarter on quarter	Above 20%: category triggers. Below 1% with no audit stream: you are not measuring, not safe
Approval rate	Approved ÷ reviewed, excluding timeouts	60–85%	Above 90%: triggers too broad, rubber-stamp drift. Below 40%: an agent quality problem, not a queue problem
Time-to-decision, p50	Median queue entry to decision	Under one third of the SLA	Near the SLA: you are staffed to the mean with no headroom
Time-to-decision, p95	95th percentile, per tier	At or under the SLA	Above it: your utilisation is in the non-linear zone even if the mean looks fine
Queue depth	Open items, sampled every 15 minutes, per tier	Below one shift's throughput	Above it and rising for three days: structurally understaffed, not having a bad week
SLA breach rate	Breached ÷ decided	Under 2%	Above 10%: either a capacity gap or an SLA nobody ever believed
Inter-reviewer agreement	Agreement on a double-reviewed sample, chance-corrected	Substantial agreement — kappa above roughly 0.6	Low: ambiguous policy, not careless reviewers. Fix the guidance first
Post-approval defect rate	Defects later found in approved actions ÷ approved	Low and trending down	Flat or rising: review is theatre. This is the metric that says whether any of it works

Post-approval defect rate is the one to fight for, and the one teams skip because it requires a feedback path from downstream outcomes — complaints, reversals, chargebacks, corrections — back to the review record. Without it every other number on that table can look excellent while the queue catches nothing, because approval rate, time-to-decision and queue depth all measure the process and none of them measures the result. Emit all eight as spans and metrics on the same trace as the agent run that produced the action, so a single trace shows the plan, the tool call, the escalation, the wait and the decision; agent observability with OpenTelemetry covers the instrumentation.

Two of these deserve a dashboard of their own rather than a row in a table. Plot approval rate and escalation rate on the same time axis: the pathological pattern is approval rate climbing while escalation rate stays flat, which means the queue is filling with items that no longer need review and reviewers are learning to say yes. And alert on queue depth as a derivative rather than a level. Depth of 300 is meaningless without knowing whether it was 100 yesterday, and the day the derivative turns positive and stays positive is the day you still had options.

Closing the loop so the queue gets cheaper

A well-designed queue is a temporary structure at any given size. If your escalation rate is the same next year as this year, the queue is a tax you have agreed to pay indefinitely. If it falls, the queue is an investment that is paying back, and the mechanism by which it falls is the loop between what reviewers reject and what the agent does next time.

The loop has four destinations, and the discipline is routing each rejection to the right one instead of letting all of them pile into a single backlog ticket.

1. Into the eval set. Every rejection becomes a test case with the reviewer's reason as the expected failure. Cluster by reason, keep the clusters that recur, and you have a regression suite grounded in production rather than imagination. This is the highest-value destination and the one that keeps paying after the specific bug is fixed.
2. Into the prompt, the tool descriptions, or the retrieval layer. A cluster of rejections that all say "used the wrong policy document" is a retrieval problem. A cluster that says "correct decision, wrong tone" is a prompt problem. A cluster that says "called the refund tool when it should have called the credit-note tool" is a tool-description problem, and the fix is in the description rather than the model.
3. Into the trigger thresholds. A cluster of rejections concentrated in a band your trigger already auto-approves means the threshold is too loose in that band. A cluster of approvals concentrated in a band you always escalate means it is too tight, and that band can move to sampled audit. This is the direct route to a falling escalation rate.
4. Into the guidance, when reviewers disagree. If two reviewers reject the same item for different reasons, or one approves what another rejects, the fix is upstream of both: the policy is ambiguous. Rewrite the guidance, re-measure agreement, and only then look at the agent.

Give this a cadence rather than an intention. A weekly half-hour where an engineer and a senior reviewer read the past week's rejections together is enough, and it is more valuable than any dashboard, because the two of them together can tell in seconds which of the four destinations a cluster belongs to. Then hold the queue to a target: if escalation rate has not fallen in two consecutive quarters, the loop is not running, whatever the meeting invitation says.

One thing to be careful about, particularly as of September 2026 when oversight expectations are being written into regulation in several jurisdictions. Regulators — including the EU under its AI Act — expect meaningful human oversight of systems classed as high-risk, and the word carrying the weight there is meaningful. An oversight function that approves ninety-seven per cent of what it sees in eleven seconds is unlikely to satisfy anyone who examines it closely, whatever the design document claims. Nothing in this guide is legal advice and no regulation prescribes a queue design; but the metrics in the previous section happen to be the same evidence you would want if someone asked you to demonstrate that your oversight was real. Build them because they make the system work, and accept the second benefit gratefully.

Common pitfalls, and where to start

Treating the queue as an internal tool. It is the interface through which every risky decision in your product passes. It deserves the design attention you would give a customer-facing screen, and it will repay it faster.

One SLA for everything. A single service level forces you to choose between a tier-1 number that makes tier-3 impossible and a tier-3 number that makes tier-1 useless. Tier first, then set an SLA per tier.

Blocking the agent while it waits. Park the action, keep planning. A blocking design turns every reviewer coffee break into user-visible latency and makes a queue problem into a product problem.

Rejections without reasons. The most expensive omission in this article. You paid a domain expert to identify a failure and then discarded the label.

Alerting on queue depth as a level. Depth without a trend tells you nothing. The derivative is the signal, and it turns positive days before the level looks alarming.

Assuming more reviewers fixes it. Sometimes true, often not. If approval rate is above ninety per cent, adding people scales the rubber stamp. Fix the trigger first, then hire against the arithmetic.

Forgetting that agents get better. A trigger calibrated against last quarter's agent is mis-calibrated against this quarter's, and always in the direction of escalating too much. Recalibrate on a schedule.

If you are starting from a queue that already exists, four measurements in the first week will tell you nearly everything. Approval rate excluding timeouts — over ninety per cent and the trigger is your problem. Median hands-on review time, which is the input your capacity plan has probably been guessing at. Queue depth sampled every fifteen minutes for five days, so you can see the derivative and the peak-hour shape. And a hand-labelled sample of a hundred auto-approved actions, which is the only way to find out what your current threshold is letting past. Then run the arithmetic in the code block above with your real numbers, and you will know within an afternoon whether you are understaffed, over-triggered, or both — which is a far better position than the one most teams are in, where the queue is a component nobody owns and everybody assumes is fine. Escalation is throughput. Design it that way from the start and the safety property follows; design it as a checkbox and you will get the checkbox and nothing else. If you want the wider context on why any of this matters, our reporting on the rise in agent security incidents is a reasonable place to start.