LLM Incident Response: Runbooks for Hallucination Spikes, Outages and Rollbacks

What you need to know

• AI incidents are their own category. A hallucination spike, a jailbreak or a regression after a model upgrade do not look like a classic 500-error page. Your existing on-call playbook will not catch them.
• Detection is a sampled scorer plus your SLOs. Grade a small slice of live traffic for quality, watch latency, error and cost against thresholds, and fold in user-report signals — tuned so the pager fires on real regressions, not noise.
• Severity drives everything. SEV1 to SEV3 decides who is paged, how fast, and how loudly you communicate. Write worked examples for each level so nobody improvises at 2am.
• The levers are failover, kill-switches, rollback, rate-limits, safe mode and cached fallbacks. Each incident type has a first mitigation and a rollback path — decided in advance, not invented mid-incident.
• No rollback path means no plan. If you cannot revert a prompt or pin a previous model in minutes with a config change, build that before anything else.
• Close the loop. A blameless postmortem that ends with the failing case added to your eval suite is the only thing that stops the same incident recurring.

This guide is the on-call companion to the two pieces that sit either side of it. Our shadow and canary deploys playbook is how you stop most incidents before they reach everyone; our production drift-detection guide is how you notice slow-burning quality decay over weeks. This one is narrower and more uncomfortable: the model is misbehaving in production right now, real users are affected, and someone has to act. What does that person do, in what order, and how do they get you back to a known-good state?

Why AI incidents are not like classic web incidents

A traditional web incident announces itself. The service returns a 500, the error-rate graph turns red, the pager fires, and the shape of the problem is legible within minutes. AI incidents are quieter and stranger. The service is up. Latency is fine. Every request returns a 200. And yet the product is broken, because the model has started confidently inventing facts, or refusing half the queries it used to answer, or looping through the same tool call forever. The classic golden signals — latency, traffic, errors, saturation — are necessary but nowhere near sufficient. You need a taxonomy built for the failure modes that are specific to language models.

There are six that recur often enough to deserve their own runbook entry:

• Hallucination or quality spike. The model's outputs stay well-formed but their truth or usefulness collapses — ungrounded claims, wrong citations, off-topic answers. Nothing in your HTTP metrics moves. Only a content-aware scorer or a rise in user complaints reveals it.
• Provider outage or latency blowout. The most classic-looking of the six, but with a twist: a provider rarely goes fully dark. It degrades — p95 latency triples, a fraction of requests time out, a region wobbles — and your users feel it long before a status page admits anything.
• Cost spike. Unique to metered inference. A retry storm, a prompt-injection that inflates context, a runaway agent loop or an accidental switch to a pricier model can multiply your bill in hours. There is no error to catch; the only symptom is money.
• Safety, jailbreak or prompt-injection breach. A user or a poisoned document convinces the model to ignore its instructions, exfiltrate context, or produce disallowed content. This is both a quality incident and a security one, and it is often discovered by a user screenshot rather than a graph.
• Tool-call failure loop. An agent calls a tool, misreads the result, calls it again, and never converges. Each individual call succeeds, so error rates look healthy while latency, token spend and user frustration all climb together.
• Quality regression after a model upgrade. The provider rotates a model behind a floating alias, or you ship a new prompt, and a behaviour your product depended on silently changes. The deploy looks green; the regression only shows up in outputs.

Notice what unites them: in five of the six, your infrastructure is perfectly healthy. That is precisely why a generic incident process misses them, and why AI features need detection and runbooks of their own.

Detection: make the incident page you before your users do

You cannot run a runbook for an incident you have not noticed. Detection for AI features layers four kinds of signal, each catching failures the others miss.

Online scorers and evals on live traffic. The defining move of AI observability is grading a sample of production output as it happens. You do not need to score every request — a sampled 1 to 5 per cent is enough to see a trend — and you do not need an expensive judge. Score for the things that actually break: groundedness against retrieved context, refusal rate, and structural validity such as whether the JSON parses or a citation resolves to a real source. Alert on the moving average over a window, never on a single call, because individual bad answers are background noise. When a scorer does fire, the offending example is also your best raw material for a postmortem — the same discipline our guide to building evals from production logs is built on.

Canary metrics. If you route a slice of traffic to a new model or prompt behind a canary, the delta between canary and baseline is one of the earliest and cleanest incident signals you have. A canary whose quality score or cost-per-request diverges from baseline is an incident caught before full rollout — which is why canarying and incident response are two halves of the same system.

Cost, latency and error SLOs. The metered, classic signals still matter. Set an SLO on p95 latency and on provider error rate, and — because inference is billed by the token — a hard alert on cost run-rate. A cost alert is not a nice-to-have; it is often the only symptom of a retry storm or a runaway agent. Wire it to page, not just to email a dashboard nobody reads.

User-report signals. Thumbs-down, "report", "regenerate" and support tickets are lagging but unambiguous. A sudden rise in the thumbs-down rate is a real regression by definition — a user told you so. Treat a step-change in negative feedback as a first-class detector, not an afterthought.

Here is a compact alerting rule that ties two of these signals to severity. The scorer alert watches groundedness; the latency alert watches the p95 SLO. The thresholds and windows are the load-bearing part — they are what page you on a real regression without drowning you in false alarms.

groups:
  - name: llm-quality-and-latency
    rules:
      - alert: HallucinationSpike
        # online groundedness scorer, sampled on ~5% of live traffic
        expr: avg_over_time(llm_groundedness_score[10m]) < 0.85
        for: 10m
        labels:
          severity: sev2
          team: ml-oncall
        annotations:
          summary: "Groundedness under 0.85 for 10m — possible hallucination spike"
          runbook: "https://runbooks.internal/llm/hallucination-spike"

      - alert: LatencyBlowout
        expr: histogram_quantile(0.95, rate(llm_request_seconds_bucket[5m])) > 8
        for: 5m
        labels:
          severity: sev1
          team: ml-oncall
        annotations:
          summary: "p95 latency over 8s for 5m — provider degradation likely"
          runbook: "https://runbooks.internal/llm/latency-blowout"

The runbook: severity, ownership and comms

A runbook is not a wiki page nobody opens. It is a short, opinionated document that answers three questions before the incident happens: how bad is this, who owns it, and what do we do. Get severity right and the rest follows, because severity decides who is paged, how fast, and how loudly you tell people.

• SEV1 — user-facing outage or safety breach. The assistant is down, returning garbage to everyone, leaking data, or jailbroken. Page immediately, appoint an incident commander, open a dedicated channel, and communicate to users and stakeholders now. Example: an upgraded model starts returning empty or malformed responses to every request; or a prompt-injection is confirmed to be exfiltrating another tenant's context.
• SEV2 — serious degradation, not total. One flow is affected, or the service is slow, or the cost curve will breach budget within the day. Page the on-call, but you may not need a full commander. Example: groundedness on the support-answer flow drops below threshold for fifteen minutes; or p95 latency doubles because a provider region is congested.
• SEV3 — contained or slow-burning. A single tenant's tool integration is failing, a minor quality dip, a non-urgent regression to schedule. Ticket it, fix it in hours, no war room. Example: one customer's webhook tool times out; or a small rise in thumbs-down on a low-traffic feature.

Ownership. Name a rota, not a hero. A single overloaded engineer who "knows the model" is a bus-factor risk; a documented on-call rota with a clear escalation path is resilience. The on-call classifies, mitigates and communicates; they escalate to the incident commander for a SEV1 and to the owning team for anything they cannot resolve inside the target time.

Comms. Decide the templates in advance. A SEV1 gets an external status-page update and an internal channel; a SEV2 gets an internal heads-up; a SEV3 gets a ticket. The point of pre-writing them is that nobody should be composing careful customer-facing prose while the fire is still burning.

The heart of the runbook is a single table mapping each incident type to its detection signal, its first mitigation, its rollback path and its owner. This is what the on-call reads at 2am. Everything above is context; this is the operational core.

Two columns in that table are worth dwelling on, because they are where most teams are weakest: the mitigations, and the rollback paths. They deserve their own section.

The levers you actually pull

An incident is only survivable if you have levers ready to pull. Building them is not incident work — it is the quiet infrastructure you put in place beforehand, so that under pressure the on-call is choosing between prepared options rather than improvising. There are six that cover almost every AI incident.

Provider failover and model routing. When a provider degrades, you re-route to a backup provider or region. This is the job of a gateway that sits in front of your models and turns an outage into a config decision — the pattern our resilient LLM gateway guide is entirely about. If you have one, failover is a line of config; if you do not, a provider outage is your outage.

Feature flags and a kill-switch. The ability to disable an agent, a tool or an entire AI feature without a deploy is the difference between a two-minute mitigation and a two-hour one. A kill-switch flips a flag; the next request reads the new state and takes the safe path. This is what you reach for during a jailbreak or a tool-call loop, when the right move is to turn the misbehaving thing off entirely.

Rollback to a previous prompt or model version. The most common cause of an AI incident is a change you made. If a new prompt or a model upgrade regresses, the fastest fix is to go back — which only works if your prompts are versioned and your model strings are pinned rather than floating on an alias. This lever is the direct payoff of the shadow and canary discipline: the same versioning that lets you canary a change safely is what lets you revert it instantly.

Rate-limiting. During a cost spike or a retry storm, capping the request or token rate on the offending route contains the blast radius while you find the root cause. It is a tourniquet, not a cure, but it stops the bleeding while you work.

A degraded safe mode. Sometimes the right answer is not the best answer but a safe one. A safe-mode reply — a template response, a "we're experiencing high demand" message, or an answer generated with tools disabled and a conservative model — keeps the product usable and honest while the real fix lands. Degrading gracefully beats failing loudly.

Cached fallbacks. For high-traffic, low-variance queries, a cache of known-good previous answers lets you keep serving during a total provider outage. It will not cover the long tail, but it keeps the common path alive when the model is unreachable.

Here is a compact kill-switch that ties several of these levers together. It is read at request time from a fast store — a Redis key or a flags service refreshed every few seconds — so the on-call can change behaviour without shipping code.

# feature_flags.py — levers the on-call flips without a deploy.
# Flags are read from Redis and refreshed every ~5s.

def answer(query, ctx):
    flags = get_flags()  # cached, hot-reloaded

    # Kill-switch: disable tools / the whole agent under a jailbreak or loop
    if flags.get("agent.tools.enabled") is False:
        return safe_mode_reply(query)          # degraded, no tool calls

    # Rollback: pin a previous known-good model if one is set
    model = flags.get("model.rollback_to") or flags.get("model.primary")

    # Rate-limit lever: contain a cost spike on this route
    if over_rate_limit(ctx.route, flags.get("route.max_rps", 50)):
        return cached_or_safe(query)           # cached fallback, else safe mode

    try:
        return call_model(model, query, timeout=8)
    except ProviderError:
        backup = flags.get("model.backup")     # failover to another provider
        if backup:
            return call_model(backup, query, timeout=8)
        return cached_or_safe(query)

Dual-market realities: regions and data residency

Two constraints matter more for teams shipping across India and the UK than they do for a single-region product, and both bite hardest during an incident.

Region and provider outages are geographic. When an AWS Mumbai region has a bad hour, an Indian SaaS serving local users feels it immediately; when a London region wobbles, a UK fintech's regulated workload is affected. A provider outage is rarely global — it is a region, or an availability zone, or a single provider's incident in one geography. Your runbook's failover lever therefore needs a regional dimension: not just "fall over to another provider", but "fall over to another region", with the data-residency implications thought through in advance. Keep an eye on the provider's own status page during the incident, because your probes and theirs will not always agree — and the status page is the reference the rest of the organisation will be watching too.

Do not log PII while capturing the incident. The instinct during an incident is to grab everything — the full prompt, the retrieved context, the raw output — and paste it into a ticket or a shared channel so the team can debug. That instinct leaks customer data. The prompt and context routinely contain personal information, and an incident channel is one of the least controlled places it can end up. Redact before you persist: run the same PII-redaction pass you already use in your pipeline — the one our PII redaction for RAG pipelines guide walks through — over anything that lands in a ticket, a trace or a message, and store a stable reference ID instead of the raw text. Respect residency: an incident in a London region should be debugged with its logs kept in-region, not pulled to a console in another jurisdiction, and the same holds for an India region. You can capture everything you need to reproduce the failure — which chunk was retrieved, which tool failed, which model version served the call, the scorer value — without capturing a single customer's name.

The blameless postmortem: close the loop

An incident you survive but do not learn from will happen again. The postmortem is where survival turns into improvement, and it has to be blameless — focused on the system and the process, not on the individual who shipped the change or acknowledged the page. People are honest about what actually happened only when they are not being set up to take the fall, and honesty is the whole point.

A good AI postmortem answers the usual questions — timeline, impact, what we saw, what we did, what slowed us down — and then does one thing that is specific to AI systems and non-negotiable: it captures the failing case into the eval suite. The exact input, context and expected-versus-actual output that broke in production becomes a permanent test. This is the mechanism that stops regression. A hallucination you fixed by hand but never encoded as an eval will drift back the next time you change a prompt or upgrade a model. A jailbreak you patched but never added to a safety eval will be re-opened by the next model version. The eval suite is your ratchet: every incident tightens it one notch, and the class of failure that caused the incident can no longer silently return. This is the same production-to-eval flywheel our drift-detection guide relies on, viewed from the incident side.

The second output of the postmortem is a guardrail update. If a prompt-injection got through, the input filter or the system-prompt defence is strengthened and the strengthened version is itself added to the eval set. If a tool looped, the max-step cap and the loop-termination check are the fix, and a test that would have caught the loop goes in. Guardrails and evals grow together, incident by incident, until the common failure modes are all fenced off.

Track two numbers over time as your honest scorecard: how long it took to detect each incident, and how long to mitigate it. If those numbers are falling, your detection and your levers are improving. If they are flat, you are firefighting the same fires repeatedly and the loop is not closing. The goal is not zero incidents — that is not achievable with a probabilistic system — but a steadily shrinking time-to-detect, a steadily shrinking time-to-mitigate, and an eval suite that grows a little more comprehensive after every single one.

Where to start

1. Build a rollback path first. Version your prompts, pin exact model strings, and wire both to a flag the on-call can flip in minutes. Without this, nothing else is an incident plan.
2. Write the six-type taxonomy and the runbook table. Map each incident type to a detection signal, a first mitigation, a rollback path and an owner. Keep it to one page.
3. Add detection in layers. A sampled online scorer, latency and error SLOs, a cost run-rate alert, and user-report signals — each with a sustained-window threshold tuned against real traffic.
4. Assign severity and a rota. Define SEV1 to SEV3 with worked examples, name an on-call rota and an escalation path, and pre-write the comms templates.
5. Rehearse and then close every loop. Run a game-day against a simulated hallucination spike or provider outage, and make "failing case added to the eval suite" the definition-of-done for every postmortem.

For the foundational discipline, the incident-management chapter of Google's SRE book is the canonical reference: read it at sre.google/sre-book/managing-incidents. For the provider-outage half of the runbook, keep a bookmark to your providers' and cloud regions' status pages — for example the AWS Health status dashboard — because during a regional incident that page is the shared source of truth the rest of your organisation will be watching too.