Build a Resilient LLM Gateway: Failover, Retries and Rate-Limit Handling (2026)

What you need to know

• One provider is a single point of failure. Every provider returns 429s, 5xx errors and timeouts, deprecates models and has regional outages. If your product calls one API directly, your uptime is capped by theirs.
• A gateway fixes this structurally. Put a proxy in front of OpenAI, Anthropic, Google and your own self-hosted vLLM, expose one OpenAI-compatible API, and you get failover, load-balancing, central key management, budget caps and observability in one place.
• Three layers of resilience. Retry the same model with backoff; fall back to the next model in a chain; and circuit-break a model into cooldown when it keeps failing — all driven by a clear error taxonomy.
• Resilience costs latency. Fallbacks add roughly 200 to 500 ms to p95 because retries are serial. Cap the blast radius with a short per-attempt timeout and a low retry count on latency-sensitive paths.
• Two off-the-shelf options, or roll your own. LiteLLM is the open-source proxy you self-host; OpenRouter is the hosted aggregator; a thin wrapper is fine for one service with one provider.

This guide is about the resilience layer that sits between your application and the model providers. It is a companion to three pieces you should read alongside it: our vLLM production playbook covers running the self-hosted backend that often sits behind the gateway as a fallback; our streaming at scale guide covers what happens once a request is flowing; and our cache, route and compress playbook covers the cost levers that routing decisions also influence. Here the question is narrower and more uncomfortable: given that a provider will fail mid-request, how do you make sure your users never see it?

Why one provider is a single point of failure

It is tempting to point your application straight at one provider's SDK and move on. It works in development, it works in the demo, and then it meets production traffic. The failures are not exotic — they are the ordinary weather of calling a shared, rate-limited, rapidly-changing remote service.

You will hit 429 rate limits when your traffic spikes or you share a tier with noisy neighbours. You will hit 5xx errors when the provider has an internal incident. You will hit timeouts when a region is congested or a long generation stalls. You will hit regional outages when an entire availability zone goes dark. And you will hit model deprecations when a provider sunsets the exact model string your code has hard-coded, often with weeks rather than months of notice. None of these are rare; collectively they are close to a certainty over any meaningful window.

A gateway — a proxy or aggregator that sits in front of multiple providers and presents one unified, OpenAI-compatible API — converts each of these from an outage into a routing decision. Instead of your application code knowing about OpenAI and Anthropic and Google and your self-hosted vLLM box, it knows about one endpoint. Behind that endpoint, the gateway handles failover between providers, load-balancing across deployments, central key management, per-key budget caps, and the logging and traces you need to actually understand what is happening. Your application gets simpler; your resilience gets dramatically better.

Consider a concrete dual-market shape. An Indian SaaS startup serving customers from an AWS Mumbai region wants its primary inference close to users for latency, but it cannot afford to go down when that region has a bad hour. A UK fintech, meanwhile, must keep regulated workloads inside a London region for data residency, but still wants a fallback when its primary provider rate-limits it at quarter-end. In both cases the gateway is the same idea: a default deployment plus a fallback chain, so that routing between an AWS Mumbai primary and a London secondary — or between a hosted provider and a self-hosted vLLM box — is a config line, not a code change.

The three layers of resilience: retry, fallback, cooldown

Resilience in a gateway is not one mechanism, it is three, applied in order. Get the order and the boundaries right and the system degrades gracefully; get them wrong and you either give up too early or hammer a dying provider into the ground.

Layer 1 — retry the same model with backoff

The first response to a transient failure is to try the same model again. A 429, a flaky 5xx or a timeout is often gone by the next attempt a moment later. The discipline here is twofold: bound the number of attempts with num_retries (set it to at least 3 on non-latency-critical paths), and space the attempts with exponential backoff so you do not retry into the same congestion that caused the failure. A retry_after_timeout of around 10 seconds gives the provider room to recover, and you should respect any Retry-After hint the provider sends back. LiteLLM supports both fixed and exponential backoff; prefer exponential, because fixed delays synchronise every client into the same retry wave.

Layer 2 — fall back to the next model in the chain

When retries on the primary are exhausted, the request should not fail — it should fail over. A fallbacks chain says: if the primary deployment cannot serve this request, try the next one, then the one after that. In LiteLLM's router model, failover and priority routing work together. When a request to a high-priority deployment fails — a connection error, a 404, a 429 or a 5xx — the router automatically tries lower-priority deployments. Each priority level gets its own set of retries before the router escalates to the next level, and exclusions accumulate across hops: a deployment that just failed is never re-picked within the same request chain. That last property is what stops the router from circling back to the broken provider it just abandoned.

Layer 3 — cooldown the model that keeps failing

Retries and fallbacks handle a single bad request. The circuit-breaker handles a bad provider. If a model fails more than allowed_fails times within a window, the gateway puts it into cooldown — it is skipped entirely for a while, so that every subsequent request fails over immediately instead of wasting a retry budget on a provider that is clearly down. This is the difference between a five-minute blip and a five-minute outage: without cooldown, every request keeps paying the full retry-then-fallback tax; with it, the broken provider is taken out of rotation until it has had time to recover.

In production this state cannot live in one process's memory. If you run several gateway instances behind a load balancer — which you should, for the gateway's own resilience — each instance would otherwise learn about a failing provider independently and slowly. Share the cooldown state, and the tpm and rpm counters, in Redis, so that the moment any instance trips the breaker, every instance agrees the provider is out. The same shared store is what lets you enforce a global rate limit across instances rather than a per-instance one that overshoots your provider tier.

A LiteLLM-style router config you can copy

Here is a router configuration that wires all three layers together: two providers in the model list, per-model rpm and tpm limits as a token-bucket, a retry count, a fallback chain, and a cooldown threshold. The < and > you would expect in YAML are not needed here, but note how the fallback chain names route across providers. Code stays in US English.

# litellm_config.yaml — one OpenAI-compatible endpoint in front of
# two providers, with retries, fallbacks and cooldown configured.
model_list:
  - model_name: chat-primary           # the name your app calls
    litellm_params:
      model: openai/gpt-4o-mini
      api_key: os.environ/OPENAI_API_KEY
      rpm: 500                          # token-bucket: requests per minute
      tpm: 200000                       # token-bucket: tokens per minute

  - model_name: chat-fallback           # second provider, same logical name space
    litellm_params:
      model: anthropic/claude-3-5-haiku-latest
      api_key: os.environ/ANTHROPIC_API_KEY
      rpm: 400
      tpm: 160000

  - model_name: chat-self-hosted        # your own vLLM box as the last resort
    litellm_params:
      model: openai/llama-3.3-70b        # OpenAI-compatible vLLM endpoint
      api_base: http://vllm.internal:8000/v1
      api_key: os.environ/VLLM_KEY

router_settings:
  num_retries: 3                        # retry the SAME model first...
  retry_after: 10                       # ...with a ~10s retry_after_timeout
  retry_policy:
    RateLimitErrorRetries: 3            # 429 -> retry + backoff before failover
    TimeoutErrorRetries: 2              # short timeouts -> fewer retries
  allowed_fails: 3                      # cooldown after 3 fails in the window
  cooldown_time: 60                     # seconds a tripped model is skipped
  redis_host: os.environ/REDIS_HOST     # share cooldown + tpm/rpm across instances
  redis_port: 6379

litellm_settings:
  # ...then fall over to the next model in the chain when retries are exhausted.
  fallbacks:
    - chat-primary: ["chat-fallback", "chat-self-hosted"]
    - chat-fallback: ["chat-self-hosted"]

The shape to internalise: retries are per-model, fallbacks are cross-model, and cooldown is the thing that decides which models are even eligible. Your application calls chat-primary and never needs to know that, under load, its request might be served by Anthropic or by your own vLLM box in a Mumbai or London region.

Calling the gateway: one unified request with a guarded fallback

Because the gateway exposes an OpenAI-compatible API, your application code is a single, ordinary call — the resilience is in the config above, not scattered through your services. That said, you will still want an application-level guard for the case where the gateway itself, or its entire fallback chain, is unreachable. Here is a unified call with a bounded retry-and-backoff wrapper around it.

# gateway_call.py — call the gateway through the OpenAI-compatible API.
# The gateway handles cross-provider failover internally; this wrapper
# only guards against the gateway/chain being wholly unreachable.
import time
from openai import OpenAI

# Point the OpenAI client at the gateway, not at a provider directly.
client = OpenAI(base_url="http://gateway.internal:4000/v1", api_key="sk-gateway")

def call_with_backoff(messages, max_retries=2, per_attempt_timeout=3.0):
    delay = 0.5
    last_err = None
    for attempt in range(max_retries + 1):
        try:
            return client.chat.completions.create(
                model="chat-primary",              # logical name; gateway routes
                messages=messages,
                timeout=per_attempt_timeout,        # short per-attempt timeout
            )
        except Exception as err:                    # gateway/chain unreachable
            last_err = err
            if attempt == max_retries:
                break
            time.sleep(delay)
            delay *= 2                              # exponential backoff
    raise RuntimeError(f"gateway unreachable after retries: {last_err}")

# On a latency-sensitive path, keep max_retries low (e.g. 2) and the
# per-attempt timeout short (2-3s) so a struggling chain is abandoned fast.

Mapping failure types to handling

The whole system is only as good as the rules that classify failures. Here is the taxonomy in one table — the contract the pro-tip above asked you to write down, made concrete.

Failure type	What it means	How the gateway handles it
429 rate limit	You have exceeded the provider's per-minute quota	Retry the same model with exponential backoff and a retry_after_timeout; if retries are exhausted, fall back to the next model in the chain
5xx server error	The provider has an internal incident	Fall back to the next provider; repeated 5xx within the window trips cooldown so the model is skipped
Timeout	A region is congested or a generation has stalled	Use a short per-attempt timeout (2–3 s) and move to the next provider rather than waiting; do not burn the whole latency budget on one stuck call
Model deprecation (404)	The model string you call no longer exists	Routing config change: point the logical name at a current model; the fallback chain absorbs the gap until you redeploy config
400 / 401 (bad request, auth)	Malformed request or invalid key	Fatal — surface immediately; never retry or fall over, because every provider will reject it the same way

LiteLLM vs OpenRouter vs rolling your own

There are two common off-the-shelf options and one DIY path. The right pick depends on how much infrastructure you want to own versus how much you want a third party to run for you.

Option	What you manage	Failover	Observability	Best for
LiteLLM (self-host proxy)	The box, the keys, the config, the Redis state	Full control — your own model list, retry, fallback and cooldown rules	Your own logs, traces and dashboards; nothing leaves your perimeter	Teams wanting control, data residency, or a self-hosted vLLM in the chain
OpenRouter (hosted aggregator)	Almost nothing — one API key, one billing relationship	Built-in across the providers it aggregates; less granular control	Provider-side dashboards; your traffic routes through a third party	Teams wanting the fastest integration and a single bill
Roll your own (thin wrapper)	Everything — every line of retry, fallback and logging logic	Whatever you build and maintain yourself	Whatever you instrument yourself	A single service with one provider, or a very specific routing need

For most teams the choice is between self-hosting LiteLLM — when control, observability or a self-hosted backend matters — and using OpenRouter for speed of integration. Rolling your own is defensible only when your needs are narrow or unusually specific; the moment you have more than one or two services calling LLMs, a shared gateway saves you from re-implementing the same retry-and-fallback logic in every codebase.

The latency cost of resilience — and how to cap it

Resilience is not free, and the bill is paid in latency. Because retries and fallbacks are serial — each failed attempt must complete or time out before the next begins — every hop you add lengthens the worst case. In practice, fallbacks increase p95 latency by roughly 200 to 500 ms. On a batch job or a background summariser that is irrelevant. On an interactive chat or an autocomplete, it is the difference between snappy and sluggish.

The lever is the blast radius. On latency-sensitive paths, cap the per-attempt timeout to 2 to 3 seconds and keep max_retries low — 2 is usually right — so a struggling provider is abandoned almost immediately and the request fails over before the user notices. On tolerant paths you can afford a longer timeout and three or more retries, because completing the request matters more than completing it fast. The mistake is using one global setting for both: a long, patient retry policy that is correct for a nightly job will quietly wreck the p95 of your chat endpoint.

The production design checklist

A gateway is more than retry logic. The features below are what separate a demo from something you can run a regulated UK fintech or a high-traffic Indian SaaS on. Treat this as the list to work through before you call the gateway production-ready.

• Health checks per deployment — actively probe each provider so the router knows what is alive before a user request finds out the hard way.
• Virtual keys with per-key budget caps — issue scoped keys per team or service, each with its own spend ceiling, so one runaway job cannot drain the account.
• Request and response logging plus traces — one place to see every call, its latency, which provider served it, and why it was routed there.
• A consistent error taxonomy — the retry/fallback/fatal classification from earlier, applied uniformly so behaviour is predictable.
• Idempotency for retried writes — if a retried request can trigger a side effect (a tool call that books, charges or sends), make it idempotent so a fallback does not duplicate the action.
• A default plus fallback chain per task tier — cheap model first, escalate on failure, with the tier matched to the task so you do not pay for a frontier model on a trivial classification.

Next steps

1. Write the error taxonomy first. Classify every status code as retry-same-model, fall-over, or fatal. Everything else is built on this.
2. Stand up a gateway with two providers and a fallback chain. LiteLLM self-hosted for control, OpenRouter for speed. Point one logical name at a primary, a secondary and your own vLLM box.
3. Configure cooldown and share state in Redis. Run more than one gateway instance and let them agree on which providers are down.
4. Split your timeout and retry policy by path. Short and shallow on interactive paths; patient and deep on batch paths. Measure the p95 you actually ship.
5. Add idempotency before any tool that writes. Then turn the failover on with confidence.

For the primary documentation, see LiteLLM's routing docs, its proxy reliability guide and its reliable completions reference, the OpenRouter docs, and this field write-up on handling provider failover. Flag names and defaults move between releases — check the version you are running.