LLM gateways compared: LiteLLM vs OpenRouter vs Portkey — how to pick the right proxy

The short version

If you call more than one model, or you plan to, you will eventually build a gateway whether you mean to or not. The only choice is whether you build it deliberately or accrete it as a mess of retry loops, environment variables and half-finished cost dashboards scattered across your services. Three tools dominate the deliberate route, and they are not really competitors — they are three different answers to the same question.

• LiteLLM is the self-hosted proxy. Open-source, MIT-licensed, one OpenAI-compatible API in front of 100+ providers. You run it, you scale it, you own the operations — and because it sits inside your own network, it adds minimal overhead.
• OpenRouter is the hosted aggregator. One API and one bill across hundreds of models from dozens of providers. You do not self-host anything; the trade is a network hop through a third party on every request.
• Portkey is the governed control plane. A gateway that leans hard into observability, guardrails, governance and prompt versioning, available both hosted and self-hostable. The angle is enterprise control rather than raw breadth.
• Pick on hosting model first. Self-host versus hosted decides your latency budget, your data-residency story for UK and EU GDPR and India's DPDP Act, and how much operational work you take on. Everything else is secondary.

What an LLM gateway actually does

Strip away the marketing and a gateway is a reverse proxy that speaks the language of language models. A raw application that calls a single provider directly works fine in a demo. It falls apart in production the first time that provider has an outage, the first time finance asks what you spent per customer, and the first time a prompt injection slips a rude answer past you. A gateway is where you solve those problems once instead of in every service.

The core responsibilities are consistent across all three tools, even though the emphasis differs:

• One API. Your code speaks a single dialect — usually the OpenAI Chat Completions shape — and the gateway translates to whichever provider you route to. Swapping Claude for a Llama model or a Mistral endpoint becomes a configuration change, not a rewrite.
• Fallback and routing. If the primary model errors or times out, the gateway retries against a secondary. Routing rules can send cheap requests to a small model and hard ones to a frontier model, or load-balance across regions.
• Cost tracking. Every request is priced and attributed, so you can answer "what did this feature, team or customer cost this month" without reverse-engineering provider invoices.
• Caching. Identical or semantically similar requests can be served from cache, cutting both spend and latency.
• Rate limiting. Per-key and per-team budgets stop one runaway job from exhausting a shared quota or a monthly budget.
• Observability. Latency, token counts, error rates and full request traces in one place, rather than scattered across provider dashboards.
• Guardrails. Input and output checks for personally identifiable information, prompt injection, toxicity or off-topic responses, applied centrally.

Where the three tools diverge is who runs the box and how much of the governance layer comes in the box. That is the whole story, and it is why a like-for-like feature grid is less useful than understanding the archetype each one represents.

The three archetypes

LiteLLM — the self-hosted proxy

LiteLLM, from BerriAI, is an open-source Python SDK and proxy server that exposes one OpenAI-compatible API in front of a large and growing list of providers — the project describes it as calling 100+ LLM APIs, and its documentation and community writeups in 2026 put the provider count at 100 or more, spanning Bedrock, Azure, Vertex AI, Anthropic, Cohere, Hugging Face, vLLM, NVIDIA NIM and the rest of the usual suspects. It is MIT-licensed, which matters if an open-source requirement or an internal licence-review process is part of your world.

The defining property is that you run it. You deploy the proxy as a container next to your application — in your own Kubernetes cluster, your VPC, a data centre in Mumbai or a region in London — and requests flow through infrastructure you control. Because the proxy is co-located with your app, a self-hosted proxy adds minimal overhead; the request is not crossing the public internet to reach a third party before it reaches the model. The flip side is equally blunt: you own the operations. Scaling, high availability, upgrades, the database that backs cost tracking, the alerting when the proxy itself falls over — all yours. LiteLLM gives you cost tracking, logging, load balancing and guardrails out of the box, but the running of it is your team's job.

OpenRouter — the hosted aggregator

OpenRouter is the opposite trade. It is a hosted gateway and marketplace: one OpenAI-compatible endpoint and one API key that reach across a very large catalogue — its documentation and independent writeups in 2026 describe access to 400+ models from 60+ providers, with automatic fallback and a single unified bill. You self-host nothing. You sign up, get a key, and every frontier and open model on the market is a string change away.

That breadth is the entire pitch. When a new model launches and everyone in your team wants to try it by Friday, OpenRouter usually has it before your procurement team has finished reading the provider's terms. Unified billing means one invoice instead of a dozen provider relationships, which is genuinely useful for a small team in Bengaluru or Manchester that does not want a finance department to manage a spread of API accounts. The cost of that convenience is architectural: every request takes a network hop through OpenRouter's infrastructure before it reaches the model, and your prompts and completions transit a third party. For breadth and speed of experimentation it is hard to beat; for tight latency budgets or strict data-residency requirements, that hop is exactly the thing you have to think hardest about.

Portkey — the governed control plane

Portkey occupies the third corner: a gateway that treats governance as the product, not an add-on. It provides a single interface to route across a very large model catalogue — its own materials in 2026 cite 1,600+ LLMs — and layers observability, guardrails (a library of 40+ pre-built checks by its own count), prompt management and versioning, and organisation-level access controls on top. The framing is the enterprise control plane: the place where a platform team sets policy and every application inherits it.

Crucially, Portkey is available both as a hosted service and as a self-hostable open-source gateway. As of 2026 the company open-sourced its gateway core on GitHub and was acquired by Palo Alto Networks, which tells you which direction the enterprise-governance wind is blowing. For a regulated fintech in London or a large services firm in Pune that needs audit trails, prompt versioning and centralised guardrails more than it needs the last exotic model, Portkey's shape fits the requirement. The trade, relative to a bare proxy, is more moving parts and more product surface to learn — you are adopting a platform, not just a router.

Side by side

The grid below compares the three across the dimensions that actually drive a decision. Treat the provider and model counts as indicative of scale rather than exact figures — every vendor's catalogue changes weekly, and you should confirm current numbers from each project's own documentation before you commit.

Dimension	LiteLLM	OpenRouter	Portkey
Hosting model	Self-hosted proxy (you run it)	Hosted only (SaaS)	Hosted or self-hostable
Providers / models	100+ providers, OpenAI-compatible	400+ models, 60+ providers	1,600+ models via one interface
Routing / fallback	Yes — load balancing and retries	Yes — auto-fallback across providers	Yes — policy-driven routing
Observability	Logging and metrics (you host the store)	Usage dashboards on the platform	Deep, first-class — the core selling point
Guardrails	Supported, extensible	Limited; breadth-focused	40+ pre-built checks
Cost tracking	Yes — per-key budgets, you own the data	Yes — one unified bill	Yes — with governance and attribution
Data residency	Fully in your control	Transits a third party — verify regions	Self-host for control; hosted otherwise
Licence	Open source (MIT)	Proprietary hosted service	Open-source gateway core (MIT) + hosted platform
Added latency	Minimal when co-located	A network hop to the aggregator	Minimal self-hosted; a hop when hosted

A minimal setup

Concrete beats abstract. Here is a minimal LiteLLM proxy configuration that fronts two providers behind one endpoint, with a cheap default and a frontier fallback. Save it as config.yaml and start the proxy with litellm --config config.yaml.

model_list:
  - model_name: chat-default
    litellm_params:
      model: anthropic/claude-sonnet
      api_key: os.environ/ANTHROPIC_API_KEY
  - model_name: chat-default          # same alias => load-balanced / fallback pool
    litellm_params:
      model: openai/gpt-5.5
      api_key: os.environ/OPENAI_API_KEY

router_settings:
  routing_strategy: simple-shuffle
  num_retries: 2
  fallbacks: [{ "chat-default": ["openai/gpt-5.5"] }]

litellm_settings:
  success_callback: ["langfuse"]       # cost + trace logging
  cache: true

Your application never learns any of this. It calls the gateway exactly as if it were OpenAI — the same request shape works whether you are pointed at LiteLLM running in your own cluster, or at a hosted aggregator's URL. Only the base_url and key change:

from openai import OpenAI

client = OpenAI(
    base_url="http://litellm-proxy.internal:4000",   # your gateway
    api_key="sk-your-gateway-key",
)

resp = client.chat.completions.create(
    model="chat-default",                            # a gateway alias, not a raw model
    messages=[{"role": "user", "content": "Summarise this ticket in one line."}],
)
print(resp.choices[0].message.content)

Because the interface is identical across all three tools, this same client code runs against OpenRouter or Portkey by changing two lines. That portability is the strongest argument for adopting any gateway at all: your application stops caring which model or vendor is behind the curtain.

The decision matrix

There is no single winner, so stop looking for one. Walk down these questions in order and the answer usually falls out by the third or fourth.

1. Do you need to self-host? If regulated data, an internal policy or a data-residency requirement means model traffic cannot transit a third party, you are choosing between LiteLLM and a self-hosted Portkey. A hosted-only aggregator is off the table before you compare features.
2. How tight is your latency budget? If tail latency is a product requirement, co-locate a self-hosted proxy in the same region as your application so it adds minimal overhead. If you can absorb a network hop, a hosted aggregator is fine and far quicker to set up.
3. How much governance and observability do you need? If deep tracing, prompt versioning, centralised guardrails and org-level access control are non-negotiable, Portkey is built for exactly that. If you need solid logging and cost tracking but not a full governance platform, LiteLLM covers it.
4. How much do you value breadth and speed of switching? If your team wants every new model available the day it ships, with one bill, a hosted aggregator wins on convenience. If you use a stable handful of models, breadth is not worth a third-party dependency.
5. How important is an open-source licence? If your stack must be open source — for compliance, for avoiding lock-in, or on principle — LiteLLM (MIT) and Portkey's open-source gateway core are your candidates.
6. What is your cost-visibility requirement? All three price and attribute spend. If you want that data inside your own database, self-host. If a vendor dashboard and one invoice are enough, a hosted option removes work.

Data residency: GDPR and DPDP

For dual-market teams this is often the deciding dimension, so it deserves its own section. Under UK and EU GDPR, and under India's Digital Personal Data Protection Act, where personal data is processed — and by whom — is a question you must be able to answer precisely. A gateway changes that answer.

A self-hosted proxy such as LiteLLM keeps request routing inside your own boundary. The gateway is your infrastructure, so it is not a new processor; your data-flow diagram gains a box you already control. That does not exempt you from the model providers themselves being processors, but it removes one third party from the chain and makes the residency story easier to reason about. Deploy the proxy in a London region for UK personal data or an Indian region for DPDP-scoped data, and the routing layer stays where you need it.

A hosted aggregator introduces a genuine third-party processor. Your prompts — which frequently contain personal data — transit its infrastructure and may be logged there. That is not disqualifying, but it is a compliance obligation: you need a data-processing agreement, you need to know the sub-processors and storage regions, and you need to confirm they satisfy your GDPR or DPDP position. As of July 2026, these details change as vendors add regions and revise terms, so verify them against the vendor's current documentation rather than a blog post — including this one. The safe default for regulated personal data remains a self-hosted gateway; reach for a hosted one when the data is non-sensitive or the convenience genuinely outweighs the added processor.

If cost is the reason you are reading this, a gateway is only half the answer — routing and caching policy is the other half. We go deep on that in LLM cost optimisation: cache, route, compress and in the practical playbook cut LLM costs with prompt caching and model routing. And once traffic flows through a gateway, wire its traces into your observability stack — see agent observability with OpenTelemetry for how to make those spans useful.

So — which one?

Reduce it to a sentence each. Choose LiteLLM when you want control, an open-source licence and data that never leaves your network, and you accept that you run the box. Choose OpenRouter when you want maximum model breadth and one bill with zero infrastructure, and a network hop through a third party is acceptable. Choose Portkey when governance, observability and centralised guardrails are the point, and you want that as a platform rather than something you assemble yourself.

The most common mistake is picking on the feature grid when the real decision was made two questions earlier, at hosting model and data residency. Answer those first for your India and UK footprint, and the rest of the matrix narrows to a single sensible choice. Build the gateway deliberately, put every service behind the OpenAI-compatible interface, and you will have bought yourself the freedom to change your mind later — which, in a market where the model you love this quarter is beaten next quarter, is the most valuable thing a gateway gives you.