Multi-Tenant RAG and LLM Serving: An Isolation Architecture Guide

What builders need to know before shipping multi-tenant RAG

• Isolation is a spectrum, not a binary. AWS's SaaS tenant-isolation framework names three working patterns — silo (dedicated resources), pool (shared resources) and bridge (a mix) — and almost every real multi-tenant RAG deployment ends up as a bridge once it has more than a handful of enterprise customers.
• The datastore enforces access control, never the model. An LLM cannot be relied on to refuse content it was handed; the boundary has to be a deterministic filter, namespace or row-level-security policy that runs before retrieval, not a system-prompt instruction.
• Metadata filters are not the same thing as isolation. A WHERE clause or metadata filter on a shared index is only as strong as the code path that sets it — physical partitioning (namespaces, RLS with FORCE enabled, per-tenant collections) fails closed when something is misconfigured; filter-only approaches can fail open.
• Ingestion needs the same rigour as query time. Most real leaks trace back to stale access-control lists at ingestion — a document deleted or re-permissioned upstream that never gets removed from the vector store — not a broken query filter.
• Noisy neighbours are a compute problem as much as a data problem. GPU inference batches don't self-partition the way OS processes do; one tenant's burst traffic can degrade latency for everyone sharing that serving instance unless you explicitly budget concurrency per tenant.

Before you design anything: prerequisites and terms

This guide assumes you already have a working single-tenant RAG pipeline — embeddings, a vector index, a retrieval step, and an LLM call — and are now being asked to serve more than one customer, team or business unit from the same infrastructure. If you haven't settled on a vector store yet, read our vector database selection guide first; the isolation model you pick constrains which engines are realistic, and retrofitting isolation onto a shipped single-tenant schema is meaningfully harder than designing for it up front.

A handful of terms recur throughout: a tenant is the unit of isolation — usually a customer account, sometimes a business unit or workspace inside one customer. Isolation is the guarantee that one tenant's data, compute and cost cannot be read, degraded or attributed to another tenant. A namespace or partition is a physical subdivision of storage a query is constrained to. Row-level security (RLS) is a database feature that filters rows based on the identity of the current session, enforced by the database engine itself rather than application code. And a noisy neighbour is a tenant whose usage pattern degrades service quality for other tenants sharing the same physical resource.

The three isolation models: silo, pool and bridge

AWS's SaaS Tenant Isolation Strategies whitepaper names this pattern for general SaaS architecture, and it maps directly onto RAG and LLM-serving infrastructure. Every design decision below — vector store, ingestion pipeline, GPU serving — ultimately picks one of these three postures, usually at a different layer for different tenant tiers.

Silo: dedicated resources per tenant

Each tenant gets a fully separate resource — a dedicated vector database instance, a dedicated OpenSearch domain, a dedicated fine-tuned model, or all three. There is no shared infrastructure at that layer, so there is no isolation logic to get wrong: a query against tenant A's silo physically cannot return tenant B's rows, because tenant B's rows don't exist anywhere tenant A's connection can reach. Silo wins on compliance defensibility, per-tenant encryption keys, clean data residency and trivial off-boarding (delete the instance). It loses on cost: most tenants sit mostly idle, and a fleet of hundreds of mostly-idle indexes is the kind of infrastructure bill that kills SaaS margins.

Pool: shared resources with logical partitioning

All tenants share the same index, cluster or serving fleet, and isolation is enforced logically — namespaces, partitions, row-level security, tenant-scoped API keys. Pool is what makes multi-tenant SaaS economics work: onboarding a new tenant is a configuration change, not a provisioning job, and idle tenants cost close to nothing. The trade-off is that isolation quality now depends entirely on how well the logical boundary is implemented and tested — which is why the next section spends most of its length on getting that boundary right rather than treating it as a checkbox.

Bridge: a deliberate mix

Almost no real platform is purely one or the other past its first few enterprise deals. The bridge pattern keeps most tenants pooled for cost efficiency while moving specific tenants — those with regulatory requirements, contractual data-residency clauses, or simply enough scale to justify dedicated capacity — into silos. A common shape: a shared pgvector cluster and shared vLLM serving fleet for the pooled tier, with select enterprise or regulated tenants getting a dedicated OpenSearch domain or a reserved GPU instance, all behind the same API surface so application code doesn't need to know which model a given tenant is on.

Dimension	Silo	Pool	Bridge
Isolation guarantee	Physical — no shared attack surface	Logical — as strong as your enforcement code	Physical for silted tenants, logical for the rest
Cost per idle tenant	High — dedicated capacity sits unused	Near zero — shared capacity, marginal cost only	Mixed, weighted toward pooled tenants
Onboarding speed	Slow — provisioning step required	Fast — a namespace create or a config row	Fast by default, slow for the silo-tier upgrade path
Off-boarding / deletion	Trivial — delete the instance	Requires a clean partition-delete operation	Depends on which tier the tenant sat in
Best fit	Regulated or enterprise tenants demanding residency	SMB, prosumer or high-volume low-margin tenants	B2B SaaS spanning a free/pro tier and a regulated enterprise tier

Read the AWS reference model directly at the AWS Well-Architected SaaS Lens — it predates the current wave of RAG-specific tooling, but the isolation trade-offs it describes for compute and storage transfer almost unchanged onto vector stores and LLM-serving fleets.

Enforcing isolation at the datastore, not the model

The single most important architectural rule in this guide: the LLM must never be the access-control layer. A large language model is a non-deterministic text generator. It has no reliable concept of "documents this user is allowed to see" beyond what you put in its context window, and it is measurably susceptible to prompt injection — an adversarial query can coax a model into surfacing or reasoning over content that should never have reached it in the first place. Every access decision — which tenant, which documents within that tenant, which roles within that document — has to be resolved deterministically before a single retrieved chunk touches the prompt. If the model can see it, treat it as already cleared; never ask the model to un-see something itself.

Postgres and pgvector: row-level security done properly

If you're running pgvector, Postgres's built-in row-level security (RLS) gives you database-enforced tenant isolation without a separate vector engine. The pattern has three parts: a tenant_id column, a policy that compares it to a session-scoped setting, and — the step most tutorials skip — FORCE ROW LEVEL SECURITY, without which a connection using the table owner's role silently bypasses every policy you wrote.

-- Enable RLS and prevent the owning role from bypassing it
ALTER TABLE document_chunks ENABLE ROW LEVEL SECURITY;
ALTER TABLE document_chunks FORCE ROW LEVEL SECURITY;

-- A row is visible (and writable) only when its tenant_id
-- matches the tenant bound to the current transaction.
-- missing_ok=true means an unset session variable returns
-- NULL rather than raising an error -- the comparison then
-- evaluates to unknown/false, so the query fails CLOSED
-- (zero rows) instead of erroring or leaking all rows.
CREATE POLICY tenant_isolation_policy ON document_chunks
  USING      (tenant_id = current_setting('app.current_tenant', true)::uuid)
  WITH CHECK (tenant_id = current_setting('app.current_tenant', true)::uuid);

-- Index the column you filter and enforce on
CREATE INDEX idx_document_chunks_tenant ON document_chunks (tenant_id);

-- The API connects as a least-privilege role -- never as
-- the table owner, which would bypass the policy above
CREATE ROLE rag_api_role LOGIN PASSWORD '<stored-in-secrets-manager>';
GRANT SELECT, INSERT, UPDATE, DELETE ON document_chunks TO rag_api_role;

One limitation worth planning around: as of mid-2026, pgvector's HNSW and IVFFlat indexes are not tenant-partitioned the way Qdrant's per-group indexing is — a similarity search still traverses one shared index structure even though RLS correctly restricts which rows are returned. For most workloads this is invisible; once a single Postgres table holds many millions of vectors across thousands of tenants, teams typically add native Postgres table partitioning by tenant_id alongside RLS so each partition gets its own, smaller vector index.

Namespace isolation in Qdrant and Pinecone

Both major managed vector databases implement pool isolation through physical partitioning rather than filter clauses, which is the pattern to prefer over ad hoc metadata filtering. Pinecone's multitenancy documentation recommends one namespace per tenant: each namespace is stored separately, queries and writes target exactly one namespace, and one tenant's activity cannot slow another's — the isolation Pinecone's own docs describe as avoiding the "noisy neighbour" problem at the storage layer, not just the compute layer. Read-unit cost is calculated per namespace size, so a 1 GB tenant namespace costs roughly 1 read unit to query, versus scanning a shared 100 GB namespace with a metadata filter, which can cost 100 read units regardless of how selective the filter is.

index = pc.Index("documents")

results = index.query(
    namespace=tenant_id,      # physical partition -- not a metadata filter
    vector=query_vector,
    top_k=8,
    include_metadata=True,
)

Qdrant takes a payload-based partitioning approach within a single collection: tag every point with a tenant_id field, build a payload index on it, and filter at query time. Qdrant's own multitenancy documentation notes that this filter runs inside the HNSW graph traversal itself rather than after it, so a well-indexed tenant filter doesn't retrieve-then-discard the way a naive metadata filter would. Since version 1.16, Qdrant also supports tiered multitenancy — keeping small tenants together in a shared shard while isolating large tenants into dedicated shards within the same collection, which is a bridge pattern implemented inside a single vector engine.

from qdrant_client import QdrantClient, models

client = QdrantClient(url="https://your-cluster.qdrant.io", api_key=QDRANT_API_KEY)

# Required once, before ingesting -- an unindexed payload
# field forces a full collection scan on every query.
client.create_payload_index(
    collection_name="documents",
    field_name="tenant_id",
    field_schema="keyword",
)

results = client.query_points(
    collection_name="documents",
    query=query_vector,
    query_filter=models.Filter(
        must=[models.FieldCondition(
            key="tenant_id",
            match=models.MatchValue(value=tenant_id),
        )]
    ),
    limit=8,
).points

Enforcing it end to end: a FastAPI dependency that injects tenant_id

Namespaces and RLS policies only protect you if every code path that reaches the datastore actually carries a verified tenant identity — and that identity has to come from something the caller cannot forge, which is where the JWT comes in. Amazon's own reference implementation for multi-tenant RAG on Bedrock and OpenSearch enriches the JWT at issuance — a pre-token-generation Lambda trigger looks up the caller's tenant and adds a tenant_id claim to the token — then has OpenSearch's fine-grained access control map that claim straight onto a backend role at query time. Notably, the same document points out that Bedrock Knowledge Bases cannot consume JWT-based OpenSearch access directly, so the orchestration layer below has to do this wiring itself rather than relying on a managed retrieval feature to enforce it for you.

This pattern is genuinely dual-market-ready without any redesign: Amazon OpenSearch Service runs in both Mumbai (ap-south-1) and London (eu-west-2), and Bedrock's cross-region inference profiles route Claude model calls through Mumbai for Indian tenants and through London or Frankfurt for UK and EU tenants who need in-region processing to satisfy GDPR or a sector data-residency rule. The isolation logic — JWT claim, FGAC role mapping, DLS filter — stays identical; only the region each tenant's domain and inference profile point to changes, which is the bridge model in miniature.

The same pattern works with any framework. The dependency below decodes and validates the JWT once, and every downstream function receives a tenant context it did not have to trust the caller for:

from fastapi import Depends, FastAPI, HTTPException, status
from fastapi.security import HTTPBearer, HTTPAuthorizationCredentials
import jwt  # PyJWT

app = FastAPI()
security = HTTPBearer()

class TenantContext:
    def __init__(self, tenant_id: str, user_id: str, roles: list[str]):
        self.tenant_id = tenant_id
        self.user_id = user_id
        self.roles = roles

async def get_tenant_context(
    credentials: HTTPAuthorizationCredentials = Depends(security),
) -> TenantContext:
    token = credentials.credentials
    try:
        payload = jwt.decode(
            token, JWT_PUBLIC_KEY, algorithms=["RS256"], audience="rag-api",
        )
    except jwt.PyJWTError:
        raise HTTPException(status.HTTP_401_UNAUTHORIZED, "Invalid token")

    tenant_id = payload.get("tenant_id")
    if not tenant_id:
        # Fail closed: a token with no tenant claim gets no access,
        # never "default" or "shared" access.
        raise HTTPException(status.HTTP_403_FORBIDDEN, "Token missing tenant_id claim")

    return TenantContext(tenant_id, payload["sub"], payload.get("roles", []))


@app.post("/query")
async def query_documents(
    request: QueryRequest,
    ctx: TenantContext = Depends(get_tenant_context),
    pool: asyncpg.Pool = Depends(get_db_pool),
):
    query_embedding = embed(request.query)

    async with pool.acquire() as conn, conn.transaction():
        # Bind this transaction to the caller's tenant for RLS.
        # is_local=true means it never leaks to the next request
        # on a pooled connection.
        await conn.execute(
            "SELECT set_config('app.current_tenant', $1, true)", ctx.tenant_id,
        )
        rows = await conn.fetch(
            """
            SELECT id, content, 1 - (embedding <=> $1) AS similarity
            FROM document_chunks
            WHERE tenant_id = $2          -- belt-and-suspenders: explicit
            ORDER BY embedding <=> $1     -- filter alongside RLS, not instead of it
            LIMIT $3
            """,
            query_embedding, ctx.tenant_id, request.top_k,
        )
    return {"results": [dict(r) for r in rows]}

Per-tenant ingestion pipelines: where isolation actually breaks

Query-time enforcement gets most of the design attention, but in practice most real-world data leaks in multi-tenant RAG trace back to ingestion, not retrieval. A document gets deleted or re-permissioned in the source system — Google Drive, Confluence, a CRM — and the vector store never hears about it. The embedding stays queryable indefinitely, invisible to any access-control review of the live application, because nobody thought to audit the ingestion pipeline's state alongside the retrieval code.

Treat ingestion as a first-class part of your isolation architecture, not a batch job that happens to run before the "real" system. Three practices matter most in production: subscribe to permission-change webhooks from source systems where they exist, and fall back to periodic reconciliation scans (nightly, at minimum) where they don't; tag every chunk at embedding time with both a tenant_id and, where the source system supports finer-grained permissions, an allowed_principals list, so query-time filtering can enforce document-level access within a tenant, not just tenant-level access; and make deletion propagation synchronous or near-synchronous rather than eventually-consistent for anything touching regulated data — a permission revoked in the morning should not still be queryable that afternoon.

If your retrieval strategy blends keyword and vector search, the same tenant boundary has to be threaded through both retrieval paths consistently — a hybrid pipeline that enforces tenant isolation on the vector leg but forgets it on the BM25 leg is a real, observed failure mode. Our hybrid retrieval guide walks through combining BM25 and vector search step by step; when adapting it for multi-tenancy, apply the tenant filter as the first stage of both retrieval paths, before any re-ranking or fusion step, not after.

Cost attribution and the noisy-neighbour problem

Data isolation and compute isolation are separate problems, and pooled LLM serving fails on the compute side in a way that has nothing to do with your vector store. GPU inference batches don't self-partition the way OS processes do: a single GPU running continuous batching might hold room for somewhere between roughly 40 and 80 concurrent sequences for a mid-sized open-weight model, and there is no automatic fairness mechanism stopping one tenant's traffic burst from consuming most of those slots and adding latency for everyone else sharing the instance.

Raw token counts are also a misleading basis for cost attribution. A short request riding on a large, uncached system prompt consumes meaningfully more GPU time than a longer request with a small, cache-hit prefix — so billing purely on total tokens over- or under-charges tenants depending on their prompt shape. Sound attribution tracks prompt and completion tokens separately, applies a discount (commonly in the 40–60% range) when the serving layer reports a cache hit on the prompt prefix, and accounts for the fact that a quantised model serving the same output at higher throughput costs less GPU-time per token than a full-precision model doing the same job. Our LLM gateway guide covers where this metering logic typically lives — the gateway layer, not scattered across individual services.

Isolation approach	How it works	Trade-off
Fair-share scheduling	Cap each tenant at roughly 1/N of available concurrency slots via an atomic counter (e.g. Redis with a Lua script to avoid race conditions on burst traffic)	Simple and cheap; a quiet tenant's unused capacity isn't available to a busy one
Priority lanes	Run separate serving pools per pricing tier (e.g. a premium pool and a standard pool) sharing the same physical fleet	Protects paying tiers from free-tier bursts; still shares hardware, so extreme load can bleed across lanes
Full compute isolation	Dedicated GPU partitions (e.g. MIG) or entire dedicated instances per tenant	Strongest guarantee, closest to silo-model economics — reserve it for tenants who explicitly pay for dedicated capacity

A decision framework: silo, pool or bridge, by tenant count, compliance tier and budget

There's no universal answer, but three signals reliably predict which model a given tenant — or your whole platform — should sit in.

Signal	Points toward Silo	Points toward Pool	Points toward Bridge
Tenant count	Small number of large, high-value accounts	Hundreds to millions of similar-sized tenants	A long tail of small tenants plus a handful of large ones
Compliance tier	Regulated sector, contractual data residency, sector-specific localisation rules (e.g. RBI/SEBI/IRDAI in India, or an EU customer requiring in-region processing under GDPR)	Low-to-moderate sensitivity, no per-tenant residency requirement	Mixed customer base — some regulated, most not
Budget / margin model	Enterprise pricing that already absorbs dedicated-infrastructure overhead	Thin per-seat margins that require amortising infrastructure across many tenants	Regulated tenants pay a premium that funds their own silo; everyone else stays pooled
Typical example	Hospital-network or government RAG deployment	Consumer or prosumer chatbot SaaS	B2B SaaS with a free/pro pooled tier and a compliance-gated Enterprise tier

In practice, start pooled by default — it's cheaper to build, cheaper to run, and the isolation techniques above (namespaces, RLS with FORCE enabled, JWT-derived tenant context) give genuine, auditable guarantees when implemented correctly. Reserve silo for the specific tenants whose contract, regulator or data-residency requirement actually demands it, and treat that as a per-tenant upgrade path rather than a platform-wide redesign. For teams selling into India and the UK simultaneously, this also means checking sector rules early: our DPDP compliance playbook covers what India's Digital Personal Data Protection Act does and does not require around localisation, which is a common point of confusion for teams assuming GDPR-equivalence covers them automatically.

Common pitfalls that break multi-tenant RAG in production

Trusting a system prompt to enforce a data boundary. "Only answer using documents belonging to tenant X" is an instruction, not a control. It degrades under adversarial input and under ordinary model drift. The boundary has to be enforced before retrieval, at the datastore or gateway, full stop.

Treating metadata filters as equivalent to namespace isolation. They solve a similar problem but with a different failure mode: a namespace that's never queried returns nothing; a metadata filter that's accidentally omitted from one code path returns everything. Prefer physical partitioning for the tenant boundary itself, and use metadata filters for finer-grained permissions within that boundary.

Letting ingestion drift out of sync with the source system's permissions. This is the pitfall this guide spends the most words on because it's the one audits actually find — a deleted or re-permissioned document that's still embedded and retrievable weeks later.

Billing by raw token count. It systematically misattributes cost between tenants with different prompt shapes and cache-hit rates, and it undersells the case for tiered compute isolation when a single tenant is genuinely consuming a disproportionate share of GPU time.

So — which model should you actually build?

Start from the pooled model with real enforcement — namespace or RLS-based physical partitioning, JWT-derived tenant context injected at every entry point, ingestion pipelines that track source-system permission changes, and compute budgets that stop one tenant from degrading another. That combination covers the overwhelming majority of multi-tenant SaaS RAG deployments, in India, the UK or anywhere else, at a cost structure that actually scales. Layer silo isolation on top for the specific tenants whose contract, regulator or scale genuinely demands it, rather than redesigning the whole platform around your most demanding customer. And regardless of which model you land on for any given tenant, hold the same line everywhere: the model answers questions, the datastore decides who gets to ask them.