Prefill-Decode Disaggregation: When Splitting LLM Inference Pays

What you need to know

Prefill — the phase that reads your prompt and produces the first token — is a set of large matrix multiplications over every prompt token at once. It saturates a GPU's floating-point units. Decode — the phase that emits every token after the first — processes exactly one token per step per sequence, and spends most of its time re-reading the key-value cache out of high-bandwidth memory. It saturates memory bandwidth and leaves the floating-point units largely idle. These are not similar workloads wearing different hats. They are different workloads.

When both run on the same GPU pool, a scheduler has to interleave them, and every interleaving choice hurts somebody. Let a long prefill run to completion and every user currently streaming tokens sees their stream pause. Chop the prefill up to protect the streamers and the prefill takes longer, so the user waiting for their first token waits longer. There is no setting that makes this go away on one pool; there is only a dial that moves the pain between two groups of users.

Disaggregation removes the dial by removing the contention. Prefill workers do nothing but prefill, decode workers do nothing but decode, and the key-value cache computed during prefill is shipped across the interconnect to the decode worker that will stream the answer out. Each pool is then sized, tuned and autoscaled on its own terms. The approach was formalised in the research literature by DistServe (arXiv:2401.09670) and Splitwise (arXiv:2311.18677), and as of August 2026 it has shipped in some form across the mainstream serving stacks — vLLM's disaggregated prefilling, SGLang's PD disaggregation, TensorRT-LLM's disaggregated service and NVIDIA's Dynamo inference framework among them. Read those pages before you plan a migration, though: at the time of writing vLLM labels its implementation experimental and TensorRT-LLM labels its own a prototype, so "shipped" here means available and documented, not necessarily settled.

The catch is that the KV cache is large, and moving it is not free. For a 70B-class model with an 8,000-token prompt, you are moving gigabytes per request. Over NVLink that is single-digit milliseconds. Over a 25 GbE link shared with everything else in the cluster, it is close to a second — which is worse than the problem you were trying to solve. The fabric between your pools is not an implementation detail; it is the whole decision.

The honest summary: disaggregation is a scale technique. With a sustained multi-node deployment, long prompts, tight first-token targets and enough traffic to keep both pools busy, it is one of the highest-leverage changes available. With four GPUs behind a chat product in an AWS Mumbai or London region, continuous batching plus chunked prefill on a single pool gets you most of the benefit without a second control plane to babysit.

Why prefill and decode want different hardware

The cleanest way to see the difference is arithmetic intensity: the ratio of floating-point operations performed to bytes moved between memory and the compute units. Every accelerator has a break-even ratio. Above it, you are compute-bound and adding memory bandwidth does nothing. Below it, you are memory-bound and adding FLOPs does nothing.

Prefill sits far above the line. A prompt of 8,000 tokens is processed as a single batched matrix multiplication per layer, so each weight matrix is loaded from memory once and then reused across thousands of token positions. The arithmetic intensity is high, the tensor cores are busy, and utilisation looks the way GPU marketing material suggests it always looks.

Decode sits far below it. To generate one token, the model reads every weight matrix in the network and every entry in that sequence's KV cache, then performs a tiny amount of arithmetic against a single new token position. The weights get read again for the next token. And the next. Batching helps — that is what continuous batching is for — but the KV cache does not amortise, because every sequence has its own. As context grows, KV reads dominate and the GPU spends its time waiting on memory.

The two phases, characterised. Directional, not measured — the point is the shape of the difference, not precise ratios.

Property	Prefill	Decode
Tokens processed per step	All prompt tokens at once	One per sequence
Dominant bottleneck	Compute (FLOPs)	Memory bandwidth
Arithmetic intensity	High	Low
Benefit from larger batches	Modest — already saturated	Large — amortises weight reads
Benefit from faster memory	Modest	Nearly linear
User-visible metric	Time to first token (TTFT)	Time per output token (TPOT)
Cost driver	Prompt length, arrival rate	Completion length, concurrency, context size

Put those two workloads on one pool and you get head-of-line blocking. A request arrives with a 32,000-token prompt. The scheduler has forty sequences mid-stream, each expecting a token every 30 milliseconds or so. Running that prefill as one unit occupies the GPU for long enough that every one of those forty streams stalls visibly. Users do not report this as "increased p95 TPOT". They report it as the answer freezing mid-sentence, which is a far more damaging experience than a slightly slower start — a point worth reading alongside our guide to latency budgets for chat UX.

Chunked prefill is the single-pool mitigation: break the long prefill into slices and interleave decode steps between them. It works, it is the correct default, and it is why most teams never need disaggregation. But it is a compromise by construction — the chunks still consume compute the decode steps wanted, and the prefill still finishes later than it would have alone. You have spread the damage rather than removed it.

The second problem with one pool is that the optimal parallelism differs by phase. Prefill prefers a configuration that maximises compute throughput per request; decode prefers one that maximises aggregate memory bandwidth and batch capacity. On a shared pool you pick one and both phases live with it. Disaggregated, each pool runs the tensor-parallel degree and batching policy that suits it, and that decoupling is frequently worth as much as the contention removal itself.

The KV-transfer tax

Everything above argues for splitting. Here is the bill. When prefill finishes on worker A, the KV cache it produced has to reach worker B, which will do the decoding. That cache is not small.

The size formula is straightforward:

kv_bytes = num_layers
         * 2                      # one tensor for K, one for V
         * num_kv_heads           # after GQA/MQA grouping, not query heads
         * head_dim
         * seq_len
         * bytes_per_element      # 2 for FP16/BF16, 1 for FP8

# Illustrative configuration — a 70B-class dense model:
#   num_layers = 80, num_kv_heads = 8, head_dim = 128, FP16
# Per token, per layer: 2 * 8 * 128 * 2 = 4,096 bytes  (4 KiB)
# Per token, all layers: 4 KiB * 80     = 320 KiB
# For an 8,192-token prompt: 320 KiB * 8,192 ≈ 2.5 GiB

State the assumption plainly, because it does most of the work: this is an illustrative 70B-class dense model with 80 layers, 8 key-value heads under grouped-query attention, a head dimension of 128 and FP16 storage. Your model will differ, sometimes by an order of magnitude. Multi-head latent attention, as used by some recent open-weight releases, compresses the KV representation dramatically. FP8 KV storage halves it again. A model with 32 key-value heads instead of 8 quadruples it. Run the formula for your own configuration before you design anything around it.

So roughly 2.5 GiB must cross the fabric for a single 8k-token request. Now the fabric matters.

Illustrative, not a benchmark — each time is simple arithmetic: 2.5 GiB divided by the bandwidth in the adjacent column, assuming the link runs at full line rate with no contention and no overlap. Real deployments land some way below line rate once protocol overhead and shared traffic are counted, so treat every figure here as a floor rather than an expectation.

Fabric	Approx. usable bandwidth	Time to move 2.5 GiB	Verdict for disaggregation
Intra-node NVLink-class interconnect	~400 GB/s	~7 ms	Effectively free
800 Gb/s InfiniBand / RDMA fabric	~100 GB/s	~27 ms	Comfortably viable
400 Gb/s InfiniBand / RoCE	~50 GB/s	~54 ms	Viable with layer-wise overlap
200 Gb/s fabric	~25 GB/s	~107 ms	Marginal — overlap is mandatory
100 GbE, TCP	~12.5 GB/s	~215 ms	Usually a net loss
25 GbE, TCP	~3.1 GB/s	~860 ms	Do not do this

Two mechanisms make the tax survivable. The first is layer-wise streaming transfer: the KV cache for layer 1 is complete long before layer 80 finishes computing, so it can start moving immediately. Done properly, most of the transfer overlaps with prefill compute that was going to happen anyway, and the marginal cost collapses to the tail of the last few layers. Without that overlap you pay the full serial cost, and the table above becomes the honest picture rather than the pessimistic one.

The second is avoiding the transfer entirely. If both workers can address the same memory — a pooled memory tier, or a KV store both sides read from — then "transfer" becomes a pointer handoff. As of August 2026 a good deal of ecosystem engineering effort is pointed this way, and it also changes the economics of prefix caching, because a KV cache that lives outside any single worker can be reused by any worker.

One more cost that gets forgotten: the cache must be held somewhere during the handover, so both pools need headroom for in-flight sequences. The prefill pool cannot free a sequence's cache until the decode pool has accepted it, so a saturated decode pool applies backpressure to prefill in a way a co-located deployment never experiences. Monitor the queue depth between the pools as a first-class signal.

Sizing the two pools

Once split, the obvious question is how many GPUs go in each pool. The answer falls out of your traffic, specifically the ratio of prompt tokens to completion tokens, and the rate at which each pool can chew through its respective token type.

The method, in four steps:

1. Measure demand. From production logs, compute prompt tokens per second and completion tokens per second at your target load — separately, and at the load you need to survive rather than the average.
2. Measure supply. Benchmark one prefill worker for prompt tokens per second at your target TTFT, and one decode worker for completion tokens per second at your target TPOT with realistic batch size and context length. Use your own hardware; published figures will not transfer.
3. Divide. Prefill GPUs equal prompt-token demand over per-worker prefill rate; decode GPUs equal completion-token demand over per-worker decode rate. The ratio between the results is your starting pool ratio.
4. Add headroom and re-derive. Round up for burst, autoscale each pool on its own queue depth, and recompute whenever traffic shape shifts materially.

Typical workload shapes and which way they push the prefill-to-decode ratio. Token ranges are illustrative of common patterns, not measurements from any specific deployment.

Workload	Typical prompt	Typical completion	Prompt:completion	Ratio direction
Long-context RAG question answering	16k–32k	200–400	~50:1 and up	Strongly prefill-heavy
Interactive chat assistant	500–1,500	300–600	~2:1	Balanced, tilting to decode
Coding agent with tool loops	8k–40k, growing per turn	200–800 per step	High, but heavily cache-reused	Depends entirely on cache hit rate
Batch document summarisation	4k–12k	200–500	~20:1	Prefill-heavy, but latency-insensitive
Structured extraction / classification	2k–6k	50–150	~40:1	Strongly prefill-heavy

Read the last column carefully, because it contains a trap. Batch summarisation is prefill-heavy, which sounds like a case for a large prefill pool — but batch work has no interactive latency target, so the head-of-line blocking that motivates disaggregation does not matter. Nobody is watching those tokens stream. For offline work a single well-batched pool at maximum utilisation is cheaper and simpler, and the right lever is quantisation and batch sizing rather than architectural separation.

Independent autoscaling is the real prize. On a shared pool, a shift in traffic shape — a new customer sending 30,000-token contracts through the same endpoint — forces you to scale the entire fleet to absorb prefill pressure, paying for decode capacity you did not need. Disaggregated, the prefill pool grows and the decode pool does not. Picture a Manchester health-tech product where document ingestion spikes at the start of each clinic session while conversational load stays flat — in a shape like that, the difference lands directly on the invoice. The mechanics are covered in autoscaling LLM inference on Kubernetes with KServe and KEDA.

When disaggregation loses

This is the section most articles skip, so let us be direct. Disaggregation adds a network hop, a router, a transfer mechanism, a second deployment, a second autoscaler and a new class of failure in which one pool is healthy and the other is not. There are entirely ordinary situations where the benefit does not cover that.

Decision matrix. Count the rows that describe your deployment. The thresholds are illustrative rules of thumb, not measured cut-offs — they mark roughly where the trade-off tends to turn, and your own measurements override them.

Condition	Favours disaggregation	Favours a single pool
Median prompt length	Over ~4k tokens	Under ~512 tokens
Concurrency	Sustained high, both pools stay busy	Low or spiky, pools idle separately
Deployment footprint	Multi-node, many GPUs	Single node, 1–8 GPUs
Interconnect	NVLink-class or RDMA fabric	Standard virtual networking
Prefix-cache hit rate	Low — most prefill is genuinely new	High — little prefill left to move
Latency targets	Strict TTFT and strict TPOT	One matters, the other is loose
Traffic shape	Prefill and decode load vary independently	They move together
Team capacity	Dedicated platform or infra function	Small team, no on-call depth
Workload interactivity	User-facing streaming	Offline batch

The prefix-cache row deserves elaboration because it catches experienced teams. Disaggregation exists to move prefill work off the decode path; a prefix-cache hit means that prefill work has already been done and will not be recomputed. If 70% of your traffic shares a long system prompt or a stable document context and hits the cache, then 70% of the prefill you planned to isolate does not exist — you have built two pools, a router and a transfer path to optimise the remaining 30%, while paying the full operational cost. Multi-turn coding agents are the classic case: the context grows every turn, but almost all of it is a prefix of the previous turn.

For most teams the ordered list is: continuous batching, chunked prefill, prefix caching, tuned batch and context limits, quantisation, speculative decoding for decode-bound traffic — and only then disaggregation. That order is roughly by benefit-to-effort ratio, and most deployments run out of problems before reaching the end of it. The groundwork is in our vLLM throughput and latency playbook.

There is also an architectural escape hatch worth naming. If your prompts are enormous because you are stuffing whole document sets into context, the highest-leverage change may not be in the serving layer at all. Retrieval that cuts a 32,000-token prompt to 4,000 relevant tokens eliminates the prefill problem rather than relocating it — the trade-off is laid out in long-context versus RAG. Solve the workload before you solve the topology.

Deploying it: what the stack looks like

Assume the decision matrix said yes. A disaggregated deployment on Kubernetes has four moving parts.

Two independent deployments. A prefill Deployment and a decode Deployment, each with its own resource requests, container arguments, readiness probes and scaling policy. Same image with different flags is fine; what matters is that they are separately schedulable and separately scalable objects.

A KV-cache-aware router in front. This makes or breaks the deployment. A plain round-robin balancer routes a request to a decode worker with no relationship to where its KV cache lives, forcing a needless transfer or recomputation. The router must know which workers hold which cached prefixes, pair prefill with decode sensibly, and apply backpressure when decode saturates. Treat it as a stateful, latency-critical service, not ingress plumbing.

A KV transfer connector. The mechanism that actually moves cache blocks between workers — RDMA-backed where the hardware allows, with a slower fallback path. This is where you will spend your debugging time.

Independent horizontal scaling per pool. Prefill scales on prompt-token arrival rate or prefill queue depth; decode scales on active sequence count or decode queue depth. Scaling either on GPU utilisation is a mistake: a memory-bandwidth-bound decode pool can report high utilisation while having ample room for more concurrent sequences.

Launching the workers looks roughly like the following. These flag names are illustrative placeholders, not the interface of any specific product or version — frameworks expose a role selector along the lines of a --role or disaggregation-mode argument, plus a connector setting and a peer address. Check your framework's docs for the real names; they change between releases.

# ILLUSTRATIVE ONLY — flag names are placeholders.
# Check your serving framework's docs for the real interface.

# Prefill worker: optimise for prompt throughput, minimal KV residency
serve-llm \
  --model /models/my-70b-instruct \
  --role prefill \
  --tensor-parallel-size 4 \
  --max-num-batched-tokens 32768 \
  --kv-transfer-connector rdma \
  --kv-transfer-role producer \
  --peer-discovery-endpoint http://kv-router.llm.svc:8100

# Decode worker: optimise for concurrent sequences and KV capacity
serve-llm \
  --model /models/my-70b-instruct \
  --role decode \
  --tensor-parallel-size 2 \
  --max-num-seqs 256 \
  --gpu-memory-utilization 0.92 \
  --kv-transfer-connector rdma \
  --kv-transfer-role consumer \
  --peer-discovery-endpoint http://kv-router.llm.svc:8100

In Kubernetes terms each becomes the args block of a container in its own Deployment, fronted by its own Service, with an HPA or KEDA ScaledObject per pool reading queue depth rather than GPU utilisation. NVIDIA documents this deployment shape — separate prefill and decode workers, a cache-aware router, per-pool scaling — in the Dynamo documentation, and the structure is broadly consistent across frameworks even where the flag names are not.

Whatever you deploy, you need client-side measurement, because server-side metrics will not show you what a user experiences across the router hop. The following measures TTFT and TPOT from a streaming OpenAI-compatible endpoint:

import time
import httpx

def measure(base_url, model, prompt, api_key="not-needed"):
    """Measure TTFT and TPOT from a streaming completion."""
    payload = {
        "model": model,
        "messages": [{"role": "user", "content": prompt}],
        "stream": True,
        "max_tokens": 512,
    }
    headers = {"Authorization": f"Bearer {api_key}"}

    start = time.perf_counter()
    ttft = None
    token_times = []

    with httpx.stream("POST", f"{base_url}/v1/chat/completions",
                      json=payload, headers=headers, timeout=120.0) as r:
        for line in r.iter_lines():
            if not line.startswith("data: "):
                continue
            if line.strip() == "data: [DONE]":
                break
            now = time.perf_counter()
            if ttft is None:
                ttft = now - start        # first streamed chunk
            token_times.append(now)

    if ttft is None or len(token_times) < 2:
        return None

    # TPOT = mean gap between successive output chunks, excluding TTFT
    gaps = [b - a for a, b in zip(token_times, token_times[1:])]
    return {
        "ttft_ms": ttft * 1000,
        "tpot_ms": (sum(gaps) / len(gaps)) * 1000,
        "output_chunks": len(token_times),
        "total_s": token_times[-1] - start,
    }

Run that concurrently at your target load, collect percentiles rather than means, and keep the raw samples. A mean TPOT hides exactly the stalls disaggregation is meant to eliminate; the p99 is where the story lives. If you front multiple backends, a gateway layer gives you one consistent place to capture these metrics — see LLM gateways compared.

Measuring whether it worked

Four numbers decide this, and only four:

• TTFT p95. How long the slowest twentieth of users wait to see anything. Driven by prefill capacity and queueing.
• TPOT p95. The gap between output tokens for the slowest twentieth. Driven by decode capacity and contention. This is the number disaggregation targets most directly.
• Throughput at a fixed latency SLO. Requests or output tokens per second sustained while both percentiles remain inside target.
• Cost per million output tokens. Total GPU-hours across both pools, divided by output tokens delivered. The number that decides whether any of this was worth it.

A template, not a result — the shape of the comparison you should produce for yourself, for example a 70B-class model at 8k prompt and 512 completion under sustained load. The cells deliberately carry directions rather than numbers, because no figure printed here would be a measurement of your stack, and none of these rows are measured results from any vendor, framework or paper.

Metric	Co-located, chunked prefill	Disaggregated	How to read it
p95 TTFT	Baseline	Target: lower or flat	Regression here means the prefill pool is undersized or the router is queueing
p95 TPOT	Baseline	Target: materially lower	The primary signal; no improvement means contention was not your bottleneck
Throughput at fixed SLO	Baseline	Target: higher	Measured only inside the SLO, never at saturation
GPU-hours consumed	Baseline	Often higher	Two pools means idle capacity in whichever is currently slack
Cost per 1M output tokens	Baseline	The verdict	Throughput gain must exceed the extra GPU-hours, or you have paid for complexity
p99 KV transfer time	Not applicable	New metric to own	If this creeps into the TTFT budget, your fabric is the constraint

Track the KV transfer percentile as a first-class metric from day one. It tells you whether a TTFT regression came from an undersized prefill pool or from the network, and those two diagnoses lead to completely different fixes.

Where teams get this wrong

Five failure patterns, with the fix for each.

1. Disaggregating before exhausting single-pool tuning. Chunked prefill, prefix caching and sensible batch limits are not on by default everywhere, and they deliver a large share of the benefit for near-zero operational cost. Fix: work the single-pool checklist and re-measure before changing the topology. Many teams stop there permanently.
2. Splitting the pools without a KV-aware router. A round-robin balancer discards the locality the architecture depends on, producing transfers and recomputation a co-located deployment would never have needed. Fix: treat the router as the primary component. If your framework ships no cache-aware scheduler, you are not ready.
3. Sizing the pools once and never again. The ratio encodes a traffic mix that will change — a new enterprise customer, a longer system prompt, a switch from chat to agentic workflows. Fix: emit prompt-token and completion-token rates as separate metrics, alert when the ratio drifts beyond a band, and re-derive quarterly at minimum.
4. Scaling both pools on GPU utilisation. Utilisation is a misleading signal for a memory-bandwidth-bound workload, and it is what most autoscaling templates reach for by default. Fix: scale prefill on queue depth or waiting prompt tokens, decode on running plus waiting sequence count. Utilisation belongs on a dashboard, not in a scaling policy.
5. Ignoring failure asymmetry. A decode worker dying mid-stream, or a KV transfer failing after prefill completed, is a new class of incident with no analogue on a single pool — prefill compute that cannot be delivered is compute you paid for and threw away. Fix: define retry semantics explicitly, and rehearse the case where the decode pool is unavailable while prefill stays healthy.

What to do next

Prefill-decode disaggregation is a genuine advance with a narrow entry requirement. As of August 2026 it is documented across the open serving ecosystem, and it is the shape the research literature and the framework roadmaps have converged on. Very few commercial providers publish their serving topology, so treat any claim about what the frontier labs run internally — including this one — as inference rather than fact. Convergence in the open stacks is a reasonable signal that the technique is sound. It is a much weaker signal that it is right for a deployment two orders of magnitude smaller.

The sequence that will not waste your time: instrument first, so you know what share of your GPU-seconds go to prefill and what share of that prefill is a cache miss. Exhaust single-pool tuning next — chunked prefill, prefix caching, batch limits, quantisation. Measure your achievable node-to-node bandwidth honestly. Only if the numbers still point at contention, and the fabric can carry gigabytes per request in tens of milliseconds, split the pools — behind a traffic percentage, with the four metrics wired up before you start.

If you do run this migration, write it up. The before-and-after at a fixed latency SLO, the pool ratio you derived and why, and the KV transfer percentiles you had to chase are exactly the evidence that separates an engineer who has operated inference infrastructure from one who has read about it. That write-up belongs on your Builder profile, where the teams hiring for this work will find it.