Cut self-hosted LLM serving costs: quantisation, continuous batching and speculative decoding, stacked

What you need to know

• Cost is a serving problem, not a model problem. The same open-weight model can cost 5–8x more to serve naively than optimised on the same GPU. You change the runtime, not the weights.
• Inference has two phases. Prefill is parallel and compute-bound; decode is sequential and memory-bandwidth-bound. The KV cache is the memory that grows with sequence length and batch, and it is the dominant cost at scale.
• Four levers stack. Quantisation (weights and KV cache), continuous batching, speculative decoding and prefix caching attack different bottlenecks, so their gains largely multiply rather than overlap.
• Speculative decoding and prefix caching are free quality-wise. Quantisation is the one lever that can degrade output — so it is the one you must gate behind an eval.
• Measure cost-per-task, not tokens-per-second. Throughput is the means; your finance team cares about pounds and rupees per thousand requests.

This guide is deliberately about the serving layer for inference you host yourself. It is a companion to two pieces you should read alongside it: our cache, route and compress playbook covers the API-layer levers (prompt caching, model routing, prompt compression), and our prompt caching and model routing guide goes deeper on routing economics. Those work whether you self-host or not. Here we assume you have already decided to run your own GPUs — if you have not, jump to the self-host versus API decision first — and we ask a narrower question: given a model and a GPU, how do you make each token as cheap as physically possible?

Why self-hosted cost is a serving problem

Most teams reach for the wrong lever first. They see a high bill, assume the model is too big, and start hunting for a smaller one or a distilled student — and distillation is a real lever, covered in our distillation guide. But before you change what you serve, it is almost always cheaper, faster and safer to change how you serve it. To see why, you have to understand what a GPU is actually doing during inference.

Generating a response happens in two phases, and they have completely different performance characteristics.

Prefill is the phase where the model reads your prompt. Every input token is processed in parallel in a single forward pass, so prefill is compute-bound — it saturates the GPU's matrix-multiply units and scales with prompt length. A long system prompt and a big retrieved context make prefill expensive, but it is a one-off cost per request.

Decode is the phase where the model writes the answer, one token at a time. Each new token requires a full forward pass that attends back over every token generated so far. Crucially, generating a single token touches the entire model's weights but does very little arithmetic, so decode is memory-bandwidth-bound: the GPU spends most of its time waiting on memory, not computing. A GPU serving one decode stream at a time is mostly idle silicon.

The thing that ties both phases together — and the thing that quietly dominates your memory budget — is the KV cache. To avoid recomputing attention over the whole sequence on every step, the engine stores the key and value tensors for every token already seen. That cache grows linearly with sequence length and linearly with the number of concurrent requests in the batch. At realistic production batch sizes, the KV cache routinely exceeds the size of the model weights themselves. It is the reason you run out of memory long before you run out of compute, and it is why "how big is your KV cache" is the most important question in serving economics.

Lever 1 — Quantisation (weights and KV cache)

Quantisation stores numbers in fewer bits. It is the highest-leverage change you can make, because it shrinks both of the things that cost you money: the weights you must hold in memory and stream on every decode step, and the KV cache that limits how many requests you can batch.

There are two distinct targets, and they are often confused.

Weight quantisation compresses the model parameters. Going from FP16 to FP8 roughly halves the weight memory; INT4 quarters it. Because decode is memory-bandwidth-bound, smaller weights mean less data to stream per token, so throughput rises as a side effect of fitting the model in less space. As of mid-2026, FP8 is the sensible default on Hopper-class and newer GPUs — it is near-lossless on most workloads and well supported across serving engines. INT4 (via schemes such as AWQ or GPTQ) is the aggressive option you reach for when you must fit a larger model on smaller cards.

KV-cache quantisation compresses the cache instead of the weights, and this is the lever many teams miss. Storing keys and values in INT8 rather than FP16 roughly halves KV memory; combined with PagedAttention (which eliminates the fragmentation that wastes cache space) the practical reduction in effective KV footprint is reported in the region of 4–8x. More aggressive research approaches go further: Google's reported TurboQuant work claims roughly 6x KV-memory reduction at around 3 bits with near-zero accuracy loss and no retraining. Less KV memory means more concurrent requests on the same card — which is exactly what feeds the next lever.

Here is a vLLM launch command enabling FP8 weights and an FP8 KV cache together. The same intent maps cleanly onto SGLang and TensorRT-LLM; the flag names differ but the levers are identical.

# vLLM — FP8 weights + FP8 KV cache, with PagedAttention on by default.
# Continuous batching is also on by default in vLLM; the flags below
# tune how aggressively it packs the batch.
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --max-model-len 8192 \
  --gpu-memory-utilization 0.92 \
  --max-num-seqs 256 \
  --enable-prefix-caching \
  --tensor-parallel-size 2

# To go more aggressive on weights, swap in a pre-quantised INT4 (AWQ) checkpoint:
#   --quantization awq  --kv-cache-dtype int8
# INT4 is more memory-efficient but more likely to move your eval numbers.

Lever 2 — Continuous (in-flight) batching

If you only do one thing, do this. Static batching — the naive approach — collects a fixed group of requests, runs them together, and waits for the slowest one to finish before starting the next batch. Because requests have wildly different output lengths, the GPU sits idle holding finished slots open while one straggler keeps decoding. At any moment, most of your expensive silicon is doing nothing.

Continuous batching (also called in-flight or iteration-level batching) fixes this by working at the granularity of a single decode step. The instant a request finishes and frees a slot, a new request is admitted into the batch. The GPU stays packed. This is the headline feature of modern serving engines — vLLM, SGLang and TensorRT-LLM all do it — and it is the reason they exist as a category separate from "load the model and call generate in a loop".

The reported gains are large precisely because decode is memory-bandwidth-bound: streaming the weights once per step to serve one request is wasteful, but streaming them once to serve dozens of requests in a packed batch amortises that cost across all of them. Published figures put continuous batching at roughly 10–20x throughput versus static batching at comparable latency, depending on how variable your output lengths are. The more your traffic mixes short and long responses, the bigger the win. There is no quality cost — you are reorganising work, not changing computation — which makes this the safest big lever on the list.

Lever 3 — Speculative decoding

Speculative decoding attacks the sequential nature of decode head-on. The problem is that you cannot generate token N+1 until you have token N — so decode is inherently serial, and you pay one memory-bandwidth-bound forward pass per token. Speculative decoding breaks that serial chain with a clever trick.

A small, cheap draft model quickly proposes the next several tokens. The large target model then verifies all of those proposed tokens in a single parallel forward pass — the same kind of parallel pass it uses during prefill. Where the draft guessed correctly, those tokens are accepted for free; at the first wrong guess, the target's own token is used and the process resumes. Because the target always verifies, the accepted output is mathematically identical to what the target would have produced on its own. Quality is preserved exactly.

When the draft model's predictions are accepted often — which they are for predictable, templated or boilerplate-heavy text — you generate several tokens per target forward pass instead of one. Reported latency reductions sit in the region of 2–3x. The cost is a little extra GPU memory to host the draft model, and some tuning of the draft length. vLLM supports several flavours, including n-gram prompt lookup (which needs no separate draft model at all) and EAGLE-style draft heads.

# vLLM — speculative decoding with a small draft model verifying against
# the large target. Output distribution is preserved; you trade a little
# memory for the draft model in exchange for fewer target forward passes.
vllm serve meta-llama/Llama-3.3-70B-Instruct \
  --quantization fp8 \
  --kv-cache-dtype fp8 \
  --enable-prefix-caching \
  --speculative-config '{
      "model": "meta-llama/Llama-3.2-1B-Instruct",
      "num_speculative_tokens": 5
  }'

# No draft model handy? n-gram / prompt-lookup speculation reuses
# the prompt itself as the draft — strong on RAG and code where the
# answer echoes the context:
#   --speculative-config '{"method": "ngram", "num_speculative_tokens": 4}'

Lever 4 — Prefix caching

Prefix caching tackles the prefill side of the bill. Many production workloads send the same long prefix on every request: a fixed system prompt, a set of few-shot examples, a tool schema, or a retrieved document shared across a conversation. Recomputing the attention KV for that identical prefix every single time is wasted prefill compute.

Prefix caching stores the computed KV for shared prefixes and reuses it across requests, so only the genuinely new tokens are prefilled. In vLLM it is a one-flag change (--enable-prefix-caching, shown in the snippets above); SGLang's RadixAttention generalises the idea to arbitrary shared sub-trees of tokens, which is especially strong for branching agent conversations and shared few-shot blocks. The win shows up directly as lower time-to-first-token and lower prefill cost, and it is largest exactly where it matters most: long, stable system prompts and RAG pipelines that reuse the same retrieved chunks.

This is the self-hosted cousin of the API-layer prompt caching covered in our prompt caching and routing guide. The mechanism is the same — reuse the KV for a shared prefix — but when you own the server, you control the cache policy, the eviction, and the TTL, rather than paying a provider's published cache-read rate.

Stacking the levers

The reason this is a stack and not a menu is that each lever attacks a different bottleneck, so the gains largely multiply rather than overlap. Quantisation shrinks memory; continuous batching converts that freed memory into concurrency; KV-cache optimisation and prefix caching extend how far that concurrency reaches; speculative decoding shortens the serial decode chain on top. Here is how they relate.

Lever	What it cuts	Reported gain	Cost / quality risk
Weight quantisation (FP8 / INT8 / INT4)	Weight memory + decode bandwidth	~2x (FP8) to ~4x (INT4) memory; throughput rises with it	FP8/INT8 near-lossless; INT4 can degrade hard tasks — gate on eval
KV-cache quantisation + PagedAttention	KV-cache memory (the batch-size limiter)	~4–8x effective KV reduction; ~6x reported at 3-bit (TurboQuant)	Low at INT8; aggressive 3-bit needs verification on your eval
Continuous (in-flight) batching	Idle GPU time between requests	~10–20x throughput vs static batching	None to quality; needs concurrent traffic to pay off
Speculative decoding	Serial decode steps (latency)	~2–3x latency reduction	None to quality; can hurt throughput on a saturated server
Prefix caching	Repeated prefill of shared prefixes	Large drop in time-to-first-token on stable prompts	None; benefit scales with prefix reuse

Two stacking results are worth committing to memory, both as reported order-of-magnitude figures rather than guarantees. First, FP8 weights plus FlashAttention-3 plus continuous batching plus speculative decoding has been reported to deliver roughly 5–8x better cost-efficiency than a naive FP16-plus-static-batching baseline on an H100-class GPU. Second, INT4 weights plus KV-cache optimisation plus continuous batching plus prefix caching has been reported to cut self-hosted inference cost by roughly 60–80% against that same naive baseline. Across layers, stacking can take inference cost down by 80% or more. The exact number depends entirely on your traffic shape — output-length variance, prefix reuse, draft acceptance — which is why the next section matters.

A worked cost-per-task comparison

Throughput is the means; cost-per-task is the end. Here is the same model on the same GPU, served two ways. The hardware figure is an illustrative blended H100-class rate; substitute your own — IndiaAI Mission subsidised GPU-hours and UK regional cloud regions both land well below on-demand US pricing, which we return to below.

Metric (per GPU-hour)	Naive: FP16 + static batching	Optimised stack
GPU cost / hour (illustrative)	$2.50	$2.50
Effective concurrent requests	~4	~48
Requests served / hour	~1,400	~16,800
Cost per 1,000 requests	~$1.79	~$0.15
Relative cost-per-task	1.0x (baseline)	~0.08x (~92% cheaper)

The arithmetic is deliberately simple, and you can reproduce it on your own numbers. Cost-per-task is just the GPU's hourly cost divided by how many tasks it completes in that hour — everything the serving stack does is in service of raising that denominator.

# cost_per_1k.py — turn throughput into the number finance cares about.
def cost_per_1k_requests(gpu_cost_per_hour, requests_per_hour):
    return gpu_cost_per_hour / requests_per_hour * 1000

naive     = cost_per_1k_requests(2.50, 1_400)    # FP16 + static batching
optimised = cost_per_1k_requests(2.50, 16_800)   # quantise + batch + speculate + prefix-cache

print(f"naive:     ${naive:.2f} / 1k requests")      # ~ $1.79
print(f"optimised: ${optimised:.2f} / 1k requests")  # ~ $0.15
print(f"reduction: {(1 - optimised / naive) * 100:.0f}%")  # ~ 92%

# Sanity check against a provider quote before you commit to self-hosting:
#   if optimised cost-per-1k << managed-API cost-per-1k at your volume,
#   self-hosting wins; otherwise the API is cheaper. See the self-host
#   vs API decision linked below.

Measure and guard quality (the eval gate)

Every lever on this list is safe except one, and that one — quantisation — is the lever with the biggest single payoff, which is exactly why it is dangerous. The discipline that makes the whole stack trustworthy is a quality gate that runs before any change reaches production.

The gate is simple to state and easy to skip: assemble a representative eval set drawn from your own traffic, score the model before the change, apply the change, score again, and refuse to ship if the delta exceeds your tolerance. Generic leaderboards are not enough — they over-represent easy questions and under-represent the long-context, code and reasoning tasks where aggressive quantisation breaks first. Our vLLM production playbook walks through the VRAM sizing and FP8/INT4 trade-offs that determine which quantisation you can even fit; this is the step that tells you whether you should.

Order your rollout by risk. Turn on continuous batching, prefix caching and speculative decoding first — they carry no quality risk, and between them they often deliver the majority of the saving. Then introduce quantisation behind the eval gate, starting at FP8 and only descending to INT4 if you genuinely need the memory and the eval holds.

When self-host beats API — and where to host it

An optimised serving stack is what makes self-hosting competitive in the first place. Naive self-hosting often loses to a managed API on cost; optimised self-hosting at steady, predictable volume frequently wins, because you are amortising a fixed GPU cost across a packed batch rather than paying per-token retail. The crossover depends on your volume, your latency requirements and your data-residency constraints — work through it properly in our self-host versus API decision guide before you buy GPUs.

Where you host changes the maths as much as how you serve. In India, the IndiaAI Mission's subsidised GPU pool puts H100-class capacity in reach at rupee-denominated rates well below on-demand US cloud pricing, which lowers the volume at which self-hosting breaks even and makes the optimised stack pay back faster. In the UK and EU, the lever is usually data residency rather than headline price: keeping inference inside a UK or EU region (and out of US jurisdiction) is often the deciding factor for regulated workloads, and self-hosting on a regional cloud is the cleanest way to guarantee it. In both markets the conclusion is the same — the cheaper or more constrained your hosting, the more the serving-layer levers in this guide are worth getting right.

Next steps

1. Instrument first. Capture time-to-first-token, inter-token latency and GPU utilisation. The numbers tell you whether you have a batching, prefill or model-size problem.
2. Turn on the free levers. Continuous batching, prefix caching and speculative decoding carry no quality risk. Get them on and drive the server with concurrent traffic.
3. Quantise behind an eval gate. FP8 weights and an INT8/FP8 KV cache by default; INT4 only if you need the memory and your workload eval holds.
4. Recompute cost-per-1k on real traffic and compare against a managed-API quote at your volume. If self-hosting still wins, you have your answer.

For the engine-level reference and primary documentation, see the vLLM docs, the SGLang docs and NVIDIA's TensorRT-LLM repository. The flags move between releases — check the version you are running.