Latency budgets for chat UX: streaming, TTFT and the art of feeling fast

What "fast" really means for an AI feature

Ask a product manager whether their chat assistant is fast and they will usually quote a number: the model returns a full answer in six seconds, say, or a summariser finishes a document in four. Ask the person actually using it, and you get a different verdict entirely. Speed, as users experience it, has almost nothing to do with total generation time. It has everything to do with how quickly the interface stops feeling dead and starts feeling alive.

This is the single most important thing to internalise before you tune a single parameter: users judge an AI feature by how fast it feels, not by how long it takes to finish. A response that begins appearing four hundred milliseconds after the user hits enter and then streams smoothly to completion over eight seconds feels quick and responsive. A response that sits behind a spinner for two seconds and then dumps the whole answer at once feels sluggish — even though, on the stopwatch, it finished sooner. Perceived speed and measured speed are two different quantities, and the gap between them is where good chat UX is won or lost.

For teams shipping across India and the UK, this distinction carries an extra weight. Your users are not on one network, one device class or one continent. A Builder in Bengaluru on a fast fibre line and a user in Manchester on patchy mobile data are hitting the same endpoint with very different physics in the wire. If you optimise only for the median case on an idle developer laptop, you will ship something that feels crisp in the demo and laggy in the field. The remedy is not a vague instinct to "make it faster" — it is a latency budget: a concrete, per-stage allocation you can measure against and defend.

Perceived versus actual latency, and why streaming changes the maths

When a large language model answers without streaming, the client waits for the entire response to be generated server-side, then receives it in one block. Under that model, perceived latency equals total latency: the user stares at a spinner for the full duration and the experience is defined by the slowest possible number — the moment the last token lands.

Streaming rewrites that equation. With server-sent events (SSE), tokens are pushed to the browser the instant they are produced, and the user begins reading the moment the first words appear. From that point on, generation and reading happen in parallel. The user's clock effectively stops at the first token; everything after it is masked by the act of reading. This is why, with streaming, perceived latency is approximately time-to-first-token (TTFT), not the full generation duration.

The practical consequence is stark. If you shave two seconds off total generation but leave TTFT untouched, users will barely notice. If you shave two hundred milliseconds off TTFT, they will tell you the product got faster. The whole discipline of chat-UX latency, therefore, reorganises itself around one question: how quickly can we get the first useful token in front of the reader, and how do we keep the stream flowing smoothly once it starts?

Building a latency budget, stage by stage

A latency budget is nothing more than a target broken into the stages a request genuinely passes through, with a millisecond allocation for each and a measurement against real traffic. The total experienced latency of a streamed answer decomposes cleanly:

total = network RTT + gateway/auth + queueing + prefill (TTFT) + decode, where decode itself is roughly output tokens divided by the tokens-per-second rate.

The value of writing it out this way is that it turns a fuzzy complaint ("it feels slow") into a diagnosis ("prefill is eating our budget on cold prompts"). You measure each line independently, compare it to its allocation, and fix the stage that is over budget rather than blindly reaching for a faster model. Here is a worked budget for a streamed chat turn, with representative allocations you can adapt to your own targets. Treat the numbers as a shape to reason about, not gospel — your real figures come from your own traces.

Stage	What happens	Target (same-region)	What blows it up
Network RTT	Client to edge to inference region and back	20–60 ms	Cross-continent hops; a Mumbai user hitting a London endpoint pays this on every request
Gateway / auth	TLS, API gateway, token validation, rate-limit check	10–40 ms	Cold connections, synchronous auth lookups, per-request JWT verification against a slow store
Queueing	Waiting for a free slot on the inference server	0–80 ms	Concurrency spikes; this is the line that explodes at p99 while p50 looks calm
Prefill (TTFT)	Reading the prompt and producing the first token	150–500 ms	Long uncached prompts; every extra thousand input tokens adds prefill cost
Decode	Streaming the remaining tokens	output length ÷ tokens-per-sec	Slow tokens-per-second under load; long outputs; small batch efficiency

Two lines in that table deserve special attention because they are where perceived speed is decided. Prefill is TTFT — it is the stage that ends the moment the first token appears, so it is the stage the user feels most acutely. Decode does not affect the start of the experience at all, but it governs whether the stream stays ahead of the reader once it has begun. Everything above prefill — network, gateway, queueing — is dead time the user spends staring at nothing, so it is disproportionately expensive to your perceived-speed budget. Trimming forty milliseconds off gateway overhead is worth more to the feel of the product than trimming a full second off decode.

Winning TTFT: first-token-fast and prompt caching

Because perceived latency tracks TTFT, the highest-leverage work you can do is get that first token out fast. There are two complementary strategies, and mature systems use both.

The first is first-token-fast: produce the opening of the response from something cheaper than a full cold call to your largest model. That might mean routing the very first fragment through a smaller, faster model while the primary model spins up, or serving a cached opening for a known prompt shape. It might mean beginning the answer with a short, deterministic framing sentence that you can emit instantly while the model does its heavier reasoning behind it. The goal is to break the blank-screen spell within a few hundred milliseconds, because that first token is what stops the user's clock. For teams that want to go deeper on the mechanics of pushing tokens out under load, our guide on streaming LLM responses at scale with SSE and backpressure covers the transport layer in detail.

The second strategy is prompt caching, and for any application with a stable prefix it is close to free money. Most production prompts carry a large, unchanging preamble: a long system prompt, a block of tool definitions, a retrieved reference document, a few-shot exemplar set. On a naive call the model recomputes the attention state for all of that on every single request. Prompt caching stores the key-value state for the static prefix and reuses it, so the model skips straight to the part that has actually changed. Reusing that cached prefix can cut input and prefill cost by up to around ninety per cent and, crucially for us here, lowers TTFT on repeated prefixes because the expensive re-reading work is simply not done. Our deep dive on prompt caching to cut LLM costs across Claude, GPT and Gemini walks through the provider-specific details.

Here is a compact SSE handler that puts the first-token-fast idea into practice. It streams tokens to the browser the moment they arrive and records TTFT as the first chunk lands, so the metric that matters is captured at the point it happens.

// Node/Express — stream LLM tokens over SSE, measure TTFT
app.get('/chat/stream', async (req, res) => {
  res.writeHead(200, {
    'Content-Type': 'text/event-stream',
    'Cache-Control': 'no-cache',
    'Connection': 'keep-alive',
    'X-Accel-Buffering': 'no',   // stop proxies buffering the stream
  });

  const started = performance.now();
  let firstTokenAt = null;

  // Static prefix is cache-flagged so prefill is cheap on repeat calls
  const stream = await model.stream({
    system: SYSTEM_PROMPT,            // long, stable -> cached prefix
    cache: { prefix: true },
    messages: req.session.history,
  });

  for await (const chunk of stream) {
    if (firstTokenAt === null) {
      firstTokenAt = performance.now();
      metrics.observe('ttft_ms', firstTokenAt - started);  // the number users feel
    }
    // Flush immediately — never batch tokens waiting for a "nicer" payload
    res.write(`data: ${JSON.stringify({ t: chunk.text })}\n\n`);
  }

  res.write('event: done\ndata: {}\n\n');
  res.end();
});

The two lines that do the real work are easy to miss. The X-Accel-Buffering: no header stops an intermediate proxy from silently hoarding your carefully streamed tokens and releasing them in one lump, which quietly destroys the entire benefit. And the cache: { prefix: true } flag is what turns a long, expensive prompt into a cheap, low-TTFT one on every call after the first.

Keeping decode ahead of reading speed

Once the first token is out, the second half of the battle is the stream's pace. Streaming only helps while the text stays ahead of the reader. If tokens arrive faster than someone reads, the words appear to flow and the experience feels effortless. If they arrive slower, the cursor stalls, the reader waits for the next word, and the streaming illusion collapses into something that feels worse than a clean spinner — because now the user is watching the machine struggle.

The rule is simple: keep your output tokens-per-second comfortably above human reading speed, with headroom for your slowest expected reader and your busiest expected moment. The trap is measuring this on an idle server. A model that comfortably out-paces reading at low concurrency can drop below it when a hundred sessions decode in parallel and per-stream throughput falls. That is precisely when your users are most numerous and least forgiving. If you cannot sustain the rate under real load, raising it is a genuine engineering problem — techniques such as speculative decoding for faster LLM inference exist specifically to lift tokens-per-second without swapping to a weaker model.

There is a subtle corollary for longer answers. Because decode time scales with output length, verbose responses cost you on the tail of the stream even when TTFT is excellent. Trimming an answer from eight hundred tokens to four hundred does not change how fast it starts, but it halves how long the reader waits for it to finish. Concise output is a latency optimisation, not just an editorial preference.

Optimistic UI: filling the gap before the first token

Even a well-tuned system has an irreducible gap between the user's action and the first token — the network RTT, the gateway, the prefill. You cannot always drive that to zero, but you can change what the user experiences during it. This is where optimistic and skeleton UI earn their place.

The techniques are familiar from the rest of front-end engineering, applied here to the pre-first-token window. Render the user's own message instantly and echo it back before the server has done anything. Show a typing indicator or an animated skeleton the moment the request is dispatched, so the interface is visibly working rather than frozen. Reserve the layout space the answer will occupy so nothing jumps when text arrives. Each of these reduces the felt weight of the wait without changing a single millisecond on the server. The vote widget on this very page uses the same principle — it updates the score the instant you click and rolls back only if the server disagrees, so the interface never feels like it is waiting on the network.

Optimistic UI is not a substitute for a real TTFT budget; a spinner over a three-second wait is still a three-second wait. But paired with genuine first-token speed, it smooths the last rough edge — the unavoidable few hundred milliseconds — into something that reads as deliberate rather than broken.

Regional routing for India and the UK

No amount of model optimisation can beat the speed of light. A request from a user in Chennai to an inference endpoint in London, and the tokens back again, pays a fixed round-trip tax measured in the low hundreds of milliseconds — before the model has done any work at all. That tax lands on the most expensive part of your budget, the dead time before the first token, and it is entirely avoidable.

The fix is to put inference close to the user. Serve Indian traffic from a region such as AWS Mumbai and UK traffic from London, so the network RTT line in your budget stays in the tens of milliseconds rather than the hundreds. Route by user location at the edge, resolve which region to hit before the request reaches your application, and keep the model endpoint in the same region as the audience it serves. For a dual-market product this is not a nice-to-have; it is the difference between a chat feature that feels local in Bengaluru and Birmingham alike and one that feels remote in at least one of them.

Regional routing interacts with prompt caching in a way worth planning for: a cache warmed in Mumbai does not help a request served from London. If you run active-active across both regions, warm the static prefix in each, and accept that a user who roams between them pays one cold prefill. For most consumer traffic that is a rounding error; for a globally mobile user base it is a thing to measure.

Budgeting for the tail: p50 lies, p99 remembers

The most common way a latency budget passes review and then fails in production is that it was written against the median. The p50 TTFT of almost any well-built system looks fine — that is the case you tested by hand. The number your users actually remember is the p99: the one time in a hundred that the queue was full, the cache was cold, the region was saturated and the first token took three seconds to appear. Users do not average their experiences; they anchor on the worst one, and they churn on it.

So budget for both. Set a p50 target and a p99 target for TTFT, and measure them separately under realistic concurrency rather than on an idle box. The queueing line in your budget is usually the culprit at the tail: it sits near zero at p50 and balloons under load, which is exactly why a system that feels instant in the demo feels erratic once traffic arrives. If your p99 TTFT is several multiples of your p50, you have a concurrency problem masquerading as a latency problem, and no amount of first-token cleverness on the median path will fix it.

Pitfalls that quietly wreck perceived speed

A handful of mistakes recur often enough to name. Watch for each of them before you ship.

• Buffering between the model and the browser. A proxy, gzip middleware or CDN that batches chunks turns streaming back into a single block. This is the most common cause of "we stream but it doesn't feel like it".
• Optimising total latency instead of TTFT. Effort spent shaving the tail of decode is largely invisible to users. The same effort spent on prefill and the pre-token stages is felt directly.
• Decode that drops below reading speed under load. A tokens-per-second rate that passes on an idle server can fail at peak concurrency, exactly when it matters most.
• Cross-continent routing. Serving both India and the UK from one region taxes half your users on every request.
• Ignoring the tail. A budget written against p50 will be broken at p99, and p99 is what users remember.
• Cold caches on a hot path. If a workload has unpredictable cold sessions, the low-TTFT economics of prompt caching evaporate; measure your cache hit rate, not just your cache configuration.

Putting it together

Fast AI UX is not a single lever. It is a budget: a stage-by-stage accounting of where the milliseconds go, tuned so the ones the user feels are the ones you spend least of. Get the first token out quickly with first-token-fast routing and prompt caching. Keep decode comfortably ahead of reading speed, and keep answers concise so the tail stays short. Mask the irreducible gap with optimistic UI. Put inference in Mumbai for India and London for the UK so the network tax stays small. And measure the tail, because p99 is the experience your users will describe to other people. Do those things and your feature will feel fast — which, as far as anyone using it is concerned, is the only kind of fast that exists. If you are building real-time voice on top of this, the same discipline applies with tighter margins; our companion piece on real-time voice agents, latency budgets and barge-in extends the ideas to the audio path.