How to stream LLM responses at scale: SSE, backpressure and disconnects

What you need to know before you build this

Streaming is one of those features that looks like a nice-to-have and turns out to be load-bearing. A non-streaming chat endpoint that takes eight seconds to return a full answer feels broken. The same answer, streamed token by token so the first words appear in under a second, feels fast — even though the total time is identical. The whole game is perceived latency, and streaming is how you win it.

The hard part is not emitting tokens; any tutorial can show you a generator that yields strings. The hard part is everything around it: the proxy that silently buffers your whole response, the user who closes the tab while you keep paying a model provider to generate tokens nobody will read, the load balancer that kills the connection after sixty idle seconds, and the question of whether one server can hold ten thousand open streams or falls over at two hundred. This guide covers all of that.

• Time-to-first-token (TTFT) is the metric that matters for how fast your product feels — optimise it ahead of total latency.
• Server-Sent Events (SSE) is the right default transport for one-way token streaming to a browser; it is plain HTTP and survives most infrastructure once buffering is off.
• Run an async server. A single async worker holds thousands of idle-waiting streams; a synchronous worker pool blocks one thread per stream and dies in the hundreds.
• Detect disconnects and cancel upstream generation. This is both a UX fix and a direct cost saving — you stop paying for tokens nobody receives.
• The proxy is usually the villain. Buffering, idle timeouts and connection limits at nginx, the load balancer or the CDN break streaming far more often than your application code does.

Prerequisites

This guide assumes you are comfortable with Python and a little JavaScript, and that you have an async web stack to hand. The examples use FastAPI on top of Starlette, served by Uvicorn, because that combination is the most common way Indian and UK teams ship LLM backends today — but the principles map cleanly onto Node, Go or anything else with non-blocking I/O.

You will want:

• Python 3.11 or later, with fastapi, uvicorn and the SDK for whichever model provider you use.
• An LLM endpoint that supports streaming. As of June 2026, OpenAI, Anthropic and Google Gemini all expose token streaming over SSE-style HTTP responses, and self-hosted servers such as vLLM and TGI do the same. If you are weighing up self-hosting against a managed API, our companion guides on self-hosting versus API inference and running an open-weight model on vLLM in production cover that decision in depth.
• A reverse proxy (nginx, or a managed load balancer) in front of the app — because that is where streaming goes to die, and you need to configure it.

Why streaming matters: TTFT versus total latency

When you call a large model, the response time splits into two phases. The first is prefill — the model reads your prompt and produces the first token. The second is decode — it generates the rest, one token at a time. Time-to-first-token measures the end of prefill: how long the user stares at a spinner before anything happens. Total latency measures the end of decode: when the answer is complete.

Here is the insight that makes streaming worth the engineering effort. If you do not stream, the user experiences total latency as dead time — a blank screen for the full duration. If you stream, the user experiences TTFT as the wait, and then reads along as tokens arrive at roughly reading speed. A response that takes six seconds to finish but starts in 600 milliseconds feels faster than a one-second response that arrives all at once, because the human is occupied the whole time. Perceived latency, not measured latency, is what people remember.

This is why every major chat product streams. It is also why streaming changes your architecture: you are no longer returning a value, you are managing a long-lived connection that can be cancelled, throttled or dropped at any moment.

Choosing a transport: SSE vs WebSockets vs gRPC vs chunked HTTP

There are four credible ways to push tokens to a client. Picking the right one saves a great deal of pain later.

Transport	Direction	Protocol	Auto-reconnect	Best for
Server-Sent Events (SSE)	Server → client	Plain HTTP/1.1 or HTTP/2	Built in (browser EventSource)	Token streaming to a browser — the default
WebSockets	Bidirectional	Upgrade from HTTP	Manual	Live voice, collaborative editing, interrupts
gRPC streaming	Uni- or bidirectional	HTTP/2 + protobuf	Manual	Service-to-service, typed contracts, polyglot backends
Plain chunked HTTP	Server → client	HTTP/1.1 chunked transfer	None	Lowest-level streaming; no event framing

SSE is the right default for browser-facing token streaming. It is one-directional, which is exactly what generating an answer needs — the client asks once, the server streams back. It rides on ordinary HTTP, so it works through firewalls and proxies that block WebSocket upgrades, and it carries a simple, well-defined wire format. As of June 2026, OpenAI, Anthropic and Google Gemini all deliver streaming tokens over SSE-style HTTP responses, so you are in good company and the parsing logic is reusable across providers.

WebSockets earn their place when you need genuine two-way, low-latency messaging — live voice, mid-stream interrupts where the user types while the model talks, or collaborative sessions with multiple participants. They are more work to operate (you manage your own reconnection and heartbeats) so do not reach for them just to stream text one way.

gRPC streaming is a service-to-service choice. If your gateway calls an internal inference service, gRPC over HTTP/2 gives you typed contracts and efficient binary framing. Browsers cannot speak raw gRPC, so it sits behind your edge, not in front of it.

Plain chunked HTTP is what SSE is built on. You can stream raw chunks without the data: event framing, but you lose the conventions — event types, reconnection IDs, the browser's native parser — for no real gain. Use SSE and get the framing for free.

Server architecture: async generators and StreamingResponse

The SSE wire format is deliberately simple. Each message is one or more lines beginning with a field name, followed by a blank line that terminates the event. The field you care about is data:. A minimal token event looks like this on the wire:

data: {"token": "Hello"}

data: {"token": " world"}

data: [DONE]

Note the blank line after each data: line — that is the event delimiter, and forgetting it is the most common reason a stream "does not work". On the server, you produce these frames from an async generator and hand it to Starlette's StreamingResponse with the correct headers. Here is a complete, working FastAPI endpoint that streams from an upstream LLM:

import asyncio
import json
from fastapi import FastAPI, Request
from fastapi.responses import StreamingResponse

app = FastAPI()

SSE_HEADERS = {
    "Content-Type": "text/event-stream",
    "Cache-Control": "no-cache",
    "Connection": "keep-alive",
    # Critical: tells nginx NOT to buffer the response body.
    "X-Accel-Buffering": "no",
}


async def llm_token_stream(prompt: str):
    """
    Replace the body of this loop with your provider's streaming
    call. Every major SDK (OpenAI, Anthropic, Gemini) and self-hosted
    server (vLLM, TGI) exposes an async iterator of token deltas.
    """
    async for delta in call_model_streaming(prompt):
        yield delta            # a string fragment, e.g. " world"


async def sse_generator(request: Request, prompt: str):
    try:
        async for token in llm_token_stream(prompt):
            # If the client has gone away, stop immediately.
            if await request.is_disconnected():
                break
            frame = json.dumps({"token": token})
            yield f"data: {frame}\n\n"
        else:
            # Loop completed normally — signal end of stream.
            yield "data: [DONE]\n\n"
    except asyncio.CancelledError:
        # Server is shutting down or the request was cancelled.
        # Let it propagate so upstream cleanup runs.
        raise


@app.post("/chat/stream")
async def chat_stream(request: Request):
    body = await request.json()
    prompt = body["prompt"]
    return StreamingResponse(
        sse_generator(request, prompt),
        media_type="text/event-stream",
        headers=SSE_HEADERS,
    )

Three things are doing the heavy lifting here. The media_type="text/event-stream" and the headers tell every layer of the stack that this is a stream, not a buffered body. The async for over the model SDK means each token is yielded the instant the model produces it — no batching. And the request.is_disconnected() check, covered in detail next, is what stops you wasting money on abandoned requests.

Backpressure and client disconnects

Two failure modes dominate streaming in production: the client that vanishes, and the client that cannot keep up. Both are about flow control, and both cost you money if ignored.

Detecting disconnects and cancelling generation

When a user closes the tab, navigates away or loses their connection, the underlying TCP socket closes — but your generator does not automatically know. If you keep iterating the model output, you keep paying the provider (or keep a self-hosted GPU slot busy) to generate tokens that go nowhere. Starlette exposes await request.is_disconnected() exactly for this. Polling it once per token, as in the code above, lets you break out of the loop the moment the client leaves.

For self-hosted inference and for providers that bill on output, you want to go one step further and actively cancel the upstream call so generation truly stops. The pattern is to run the upstream stream as a cancellable task and to close it when the client goes away:

async def sse_generator(request: Request, prompt: str):
    upstream = llm_token_stream(prompt)        # async generator
    try:
        async for token in upstream:
            if await request.is_disconnected():
                # Tell the model provider to stop generating.
                await upstream.aclose()
                break
            yield f"data: {json.dumps({'token': token})}\n\n"
        else:
            yield "data: [DONE]\n\n"
    finally:
        # Always release the upstream connection / GPU slot.
        await upstream.aclose()

Calling aclose() on the upstream async generator propagates a cancellation into the provider SDK, which closes the HTTP connection to the model. For most managed providers that is enough to stop the billed token meter; for vLLM and similar self-hosted servers it frees the decode slot so the next queued request can start sooner. This single pattern is one of the highest-leverage cost optimisations in any streaming LLM service.

Slow consumers and bounded queues

The opposite problem is a client that connects fine but reads slowly — a mobile user on a weak signal, or a browser tab throttled in the background. The model may produce tokens faster than the network can drain them. If you buffer everything in memory while you wait, a few thousand slow consumers can exhaust your server's RAM.

The clean fix is a bounded queue between the producer (the model loop) and the consumer (the SSE response). When the queue is full, the producer waits, which naturally throttles generation to the speed the client can actually consume. This is backpressure: the slow consumer applies pressure back up the chain rather than letting work pile up unboundedly.

async def sse_generator(request: Request, prompt: str):
    # maxsize caps how many tokens we hold in memory per stream.
    queue: asyncio.Queue = asyncio.Queue(maxsize=64)

    async def produce():
        upstream = llm_token_stream(prompt)
        try:
            async for token in upstream:
                # Blocks when the queue is full, applying backpressure.
                await queue.put(token)
        finally:
            await upstream.aclose()
            await queue.put(None)          # sentinel: stream finished

    producer = asyncio.create_task(produce())
    try:
        while True:
            if await request.is_disconnected():
                break
            token = await queue.get()
            if token is None:
                yield "data: [DONE]\n\n"
                break
            yield f"data: {json.dumps({'token': token})}\n\n"
    finally:
        producer.cancel()                  # stop generating on exit

With maxsize=64, each stream holds at most 64 buffered tokens. A slow client makes the producer block on queue.put(), which (for self-hosted inference) slows or pauses decode rather than racing ahead. Memory per connection stays bounded and predictable, which is the property you need to size capacity confidently.

Infrastructure pitfalls: proxies, timeouts and buffering

Your application code can be flawless and the stream can still arrive in one lump, because something between your app and the user is buffering it. This is the single most common streaming bug, and it is almost never in your code.

Response buffering

By default, nginx buffers proxied responses — it collects the body and forwards it in batches, which is efficient for static files and catastrophic for streams. You must turn it off for the streaming location:

# nginx — streaming location block
location /chat/stream {
    proxy_pass http://app_upstream;
    proxy_http_version 1.1;

    # Disable response buffering so tokens flush immediately.
    proxy_buffering off;
    proxy_cache off;

    # Keep the connection alive and idle-tolerant.
    proxy_read_timeout 3600s;
    proxy_send_timeout 3600s;

    # Don't let intermediaries buffer either.
    add_header X-Accel-Buffering no;
}

The X-Accel-Buffering: no header you set on the FastAPI response is nginx's own signal to skip buffering for that response; setting both the header and proxy_buffering off belt-and-braces the behaviour. If you sit behind a managed load balancer (an AWS Application Load Balancer, for example) or a CDN, find the equivalent setting — most CDNs buffer by default and need streaming or "no transform / no buffering" explicitly enabled for the route.

Idle and read timeouts

During the prefill phase a long prompt can take several seconds to produce the first token, and during decode there can be gaps between tokens. Many proxies and load balancers close a connection that has been idle for 30 or 60 seconds by default, killing your stream mid-answer. Raise the read/idle timeout on the streaming route to comfortably exceed your worst-case generation time, as shown above. A common belt-and-braces technique is to emit an SSE comment line (: keep-alive\n\n) every 15 seconds — it counts as activity to keep the connection warm, and clients ignore comment lines.

Connection limits, sticky sessions and HTTP/1.1

Browsers cap simultaneous HTTP/1.1 connections to the same origin at six. A user with several streaming tabs open can exhaust that budget and find new streams queued behind old ones. Serving over HTTP/2 removes the per-origin limit through multiplexing, so prefer HTTP/2 at the edge for streaming-heavy apps. Streaming itself does not require sticky sessions — each stream is a self-contained request — but if you keep any per-connection state in worker memory, you will need session affinity or, better, externalise that state so any worker can serve any stream.

Scaling to thousands of concurrent streams

An open stream spends almost all of its life idle — waiting for the next token from the model. That single fact decides your scaling story. Under an async server, an idle stream is just a suspended coroutine costing a file descriptor and a few kilobytes; one Uvicorn worker can hold thousands of them concurrently. Under a synchronous worker pool, every open stream pins a whole thread or process, and you run out in the low hundreds.

Model	Cost per open stream	Practical ceiling per process	Verdict for streaming
Async (Uvicorn / Hypercorn)	One coroutine + one socket	Thousands (FD-limited)	Use this
Sync worker pool (threads/processes)	One thread or process	Low hundreds	Avoid for streaming

So the first scaling lever is simply: run async. After that, the bottleneck moves off the web layer entirely and onto your inference capacity — how many tokens per second your provider quota or your GPU fleet can actually decode. The web tier holding ten thousand idle connections is cheap; the model generating for ten thousand active streams is not. Capacity-plan against inference throughput, not connection count.

Two operational details matter at scale. First, raise the file-descriptor limit (ulimit -n) on the host and in your process manager — each stream consumes at least one FD, and the default 1024 will cap you well before the event loop does. Second, scale horizontally by adding stateless workers behind your load balancer; because streams are independent and (if you have externalised state) any worker can serve any request, this scales linearly.

Graceful shutdown: draining in-flight streams

When you deploy a new version, you do not want to sever every active answer mid-sentence. A graceful shutdown stops accepting new connections, then waits a bounded grace period for in-flight streams to finish before terminating. Uvicorn handles the SIGTERM lifecycle, and on Kubernetes you set terminationGracePeriodSeconds long enough to cover a typical generation. Pair that with the disconnect handling above — clients whose streams are cut can reconnect — and rolling deploys stop being a visible outage.

The client side: consuming a stream in the browser

The browser ships a native SSE client, EventSource, and for the simplest cases it is delightful — it parses frames and reconnects for you. But it has two limits that disqualify it for most LLM apps: it is GET-only, and it cannot set custom headers such as Authorization. Since you usually need to POST a prompt and pass an auth token, the modern pattern is to use fetch with a POST body and read the streamed response through a ReadableStream reader, parsing SSE frames yourself.

async function streamChat(prompt, onToken) {
  const res = await fetch("/chat/stream", {
    method: "POST",
    headers: {
      "Content-Type": "application/json",
      "Authorization": "Bearer " + getToken(),
    },
    body: JSON.stringify({ prompt }),
  });

  const reader = res.body.getReader();
  const decoder = new TextDecoder();
  let buffer = "";

  while (true) {
    const { value, done } = await reader.read();
    if (done) break;

    buffer += decoder.decode(value, { stream: true });

    // SSE events are separated by a blank line.
    const events = buffer.split("\n\n");
    buffer = events.pop();           // keep the last partial frame

    for (const evt of events) {
      const line = evt.split("\n").find(l => l.startsWith("data:"));
      if (!line) continue;
      const data = line.slice(5).trim();
      if (data === "[DONE]") return;
      const { token } = JSON.parse(data);
      onToken(token);                // append to the UI as it arrives
    }
  }
}

// Usage: append each token to a chat bubble.
streamChat("Explain backpressure", (t) => {
  document.getElementById("answer").textContent += t;
});

The two subtle bits are the buffer and the partial-frame handling. Network chunks do not line up with SSE event boundaries — a single read() might give you one and a half events. By splitting on the blank-line delimiter and keeping the trailing fragment in buffer for the next iteration, you assemble whole frames correctly and never drop a half-arrived token. The TextDecoder with { stream: true } similarly handles multi-byte characters that straddle chunk boundaries — important for non-ASCII output, which matters a great deal for Hindi, Tamil, Bengali or any non-Latin script.

For reconnection, decide whether a dropped stream should resume or restart. Resuming requires the server to track what it already sent (SSE's id: field and the Last-Event-ID header exist for this); for chat, restarting the request is usually simpler and good enough. Whichever you choose, do not silently leave the user staring at a half-finished answer.

Observability: measuring what users feel

You cannot optimise streaming without measuring the right things, and the right things are not the metrics most APM tools default to. Total request duration is nearly useless for a stream. Instrument these instead:

• Time-to-first-token (TTFT) — from request received to the first data: frame flushed. This is the headline number for perceived speed. Track its p50 and p95.
• Inter-token latency — the gap between successive tokens. Spikes here mean the model, the queue or the network is stalling mid-answer.
• Tokens per second — the sustained decode rate the user actually receives, which can differ from the model's raw rate once your network and backpressure are in play.
• Stream completion rate — what fraction of streams reach [DONE] versus being abandoned. This directly quantifies wasted spend and validates your disconnect handling.

A small structured-logging line emitted at first token and at stream end, carrying these four numbers, is enough to build dashboards that tell you how the product actually feels — and to catch a proxy change that quietly reintroduced buffering.

The cost angle, briefly

Streaming intersects with cost in one specific, valuable way: cancelling generation on disconnect. Every token a managed provider generates is billed whether or not it reaches the user, and every token a self-hosted server decodes occupies a GPU that could be serving someone else. Wiring up is_disconnected() plus upstream aclose() means abandoned streams stop costing you almost immediately. On a service with a meaningful abandonment rate it is a direct, ongoing saving for an afternoon's work. For the wider picture of what inference actually costs and how to model it, see our breakdown of AI inference cost economics.

Dual-market note: it works the same in Mumbai and London

None of this is region-specific. The same FastAPI service, the same SSE format and the same nginx settings behave identically whether you deploy to AWS Mumbai (ap-south-1) for Indian users or AWS London (eu-west-2) for UK users. What does change by region is the network distance to your inference, which lands almost entirely in TTFT. If your model runs in-region — self-hosted in Mumbai for Indian traffic, or in London for UK traffic — first-token latency is lower than routing every request across continents to a single provider region. So the engineering is portable; the only regional decision is where your inference lives, and putting it close to your users is the cheapest TTFT win available. Teams serving both markets often run identical stacks in both regions and route by user location.

Conclusion and next steps

Streaming LLM responses well is mostly about the unglamorous edges: flush the headers, kill the buffering, detect the disconnect, bound the queue, run async, raise the timeouts, and measure TTFT instead of total latency. Get those right and a single modest server will hold thousands of concurrent streams while feeling instant to every user, in Bengaluru or Birmingham alike.

A sensible build order: start with the FastAPI StreamingResponse and the fetch-plus-reader client; add disconnect detection and upstream cancellation next, because that is where the cost savings live; then fix your proxy buffering and timeouts and test through the full edge path; and finally add the four observability metrics so you can see what you have built. With those in place, you have a streaming layer that scales, saves money on abandoned requests, and feels fast — which is the whole point.