Semantic Caching for LLM Apps: Cut Token Spend 40-80%

What semantic caching actually is

Most LLM applications answer the same questions over and over, phrased slightly differently each time. A support assistant fields "how do I reset my password", "I forgot my password" and "can't log in, need a new password" as three distinct requests, and pays for three full inferences to produce three near-identical answers. Semantic caching removes that waste. It embeds each incoming query as a vector, cosine-matches that vector against the vectors of queries it has already answered, and — if the closest stored query is similar enough — serves the response it saved last time, without calling the model at all.

The loop is short: embed the query, search the vector store for the nearest neighbour, compare the similarity score against a threshold, and either serve the stored answer on a hit or run inference and write the fresh result back on a miss. On a hit you pay for one small embedding call and a millisecond of vector search instead of a full generation. That is the whole idea, and it is durable precisely because it does not depend on any particular price sheet: it is a property of how repetitive real traffic is, not of what a token costs this quarter.

The word that matters is semantic. This is not the same as the caching most builders already know. It is worth drawing the line against its two relatives clearly, because they are frequently muddled and they save money in completely different ways.

The first relative is provider prompt caching — sometimes called KV reuse — which we cover in depth in our guide to prompt caching across Claude, GPT and Gemini. That technique reuses the processed prefix inside the model, the attention key-value state, so the provider does not reprocess your big system prompt on every call and bills those cached input tokens at up to roughly 90 percent off. Crucially, prompt caching still runs the model on every request; it just makes the input cheaper. Semantic caching, by contrast, skips the model entirely on a hit. That is the fundamental difference: prompt caching discounts a call you still make, while semantic caching removes the call.

The second relative is exact-match caching, the classic key-value cache that hashes the full request string and returns a stored response only when the next request is byte-identical. It is trivial to build and carries almost no risk, but it barely fires, because natural-language queries rarely repeat verbatim. Semantic caching is what you reach for when you want the exact-match cache's total saving — no model call at all — but with a hit condition loose enough to catch paraphrases. The table below lines up all three.

Approximate, as of mid-2026. Savings, hit rates and thresholds vary by workload, embedding model and traffic pattern; treat every figure as a shape to validate, not a guarantee.

When it pays off — and when it doesn't

Semantic caching is not a universal win, and the fastest way to waste a fortnight is to bolt it onto traffic that never repeats. The technique pays when questions rhyme — when a meaningful slice of your requests are semantically equivalent to earlier ones, even if worded differently. That describes a large share of real production workloads.

It shines on repetitive question-and-answer surfaces: customer-support assistants, internal help desks, documentation bots, and the frequently-asked-question tier of a retrieval-augmented-generation system, where a long tail of users ask the same forty things about pricing, onboarding or returns. It shines on classification and extraction where inputs cluster — an Indian fintech tagging KYC queries or a UK insurer triaging claims emails will see the same handful of intents recur thousands of times a day. And it shines anywhere a public or semi-public surface faces bursty, correlated traffic, such as a launch-day FAQ where ten thousand people ask three questions.

It does not pay, and can actively harm, where each request is genuinely unique or must be fresh. Highly personalised generation — a bespoke cover letter, a per-user summary of that user's own private data, a creative draft meant to differ every time — has almost no reusable surface, and any hit you do get is a bug, not a saving. Anything real-time or account-specific — a balance, an order status, a live price — must never be served from a stale cache. And long, open-ended agentic conversations where every turn depends on a growing, unique history rarely present the same state twice.

There is also an architectural fork worth naming. If your instinct is to cache retrieved passages so you stop paying to stuff them into context, first decide whether you even need retrieval at that scale — our guide on long context versus RAG walks through when a large context window replaces the retrieval pipeline entirely. Semantic caching sits on top of whichever answer you reach: it caches the final response to a repeated question, regardless of whether that answer came from RAG, a long-context read or a plain prompt.

How to build it: embed, store, serve

A semantic cache has three moving parts: an embedding model that turns text into a vector, a vector store that finds the nearest neighbour fast, and a threshold that decides whether the nearest neighbour is near enough to trust.

For the embedding model, pick a small, cheap, fast one — the cache is only worth building if the embedding call costs a tiny fraction of the inference it replaces. A hosted small embedding model works, but many teams self-host an open-source embedder precisely so the cache lookup adds no per-query API cost and no network hop. For Indian and UK builders that self-hosting choice doubles as a data-residency and latency win: run the embedder and the vector store together in AWS Mumbai (ap-south-1) or London (eu-west-2) and the query text, the cached answers and the whole lookup stay in-region, close to your users, rather than round-tripping to another continent on every request.

For the vector store, you have good open-source options and do not need a heavyweight system for a cache. Redis with its vector search is a natural fit because a cache is exactly the low-latency, TTL-friendly workload Redis was built for, and Redis's own LangCache productises this pattern. If your data already lives in Postgres, pgvector keeps the cache in the same database you already operate — our comparison of pgvector, Qdrant, Pinecone and Weaviate covers the trade-offs when the cache grows large enough to matter. And GPTCache is a purpose-built open-source semantic-caching library that wires the embed-search-serve loop together for you if you would rather not assemble it by hand.

The loop itself is short. Here it is in Python, deliberately explicit so every decision is visible:

import numpy as np
from openai import OpenAI

client = OpenAI()
EMBED_MODEL   = "text-embedding-3-small"
SIM_THRESHOLD = 0.92          # cosine band: 0.90-0.95, tuned on your data

def embed(text):
    v = client.embeddings.create(model=EMBED_MODEL, input=text).data[0].embedding
    return np.asarray(v, dtype=np.float32)

def cosine(a, b):
    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

# vector_store.search() returns the nearest neighbour: (response, score, meta)
def answer(query, prompt_version="v7", model_version="claude-opus-4-8"):
    q_vec = embed(query)
    hit = vector_store.search(
        q_vec, k=1,
        namespace=f"{model_version}:{prompt_version}",   # version isolation
    )
    if hit and hit.score >= SIM_THRESHOLD and not is_stale(hit.meta):
        metrics.incr("semantic_cache.hit")
        return hit.response          # served in <5ms, no model call

    metrics.incr("semantic_cache.miss")
    response = call_llm(query, prompt_version, model_version)   # full inference
    vector_store.upsert(
        q_vec, response,
        namespace=f"{model_version}:{prompt_version}",
        meta={"query": query, "created_at": now(), "ttl": 86_400},
    )
    return response

Three details in that loop carry most of the weight. The namespace isolates entries by prompt and model version, so a prompt edit or a model swap cannot serve you answers from a system that no longer exists. The SIM_THRESHOLD comparison is the entire safety mechanism — everything below turns on getting it right. And the hit/miss counters are non-negotiable: a cache you cannot measure is a cache you cannot trust, and instrumenting from the first line is far easier than retrofitting it after a false hit reaches a user.

Tuning the threshold: the 0.90–0.95 band

The similarity threshold is the single most important number in a semantic cache, and it is a precision-recall trade-off, not a constant you can copy from a blog. Most production deployments settle in a cosine band of roughly 0.90 to 0.95, but the right point inside that band depends on your embedding model and your tolerance for a wrong answer.

The trade-off runs in two directions. Set the threshold too low — say 0.82 — and the cache fires on questions that merely look similar in embedding space but mean different things, serving a stored answer to a query it does not actually match. That is a false hit, and it is the failure mode that turns a cost optimisation into a correctness bug. Set the threshold too high — say 0.99 — and you reject genuine paraphrases, so the hit rate collapses towards the exact-match cache's, the savings evaporate, and you have paid the embedding overhead for nothing.

The disciplined way to choose is empirical. Assemble a small labelled set: pairs of queries you consider equivalent (they should hit) and pairs you consider different (they should miss), drawn from your own logs. Sweep the threshold across the 0.90 to 0.95 band, and at each value measure how many equivalent pairs you correctly serve (recall) against how many different pairs you wrongly serve (false-hit rate). Pick the highest recall you can reach while keeping false hits inside your tolerance — which for anything customer-facing should be low. Because the geometry of the space is set by the embedding model, this whole exercise has to be re-run whenever you change embedders; a threshold tuned for one model is meaningless for another.

Staleness and invalidation

The second risk after false hits is staleness. A cached answer is frozen at the moment it was written, while the facts behind it keep moving. Cache "what is your refund window" today, change the policy next week, and without invalidation the cache cheerfully serves the old window until something forces it out. Three controls keep a cache honest.

The first is a time-to-live on every entry, set to match how fast the underlying facts change. Prices and stock levels might warrant minutes; policy and documentation answers, days; stable reference material, longer still. A short TTL trades some hit rate for freshness, and that is usually the right trade for anything a user acts on.

The second is versioning. Namespace every entry by the prompt version and the model version that produced it — exactly the namespace key in the loop above. When you edit the prompt or upgrade the model, the new namespace has no entries, so the old cache is orphaned instantly rather than serving answers from a configuration you have retired. This is what stops a "we improved the system prompt" change from being silently undone by a cache still serving the pre-improvement responses.

The third is event-driven busting. When a source document, a price or a policy changes, actively delete or invalidate the cache entries derived from it rather than waiting for the TTL to lapse. This needs a link from your content to the cache keys — tag entries with the source identifiers that fed them, so a content update can fan out to the exact entries it makes stale. Here is the versioning and guardrail scaffolding in practice:

import random

# Every entry is namespaced by what actually determines the answer.
def namespace(model_version, prompt_version):
    return f"{model_version}:{prompt_version}"

def is_stale(meta):
    return (now() - meta["created_at"]) > meta["ttl"]

# Bust every entry derived from a source that just changed.
def invalidate_source(source_id):
    vector_store.delete(filter={"source_id": source_id})

# Shadow-mode guardrail: sample a slice of served hits and have a judge
# model score the cached answer against a fresh generation. Alert on drift.
def audit_hit(query, cached_answer):
    if random.random() > 0.02:          # sample 2% of served hits
        return
    fresh   = call_llm(query)
    verdict = judge(cached_answer, fresh)   # 0.0 (disagree) - 1.0 (agree)
    metrics.record("semantic_cache.hit_quality", verdict)
    if verdict < 0.7:
        log.warning("low-quality cache hit", query=query)

Note the rule the code implies but does not enforce for you: never cache anything genuinely personalised or real-time. An account balance, an order status, a user's own private summary — none of it belongs in a shared semantic cache, because the whole point of the cache is to reuse an answer across users, and those answers must not be reused across users at all.

Measuring it honestly

Vendor headlines and production reality diverge sharply here, and the gap is where most disappointment lives. Published figures put the achievable saving at roughly 40 to 80 percent off cost with up to about 250 times faster responses on hits. Redis reports its LangCache service cutting cost by up to 73 percent, and a GPT Semantic Cache study cutting API calls by up to 68.8 percent at hit rates between roughly 61.6 and 68.8 percent. Those numbers are real, but they come from favourable, repetitive workloads and represent a ceiling.

The number that matters is your own hit rate on your traffic, and independent production write-ups are blunt about it: a realistic cache serves roughly 20 to 45 percent of traffic, not the 95 percent sometimes claimed, yet a 30 to 40 percent hit rate still yields meaningful savings, and cache hits come back in under 5 milliseconds against 2 to 5 seconds for a full inference. So plan for 20 to 45 percent, be pleased if you land higher, and never size a business case on the vendor ceiling.

Three things belong on a dashboard from day one. First, the true hit rate — hits over total requests — tracked over time, so you can see it move when you change the threshold or the traffic mix shifts. Second, cost and latency saved: multiply hits by the inference cost and latency they displaced, so the saving is a monitored number rather than a hopeful estimate. Third, and most important, a hit-quality metric: sample a small percentage of served hits, have a judge model or an embedding-agreement check score the cached answer against a fresh generation, and alert if the agreement rate drifts down. That last metric is what separates a cache you can trust from one that is quietly serving wrong answers to save money.

Roll it out in shadow mode first. Run the cache alongside live inference without serving its results — log what it would have returned, and compare that against the real generation for every request. Only once the shadow false-hit rate sits inside your tolerance do you start serving hits to real users. It is the cheapest possible way to discover that your threshold is too loose before a customer does.

Pitfalls and a rollout checklist

A handful of mistakes account for most failed semantic caches. The threshold set by vibes rather than measured against labelled data, so it either false-hits or never fires. An embedding model swapped without re-tuning the threshold, silently changing the geometry the whole cache depends on. No versioning, so a prompt or model change keeps serving pre-change answers. No TTL, so stale answers linger indefinitely. Caching personalised or real-time content that should never have been shared across users. And shipping straight to production with no shadow phase and no hit-quality monitor, so the first sign of trouble is a support ticket about a wrong answer.

The rollout that avoids all of that follows a fixed order:

1. Confirm the traffic repeats. Sample your logs and estimate how many requests are semantically equivalent to earlier ones. If it is a small fraction, stop here — the cache is not worth it.
2. Pick a small, cheap embedder and a vector store you already operate (Redis or pgvector), and self-host in-region — Mumbai or London — for latency and data residency.
3. Tune the threshold on a labelled set of equivalent and non-equivalent query pairs, sweeping the 0.90 to 0.95 band for the best recall inside your false-hit tolerance.
4. Add TTLs and version namespaces so entries expire and orphan correctly, and wire event-driven busting for anything backed by changeable source content.
5. Run in shadow mode, comparing would-be hits against fresh generations, until the false-hit rate is acceptable.
6. Instrument hit rate, saving and hit quality on a dashboard, then serve live hits and watch the numbers for a week before trusting them.

The bottom line for builders

Semantic caching is the cost lever that removes calls rather than discounting them, and on repetitive traffic that makes it one of the highest-leverage optimisations available — a plausible 40 to 80 percent off token spend and answers returned in milliseconds. It is also the lever with the most teeth: unlike provider prompt caching, which never changes what the model says, a semantic cache can serve a subtly wrong answer if the threshold is loose or an entry has gone stale. The two techniques are partners, not substitutes — cache semantically to skip the calls you can, and let prompt caching discount the calls that still land on the model.

The durable skill is not the plumbing but the discipline around it: tune the threshold against real data in the 0.90 to 0.95 band, version and expire every entry, judge a sample of live hits, and measure your own hit rate instead of trusting the vendor's. Build it that way and it keeps paying out on every repeated question for as long as your traffic rhymes — which, for most production applications, is a very long time.

Sources

• Maxim — Semantic caching for LLMs: how to cut token spend with AI gateways (Redis LangCache, GPT Semantic Cache figures)
• Tian Pan — Semantic caching in LLM production (realistic hit rates, latency)
• Percona — Semantic caching for LLM apps: reduce costs 40-80% and speed up by 250x