Agent Memory That Scales: Storage, Forgetting and Retrieval

What good agent memory actually is

An agent with a big context window is not an agent with a good memory, any more than a person who can hold a long conversation has a good memory of last week's meeting. Memory is what survives between turns, between sessions, and between deployments — and getting it right is now a production engineering discipline in its own right, with its own benchmarks and a growing research literature. As of mid-2026, roughly twenty-one memory frameworks and platforms exist, and no single one has won. Most teams shipping reliable agents build across several.

The central claim of this guide is simple and slightly counter-intuitive: reliability comes from rigorous context management, not bigger windows. Context engineering is not a temporary workaround that a future 10M-token model will make obsolete; it is a permanent discipline. The more an agent can hold, the more it can hold wrongly — and a context window stuffed with stale, duplicated or irrelevant history degrades reasoning just as surely as one that is too small. This piece walks through the memory types that matter, the storage back-ends that hold them, why deliberate forgetting is a feature rather than a bug, how to diagnose retrieval quality before you reach for a bigger model, and how to actually test the whole thing.

Pre-requisites: what to have in place first

You do not need a memory framework to start. You need three things. First, a golden set — a small, hand-curated collection of facts and past interactions your agent ought to recall, paired with the queries that should surface them. Fifty to two hundred items is plenty to begin. Second, a way to log every retrieval: what was asked, what came back, and whether the right item was in the result. Third, a cost and latency budget per task, because memory decisions are economic decisions. If you have only a vague sense that the agent "sometimes forgets things", you cannot improve it; you can only thrash.

If you are building an agent from scratch, our companion guide on building a local AI agent with Ollama and MCP is a sensible place to start before layering memory on top. Memory is something you add to a working loop, not something you design in the abstract.

The five memory types every agent has (whether you designed them or not)

Borrowing loosely from cognitive science, it helps to name the distinct kinds of memory an agent uses. They have different lifetimes, different access patterns and different ideal storage back-ends, so conflating them is the root of a lot of pain.

Working (short-term) memory

This is the live context window: the current conversation, the tool outputs from this task, the scratchpad the model is reasoning over right now. It is fast, expensive per token, and volatile — it vanishes when the session ends. The discipline here is keeping it lean. Everything that does not need to be in front of the model this turn should be offloaded.

Long-term factual memory

Durable facts about the world and the user: "the customer's billing currency is GBP", "the deployment region is AWS Mumbai", "the contract renews in March". These are the things you want to be auditable and exactly correct, which is why a relational store usually wins for them.

Episodic memory

Records of past interactions and events: what happened in the support thread three weeks ago, which approach the agent tried last time and how it went. Episodic memory is inherently relational and temporal — "what did we discuss, in what order, and what followed from it" — which is exactly what a graph traverses well.

Procedural memory

Skills and how-to knowledge the agent has accumulated: the reliable sequence of steps to reconcile an invoice, the tool-call pattern that works for this API. Procedural memory often emerges from successful episodes being rolled up into reusable routines.

Semantic memory

General, mostly static knowledge — documentation, policies, reference material. This is the classic RAG corpus, and it is where vector similarity earns its keep.

Storage back-ends and when to reach for each

There is no single correct database for agent memory, and the most common production shape as of mid-2026 is a hybrid stack — typically Postgres with pgvector for the bulk of it, plus a graph store where relationships matter. The table below is the decision aid we hand to teams.

Back-end	Best for	Strength	Weakness	When to reach for it
Vector DB / pgvector	Semantic + factual memory	Similarity recall over fuzzy, unstructured text; "find me things like this"	No notion of relationships; ANN recall is approximate; can return plausible-but-wrong neighbours	Most retrieval. Start here. pgvector on managed Postgres in AWS Mumbai or London keeps it one database.
Graph DB	Episodic + procedural memory	Fast multi-hop relationship traversal; "what connects to what, and in what order"	Operational overhead of a second store; modelling discipline required; weaker at fuzzy text match	When the value is in connections — who said what to whom, which step followed which.
SQL / Postgres	Long-term factual memory	ACID guarantees, auditability, exact lookups, mature tooling and access control	Poor at semantic or fuzzy recall on its own	Facts that must be exactly right and reviewable — and the system of record behind everything else.

Notice that pgvector appears as both a vector store and part of a Postgres deployment. That is the pragmatic point: for many teams the answer is one managed Postgres instance with the vector extension, audited facts in ordinary tables, and a graph store added only when relationship traversal becomes the bottleneck. Adding databases adds operational surface; earn each one.

Hierarchical memory and dynamic forgetting

Here is the trade-off that breaks naive designs. Storing full history feels safe — keep everything, retrieve later. In practice it explodes both cost and retrieval noise. Every extra memory is another candidate the retriever must rank, another near-duplicate that crowds out the one that matters, and another token you pay to embed and store. Past a certain volume, adding memories makes recall worse. The discipline that fixes this has three parts: hierarchy, importance scoring, and forgetting.

Hierarchy: roll the old up into summaries

Rather than keeping a thousand raw turns, keep the recent turns verbatim, summarise older ones into mid-level summaries, and roll those up again into a compact long-term gist. This is hierarchical summarisation, and it mirrors how people remember: you recall yesterday in detail, last month in themes, and last year in headlines. A sliding window keeps the freshest turns live; older content is offloaded to the external store as progressively coarser summaries. Retrieval can then pull a precise recent memory or a broad summary depending on the query — and fall back to the gist when nothing specific matches.

Importance scoring: not all memories are equal

At write time, assign each memory an importance weight. "The user's name is Anika" is high-importance and should resist decay; "the user said 'ok' at 14:32" is low-importance and can fade fast. Importance can be heuristic (does it contain an entity, a preference, a decision?) or model-scored. It becomes a first-class factor in retrieval ranking.

Forgetting: temporal decay and retention policies

Dynamic forgetting combines a temporal decay function (older memories lose weight unless reinforced), relevance scoring (memories never retrieved are candidates for eviction), and user-defined retention policies (this class of memory lives 30 days, that one is permanent). The slogan worth internalising: not remembering everything is a feature. Forgetting is what keeps recall sharp, cost bounded, and — as we will see — compliance tractable.

A hierarchical retrieval function you can actually ship

Most of the theory above collapses into one scoring function: rank candidate memories by semantic_similarity × recency_decay × importance, apply a threshold, and fall back to a rolled-up summary when nothing clears the bar. Here is a compact, dependency-light version you can adapt. (Code is in US English by convention.)

import math
import time
from dataclasses import dataclass

@dataclass
class Memory:
    id: str
    text: str
    embedding: list[float]
    importance: float      # 0..1, assigned at write time
    created_at: float      # epoch seconds

def cosine(a: list[float], b: list[float]) -> float:
    dot = sum(x * y for x, y in zip(a, b))
    na = math.sqrt(sum(x * x for x in a)) or 1e-9
    nb = math.sqrt(sum(y * y for y in b)) or 1e-9
    return dot / (na * nb)

def recency_decay(created_at: float, half_life_days: float = 14.0) -> float:
    """Exponential decay: a memory is worth half as much after `half_life_days`."""
    age_days = (time.time() - created_at) / 86_400.0
    return 0.5 ** (age_days / half_life_days)

def retrieve(
    query_embedding: list[float],
    memories: list[Memory],
    summary_fallback: str,
    k: int = 5,
    threshold: float = 0.30,
) -> list[str]:
    scored = []
    for m in memories:
        sim = cosine(query_embedding, m.embedding)   # semantic similarity
        if sim <= 0:
            continue
        score = sim * recency_decay(m.created_at) * (0.5 + m.importance)
        scored.append((score, m))

    scored.sort(key=lambda pair: pair[0], reverse=True)
    top = [m.text for score, m in scored[:k] if score >= threshold]

    # Fall back to the rolled-up summary when nothing is relevant enough.
    return top if top else [summary_fallback]

Three details matter more than the exact arithmetic. The threshold is what stops the agent from injecting weakly-related memories that add noise — tune it on your golden set, do not guess it. The importance floor (0.5 + importance) ensures a high-importance memory is never zeroed out by recency alone. And the summary fallback means a cold or irrelevant query still returns useful context instead of an empty list. In production you would push the similarity step into pgvector and only re-rank the top candidates in application code, which is both faster and cheaper than scoring every memory in Python.

Cost and latency: why top-k beats dumping history

The economic case for all of this is stark. Consider an agent that, on every call, prepends 200,000 tokens of raw history versus one that retrieves the top eight memories — perhaps 2,000 tokens — from an external store. The token volume differs by two orders of magnitude, and so does the bill and the latency.

Approach	Context tokens / call	Relative input cost	Effect on latency	Effect on reasoning
Dump full history	~200,000	~100×	High — large prompts are slower to process	Degrades — relevant facts buried in noise
Retrieve top-k memories	~2,000	~1× (baseline)	Low — small, focused prompt	Improves — only relevant context present

Retrieval quality: the diagnostic that saves your budget

This is the single most important habit in the whole discipline. Before you expand the context window or upgrade the model, measure your retrieval hit rate. Run your golden set through the agent and, for each query, check whether the memory that should have surfaced actually did. If the agent is missing memories that exist in the store, the bottleneck is retrieval quality, not window size — and a bigger window will cost you more while fixing nothing.

When retrieval is the problem, the levers are well understood. Tighten or loosen your relevance threshold. Improve write hygiene with deduplication, so near-identical memories do not split the vote and crowd each other out. Re-rank the retrieved set with a cross-encoder or a hosted reranker before it goes to the model — our guide on RAG reranking with cross-encoders, ColBERT and hosted rerankers covers this in depth and applies directly to memory retrieval. And design clean read and write policies: decide what is worth writing at all, and what is worth reading back. The tooling that surrounds your memory store — how the agent calls it — matters too; our guide on designing tools for AI agents with good schemas, errors and retries is a useful companion when memory is exposed to the model as a tool.

How to evaluate a memory system honestly

A memory system you cannot measure is a memory system you cannot improve. Evaluate on four axes, and never on just one.

• Recall@k on a golden set. Of the facts and interactions that should be retrievable, what fraction appear in the top-k results? This is your core retrieval-quality number. Track it as you change thresholds, decay rates and rerankers.
• End-to-end task success. Recall@k can look healthy while the agent still fails the job, because retrieval is necessary but not sufficient. Score whole tasks on realistic inputs, not just the retrieval step.
• Cost per task. A memory design that wins on recall but doubles token spend may not be worth shipping. Put a money figure on each task and watch it move.
• p95 latency. Median latency hides the tail that users actually feel. A memory system that is fast on average but occasionally takes four seconds to retrieve will be remembered for the four seconds.

These four pull against each other, which is the point — improving recall by retrieving more can raise cost and latency and even hurt task success by adding noise. Holding all four in view is what separates a memory system that demos well from one that survives production. If you are building out evaluation discipline more broadly, the principles in our guide on RAG evaluation with RAGAS, faithfulness and context precision transfer cleanly to memory: context precision, in particular, is just recall@k wearing a different hat.

Common pitfalls

• Reaching for a bigger window first. The most expensive mistake in the field. Measure retrieval before you scale context — the bottleneck is usually recall, not capacity.
• Storing everything "to be safe". Full history is a cost and noise generator, and a compliance liability. Forgetting is a feature; design it in.
• One database for every memory type. Forcing episodic, semantic and factual memory into a single store means none of them is served well. Type your memories and route them.
• No deduplication. Near-duplicate memories split retrieval votes and crowd out the canonical fact. Merge on write.
• Ignoring erasure. If you cannot trace and delete a subject's personal data across summaries and embeddings, you are exposed under UK GDPR and India's DPDP Act. Make every memory deletable by subject.
• Evaluating on a single axis. A great recall@k that blows the latency or cost budget is not production-ready. Hold all four metrics together.

Next steps

Start small and empirical. Build a golden set this week. Instrument retrieval so every miss is visible. Put your memories behind a single typed schema, route them to the right back-end, and add a graph store only when traversal queries demand it. Add importance scoring and a temporal-decay forgetting policy before your store grows past the point where recall starts to slip. And measure recall@k, task success, cost and p95 latency together, every time you change the design.

The research foundation here is worth reading directly. The Mem0 work on scalable long-term memory for agents — arXiv 2504.19413 — and the broader "State of AI Agent Memory 2026" surveys are good primary sources, and a reminder that this is now a measurable engineering discipline rather than a dark art. As of mid-2026 the field has roughly twenty-one frameworks competing and no clear winner, so treat any single tool as a component, not a religion. The durable skill is not the framework; it is knowing what to store, what to forget, and how to prove your retrieval works.

None of this is provider-specific, and all of it outlives the next model release. A 10M-token window will not retire context engineering — it will raise the stakes of doing it badly. The teams that win are the ones that treat memory as something to be designed, measured and pruned, not merely accumulated.