Context Engineering: Compaction for Long-Running AI Agents

What you need to know

• Drift, not exhaustion, is the failure mode. Quality degrades long before the window is full — the effect Chroma named context rot. Managing the ceiling is not the same as managing the signal.
• Compaction is context engineering's first lever. Distil the window into a high-fidelity summary and continue in a fresh one, so the agent keeps its plot across hours of work.
• Anchor before you summarise. Pin the task, success criteria and key artefacts so they survive every pass verbatim; iterate on a running summary rather than re-summarising from scratch.
• Budget the window like a cache. When it fills, evict the lowest value-per-token segments to external storage — don't blindly truncate the oldest turns.
• Compact and offload are complementary. Small, always-needed state gets compacted; large, retrieve-on-demand material goes to external memory. Most real agents use both.
• Measure task success, not token count. A shorter context that forgets the goal is worse than a longer one that keeps it. Evaluate on long-horizon tasks.

Why drift, not exhaustion, degrades long agents

The intuitive story is that agents fail when they run out of room: the transcript grows, hits the token ceiling, and the request is truncated or rejected. That does happen, and it is easy to guard against. The harder, quieter problem is that a long-running agent gets worse long before it runs out of room — and it does so silently, with no error to catch.

Two lines of research explain why. The first is Lost in the Middle (Liu et al., 2023), which showed that models use their context unevenly: accuracy is highest when the relevant information sits near the start or the end of the input and dips markedly when it is buried in the middle. As you concatenate more turns, tool output and retrieved documents, the facts an agent needs migrate into exactly that low-recall middle band.

The second is context rot, formalised by Chroma's 2025 study. Across 18 production models — including frontier Claude, GPT and Gemini variants — accuracy fell monotonically as input length grew, on multi-hop reasoning as well as retrieval. A model advertised with a 200K-token window can show meaningful degradation well before it, sometimes tens of percentage points down at a fraction of the stated limit. One of the study's more uncomfortable findings for agent builders: coherent, well-structured input can degrade attention more than shuffled input, so simply piling in a tidy transcript is not free.

Put those together and you get context drift: as the window fills, the original goal gets diluted, early constraints slip into the unreliable middle, and the model starts answering from a blurred, generic reconstruction of the task rather than the exact thing you asked for. As of mid-2026, typical frontier windows run from roughly 200K to 1M tokens, with a couple stretching to 2M — but a bigger window mostly buys you more room to drift, not immunity from it. The window is a budget, not a filing cabinet.

Anchored iterative summarisation

Compaction, in Anthropic's framing of effective context engineering, means taking a conversation nearing its limit, summarising it faithfully, and reinitialising a fresh window with that summary so the agent continues with minimal loss. Done naively — "summarise the whole transcript every time it gets long" — it introduces its own drift. Each pass silently discards low-frequency details; a preference the user stated on turn three becomes generic language by turn twenty-five, and the agent confidently forgets it.

Two disciplines make the difference. First, anchoring: a small set of turns — the task statement, the success criteria, key artefacts and identifiers — are pinned and reproduced verbatim in every compaction. They never pass through the lossy summariser. Second, iteration: instead of re-summarising raw history each time, you fold new material into the previous summary, so the model refines a stable structure rather than rebuilding it from scratch. This is closer to how the human works next to the agent: you keep the brief on the desk and update your notes, you do not re-read the whole thread every hour.

Here is a compact, provider-agnostic routine. llm.complete() is any chat model — Claude, GPT, Gemini, an open-weight model behind vLLM. Nothing here is vendor-specific.

from dataclasses import dataclass

@dataclass
class Turn:
    role: str            # "user" | "assistant" | "tool"
    content: str
    pinned: bool = False # anchors that survive every compaction, verbatim

COMPACT_SYSTEM = (
    "You are compacting an agent transcript so work can continue in a fresh "
    "context window. Preserve verbatim: the task and success criteria, decisions "
    "and why they were made, file paths, IDs, exact values and error messages, "
    "and the next planned action. Drop greetings, resolved dead ends and "
    "duplicated tool output. Reply under these headings, and only these: "
    "GOAL, STATE, DECISIONS, ARTEFACTS, OPEN, NEXT."
)

def compact(turns, llm, keep_recent=6, prev_summary=None):
    """Anchored iterative summarisation.

    Anchors (pinned turns) and the last `keep_recent` turns stay in full
    fidelity. Everything between them is folded into a running summary that is
    fed back in on the next pass, so each compaction refines rather than
    restarts. Returns a new, shorter turn list.
    """
    pinned  = [t for t in turns if t.pinned]
    movable = [t for t in turns if not t.pinned]
    if len(movable) <= keep_recent:
        return turns                      # nothing worth compacting yet

    recent = movable[-keep_recent:]
    middle = movable[:-keep_recent]

    transcript = "\n".join(f"[{t.role}] {t.content}" for t in middle)
    prompt = [{"role": "system", "content": COMPACT_SYSTEM}]
    if prev_summary:
        prompt.append({"role": "user",
                       "content": f"Running summary so far:\n{prev_summary}"})
    prompt.append({"role": "user",
                   "content": f"New transcript to fold in:\n{transcript}"})

    summary = llm.complete(prompt)
    summary_turn = Turn(role="assistant",
                        content=f"[COMPACTED SUMMARY]\n{summary}",
                        pinned=True)
    # anchors first, then the running summary, then full-fidelity recent turns
    return pinned + [summary_turn] + recent

Three details earn their keep. The rigid heading set (GOAL, STATE, DECISIONS, ARTEFACTS, OPEN, NEXT) gives the summariser a schema, which resists the slow slide into vague prose. Keeping the last few turns verbatim means the agent's immediate working set — the file it just edited, the error it just saw — never goes through compression. And feeding the prior summary back in on every call is what makes it iterative: the running summary becomes a durable spine that new work attaches to.

Treat the token budget like a cache-eviction problem

Summarisation answers "how do I shrink history?" but there is an earlier question: "what deserves to be in the window at all?" Here the operating-systems analogy from MemGPT (Packer et al., 2023) is the right mental model. The context window is RAM — small, fast, scarce, directly usable. External stores are disk — large, slow, retrieved on demand. Managing an agent's window is a paging problem, and paging is fundamentally about eviction.

Naive truncation — drop the oldest turns — is the agent equivalent of a blind FIFO cache. It is cheap and it is wrong, because age is a poor proxy for value. The task statement is the oldest turn in the transcript and the last one you want to evict. A better policy scores each segment by how much value it is likely to add per token it costs, then evicts the worst first — and evicts to external storage rather than deleting, so nothing is truly lost. This is the spirit of cache-replacement policies like GDSF (greedy dual-size frequency), which weigh size against value and recency rather than age alone.

# Treat the window as a fixed-size cache. Each segment has a size (tokens) and
# a value (how likely it is to matter next). Evict lowest value-per-token first,
# spilling to external memory instead of deleting.
import time

def score(segment, now):
    # relevance: retrieval / task similarity, 0..1
    # weight:    importance class (a decision > a tool echo)
    # recency:   decays over minutes since last used
    recency = 1.0 / (1.0 + (now - segment["last_used"]) / 60.0)
    return segment["relevance"] * segment["weight"] * recency

def evict_to_budget(segments, budget_tokens, now=None):
    now  = now or time.time()
    kept = [s for s in segments if s["pinned"]]
    used = sum(s["tokens"] for s in kept)

    movable = [s for s in segments if not s["pinned"]]
    # highest value-per-token first
    movable.sort(key=lambda s: score(s, now) / max(s["tokens"], 1),
                 reverse=True)

    evicted = []
    for s in movable:
        if used + s["tokens"] <= budget_tokens:
            kept.append(s)
            used += s["tokens"]
        else:
            evicted.append(s)   # spill to external memory; do not delete
    return kept, evicted

The two routines compose. Eviction decides what stays resident this step; compaction decides how the resident history is represented. In a real loop you run eviction on every turn to hold the budget, and fire compaction only when the resident set is still too heavy after eviction — the expensive summariser call is the fallback, not the first move. MemGPT's own design applies the same instinct: it warns at around 70% of capacity and forces a flush at 100%, generating a recursive summary of evicted messages so information degrades gracefully rather than vanishing.

Compact, or offload to external memory?

Three strategies dominate production agents, and the mistake is treating them as rivals. They solve different problems and the best systems layer them. A sliding window keeps the last N turns and drops the rest — predictable and free, but brutal: at 500 messages with a 20-message window, 96% of the conversation is gone. Summarisation preserves the plot at the cost of fine detail and some latency. External memory pushes material to a searchable store the agent pages in on demand — unbounded and durable, but only as good as its retrieval.

Strategy	How it works	Pros	Cons	Use when
Sliding window	Keep the last N turns, drop older ones	Zero extra cost or latency; predictable token use; trivial to implement	Permanently loses early context; forgets the goal on long jobs	Short chats (< 20–30 turns) where recent context is all that matters
Summarisation / compaction	Fold older turns into a running, anchored summary	Keeps mission and decisions alive across hours; strong drift resistance when anchored	Loses low-frequency detail; adds a summariser call and latency; can drift if unanchored	Long-horizon single tasks: refactors, research runs, multi-step workflows
External memory	Page large content to a searchable store; retrieve on demand	Effectively unbounded; durable across sessions; nothing is deleted	Retrieval quality is the ceiling; adds infra; wrong recall reintroduces noise	Big or cross-session material: documents, transcripts, prior runs, shared knowledge

The decision rule is about access pattern, not size alone. If the state is needed on essentially every step and fits in a tight summary — the goal, the current plan, the open questions — compact it and keep it resident. If it is large, needed only occasionally and cheaply retrievable — the full 40-interview research corpus, last week's session, the API reference — offload it and page it in when a step actually calls for it. A UK fintech running compliance review and an Indian logistics team running a multi-day data-migration agent land on the same architecture: anchored compaction for the working spine, external memory for the archive, a sliding window only for the freshest turns. For the storage-and-forgetting side of that archive, our companion guide on agent memory management patterns goes deeper, and short-term versus long-term memory in production covers the retrieval mechanics.

Measuring whether compaction actually helps

Compaction changes what the model sees, so you cannot eyeball whether it helped — you have to measure task outcomes. The trap is optimising the wrong number: a strategy that halves token use but drops the goal every third compaction is a regression, even though the graph of "tokens in window" looks great. Measure the thing you actually care about.

• Task success rate on a fixed suite of long-horizon jobs, run with and without each strategy. This is the headline metric; everything else is diagnostic.
• State retention: after N compactions, probe the agent — does it still state the goal, the constraints and the key facts correctly? A cheap way is to inject a canary fact early and check it survives.
• Tokens per completed task, not tokens per call. Cheaper-per-call but more calls is not a win.
• Cost and latency per completed task, including the summariser calls compaction itself adds.

Because offline evals miss the messy interactions of a live loop, treat this as an online experiment: route real traffic across strategies and compare outcomes. Instrument every compaction and eviction event as a span so you can trace what got dropped when a run goes wrong — our guides on instrumenting agents with OpenTelemetry and the broader shift to context engineering over prompt engineering cover the tooling. If you are still assembling the loop itself, the production AI agent build guide is the place to start.

Pitfalls to avoid

1. Summarising rotted context. If you only compact at 100% of the window, you are asking the model to summarise input it is already reading badly. Compact in the 70–80% band while the context is still healthy.
2. Unanchored summaries. Without pinned anchors, the goal degrades a little on every pass until the agent is solving a generic version of your task. Always reproduce the brief verbatim.
3. Deleting instead of evicting. Truncation is lossy and irreversible. Spill to external memory so a later step can page material back in if it turns out to matter.
4. Compacting tool schemas and system state. Active file attachments, tool definitions and the plan should be preserved structurally, never fed to the lossy summariser. Production tools reserve these deliberately.
5. Trusting a bigger window to save you. A 1M-token window does not repeal context rot; it just raises the ceiling. The discipline is the same at 200K and at 2M — curate the smallest set of high-signal tokens that maximises the odds of the outcome you want.

Where this leaves you

Long-running agents live or die on context hygiene. Start by naming the real enemy: drift, the slow rot of signal that arrives long before you run out of room. Then layer the defences — anchor the brief so it never degrades, budget the window like a cache and evict the low-value tokens to external storage, and fall back to anchored iterative summarisation when the resident set is still too heavy. Keep the archive in external memory and page it in on demand. Above all, measure task success on long-horizon jobs, because a tidy token graph tells you nothing about whether the agent still knows what it is doing. These patterns are provider-agnostic and model-agnostic by design: they will outlast whichever window size is fashionable this quarter.