Background Agents: Durable Execution for Agent Jobs That Run for Hours

When the request-response loop stops working

Most agent tutorials are written against a mental model borrowed from web handlers. A request arrives, the model thinks, a tool or two fires, a response goes out, and the whole thing is finished in four seconds. That is adequate for a chat turn. It falls apart the moment the work is genuinely long: a research sweep across two hundred sources, a codebase migration touching nine hundred files, a document-processing run over a quarter of invoices, a nightly reconciliation between two ledgers that disagree.

The symptoms arrive in a predictable order. First the timeouts. A request that runs past the gateway's idle limit dies with a 504 and no useful error, and those limits are commonly in the thirty-to-sixty-second range — an AWS Application Load Balancer defaults to a sixty-second idle timeout, and Cloudflare's proxy gives up on a slow origin after about a hundred seconds. Then the deploys: a routine rollout terminates the container mid-run, and an hour of accumulated agent state that existed only in one process's memory goes with it. Then the closed tab, because nothing was holding the work open except a socket. And finally the worst one, the retries — somebody wraps the call in a retry decorator to make the flakiness go away, the agent re-executes from the beginning, and a customer in Pune or Leeds receives the same onboarding email three times because not one tool in the chain was idempotent.

Every one of those is the same root cause wearing a different costume: the agent run has no existence independent of the process that started it. The fix is conceptual before it is technical. The run stops being a function call and becomes a job — something with an identity you can quote in a support ticket, a status you can query, a state you can inspect halfway through, a lifecycle with defined terminal states, and an owner that is a queue rather than a socket.

This is a guide to the execution substrate, not to agent design. Agent memory, context compaction, observability instrumentation and framework choice are covered elsewhere on this site and are assumed settled; if the run's context window is the problem rather than the run's lifetime, start instead with the guide to context engineering and compaction for long-running agents. What remains here is the layer underneath: how a run survives a restart, resumes without repeating itself, pauses for a human without occupying a worker, and stops when you tell it to.

Three architectures, and when each is right

There are broadly three shapes for running agent work. The table below is a triage aid rather than a ranking — the third option is not the mature version of the first two, it is a different set of trade-offs with a higher floor and a higher ceiling.

Shape	Typical run length	Survives process restart	Human-in-the-loop	Operational cost	Pick it when
Synchronous with streaming	Seconds to about a minute	No — the run dies with the connection	Only within the open connection	Lowest; no new infrastructure	Interactive chat, single-tool lookups, anything a user watches
Queue plus worker with an external checkpointer	Minutes to hours	Yes, from the last committed step	Yes, by parking the run in a paused status	Moderate; you own the replay logic	Almost every real background agent, and the right default
Durable execution engine	Minutes to weeks	Yes, by deterministic replay of a recorded log	First-class, via signals and long timers	Highest; a new runtime and determinism rules	Multi-day workflows, compensation logic, several teams sharing guarantees

Synchronous with streaming is what you already have. The user opens a connection, tokens and tool-call events stream back over server-sent events, and the run lives exactly as long as the socket does. It is the right answer for interactive work, and the mechanics of doing it well at scale — backpressure, heartbeats, reconnection — are covered in the guide to streaming LLM responses at scale. The failure mode is that there is no durability at all: no restart survives, no deploy survives, and a user who closes the tab has cancelled the work whether they meant to or not.

Queue plus worker with an external checkpointer is the pragmatic middle and the shape most teams should build first. Admission goes through a durable queue — Redis Streams, RabbitMQ, Amazon SQS, or a queue implemented directly in Postgres, which is respectable at moderate volume and one fewer system to operate. A pool of workers claims jobs from it, and after every step the agent's state is written to a transactional database. A worker that dies loses at most the step in flight, because another worker claims the run and picks up from the last committed step index. The cost is that you write the resumption logic yourself, and that logic is where the subtle bugs live.

Durable execution engines — Temporal, Restate, Inngest and their peers — invert the problem. The engine records the result of every step in a durable log, and after a crash it replays your function from the top, feeding recorded results back in place of re-execution until it reaches the point where the original run stopped. Your code then reads like a straight-line function with await calls and sleeps, including sleeps measured in days. The price is real: workflow code must be deterministic — no unguarded clock reads, no unguarded random values, no iterating a dictionary whose ordering might vary — plus a versioning discipline for changing that code while runs are in flight.

Be fair to yourself about the trade. An engine solves problems a queue-and-worker design leaves to you, but it also introduces a runtime your on-call rota must understand at three in the morning. Start in the middle and graduate when hand-written replay logic starts causing incidents. Note too that this choice is orthogonal to your agent framework — the comparison in LangGraph versus CrewAI versus the OpenAI Agents SDK is about how you express the agent's logic, not about what keeps it alive across a restart.

The four properties that make a run durable

Underneath all three architectures sit four properties. A system that has all four is durable regardless of which libraries implement them; a system missing any one of them has a specific, predictable failure waiting for it.

Persisted state after every step

The agent's message list, its tool results, its accumulated findings and its step index live outside process memory, written to durable storage at every step boundary. Not at the end. Not every ten steps. Every step, because the step you skip persisting is the one the deploy interrupts. This prevents the failure where a rollout 40 minutes into a 55-minute run costs you all 40 minutes. What exactly you persist overlaps with the guide to agent memory in production, but the durability rule is narrower: whatever the agent needs to continue must be reconstructable from the row, and none of it may exist only in a local variable.

Idempotent side effects

Every tool call that touches the outside world carries an idempotency key, so that executing it twice has the same observable effect as executing it once. The key must be deterministic — derived from the run identifier and the step index — because a key generated freshly at call time changes on every replay and therefore deduplicates nothing at all. Most serious APIs accept one; Stripe's idempotent requests are the canonical implementation and a reasonable model for your own internal services. Where a downstream provider offers nothing, enforce it yourself with a unique constraint on the key in your own database, checked inside the same transaction that records the effect.

Replayable step results

A step that has already completed returns its recorded result instead of executing again. This is what makes resumption cheap: a worker picking up a half-finished run does not redo forty minutes of retrieval, it reads forty recorded results and continues from step forty-one. The requirement this imposes is determinism, and the plain-English version is simple — anything that would give a different answer the second time must be recorded rather than re-rolled. Model calls are the obvious case, since sampling is stochastic and a replayed call yields different text, diverging the run from its own history. Clock reads are the same problem in slower motion, as are random values, generated identifiers, and anything derived from the current state of an external system. This is also why tool schemas should be stable and versioned; a tool whose arguments change shape between deploys will not replay cleanly, which is one more argument for the discipline in designing tools for AI agents.

Explicit terminal states

Every run ends, and it ends in a state you named on purpose: succeeded, failed, cancelled or timed out. "Probably still going" is not a state, and a run that has been in running for nine hours with no lease renewal is not running, it is lost. Terminal states make the fleet legible — you can count them, alert on their distribution, bill against them and explain them to a customer. They also make cleanup possible, because a reaper cannot safely delete rows in a status that has no defined end.

A queue-and-worker implementation

Here is the middle architecture in concrete form. The design below uses PostgreSQL for both the queue and the state store, which keeps the moving parts to a minimum and is adequate up to thousands of runs a day; swapping in Redis Streams or SQS for admission later changes the claim query and nothing else. Two tables: one row per run holding identity, status and state, and one row per completed step forming the replay log.

-- The run row IS the job's identity. Everything else hangs off it.
CREATE TYPE run_status AS ENUM (
  'queued', 'runnable', 'running', 'awaiting_approval',
  'succeeded', 'failed', 'cancelled', 'timed_out', 'dead_letter'
);

CREATE TABLE agent_runs (
  id                BIGSERIAL PRIMARY KEY,
  idempotency_key   TEXT        NOT NULL UNIQUE,   -- caller-supplied; dedupes submissions
  status            run_status  NOT NULL DEFAULT 'queued',
  input             JSONB       NOT NULL,
  state             JSONB       NOT NULL DEFAULT '{}'::jsonb,
  state_version     INT         NOT NULL DEFAULT 1, -- bump when the blob's shape changes
  current_step      INT         NOT NULL DEFAULT 0,
  attempt           INT         NOT NULL DEFAULT 0, -- claims, not retries within a claim
  max_attempts      INT         NOT NULL DEFAULT 5,
  cancel_requested  BOOLEAN     NOT NULL DEFAULT FALSE,
  deadline_at       TIMESTAMPTZ,                    -- wall-clock budget
  step_budget       INT         NOT NULL DEFAULT 200,
  token_budget      INT,
  tokens_used       INT         NOT NULL DEFAULT 0,
  lease_expires_at  TIMESTAMPTZ,                    -- a dead worker's lease lapses
  last_error        TEXT,
  created_at        TIMESTAMPTZ NOT NULL DEFAULT now(),
  updated_at        TIMESTAMPTZ NOT NULL DEFAULT now()
);

-- Partial index: the claim query only ever looks at claimable rows.
CREATE INDEX agent_runs_claimable_idx
    ON agent_runs (created_at)
 WHERE status IN ('queued', 'runnable', 'running');

-- One row per completed step. This is the replay log.
CREATE TABLE agent_steps (
  run_id      BIGINT      NOT NULL REFERENCES agent_runs(id) ON DELETE CASCADE,
  step_index  INT         NOT NULL,
  tool_name   TEXT        NOT NULL,
  args_hash   TEXT        NOT NULL,
  result      JSONB       NOT NULL,
  recorded_at TIMESTAMPTZ NOT NULL DEFAULT now(),
  PRIMARY KEY (run_id, step_index)
);

The primary key on (run_id, step_index) does quiet but essential work: it makes it structurally impossible to record two results for the same step, which is the database enforcing replay correctness rather than your code remembering to.

Now the worker. It claims a run with SELECT ... FOR UPDATE SKIP LOCKED, the standard PostgreSQL idiom for a work queue — the lock skips rows other workers already hold instead of blocking behind them, so a pool of twenty workers picks up twenty different runs without contention. The PostgreSQL SELECT documentation covers the locking clauses; the ordering that matters is that the locking clause comes after LIMIT.

# worker.py — claim one run, drive it forward, commit after every step.
import psycopg
from psycopg.rows import dict_row
from psycopg.types.json import Json

# Connect with autocommit ON — conn = psycopg.connect(DSN, autocommit=True).
# The claim, each step write and each state write are separate transactions.
# Without autocommit, psycopg keeps a single transaction — and the claim's
# row lock — open across every model call in the run.


class LeaseLost(Exception):
    """Our lease lapsed and another worker re-claimed this run."""


CLAIM = """
UPDATE agent_runs
   SET status           = 'running',
       attempt          = attempt + 1,
       lease_expires_at = now() + interval '90 seconds',
       updated_at       = now()
 WHERE id = (
       SELECT id
         FROM agent_runs
        WHERE status IN ('queued', 'runnable')
           OR (status = 'running' AND lease_expires_at < now())
        ORDER BY created_at
        LIMIT 1
        FOR UPDATE SKIP LOCKED
       )
 RETURNING id, state, current_step, attempt, max_attempts,
           deadline_at, step_budget, token_budget, tokens_used;
"""


def work_once(conn) -> bool:
    """Claim and drive one run. Returns False when the queue is empty."""
    with conn.cursor(row_factory=dict_row) as cur:
        cur.execute(CLAIM)
        run = cur.fetchone()
    if run is None:
        return False

    run_id, state, index = run["id"], run["state"], run["current_step"]

    if run["attempt"] > run["max_attempts"]:
        finish(conn, run_id, "dead_letter", "claim budget exhausted")
        return True

    while True:
        stop = check_stop_conditions(conn, run_id, run, index)
        if stop:                                  # cancelled / timed_out / budget
            finish(conn, run_id, stop)
            return True

        plan = decide_next(state)                 # your agent's policy
        if plan.done:
            finish(conn, run_id, "succeeded")
            return True
        if plan.needs_approval:
            park_for_approval(conn, run_id, state, plan.request)
            return True

        # The model call and the tool call happen OUTSIDE any transaction.
        # Holding a row lock across a 40-second model call is how a
        # connection pool dies at 09:00 on a Monday.
        result, record = step(conn, run_id, index, plan.tool_name, plan.args)
        state = apply_result(state, plan, result)
        index += 1

        # One transaction: the step result and the new run state land
        # together, or neither of them does. `attempt` is a fencing token —
        # if our lease lapsed and somebody else re-claimed the run, attempt
        # has moved on and this write is refused rather than clobbering them.
        try:
            with conn.transaction():
                if record is not None:
                    conn.execute(
                        "INSERT INTO agent_steps"
                        " (run_id, step_index, tool_name, args_hash, result)"
                        " VALUES (%s, %s, %s, %s, %s)"
                        " ON CONFLICT (run_id, step_index) DO NOTHING",
                        record,
                    )
                fenced = conn.execute(
                    "UPDATE agent_runs"
                    "   SET state = %s, current_step = %s,"
                    "       tokens_used = tokens_used + %s,"
                    "       lease_expires_at = now() + interval '90 seconds',"
                    "       updated_at = now()"
                    " WHERE id = %s AND attempt = %s",
                    (Json(state), index, plan.tokens, run_id, run["attempt"]),
                )
                if fenced.rowcount == 0:
                    raise LeaseLost(run_id)
        except LeaseLost:
            return True      # re-claimed elsewhere; stand down, do not write

Three details in that loop repay attention. The lease is renewed on every step write, so a live worker keeps its claim while a killed one lets the lease lapse and the claim query re-selects the run for somebody else. A lease is a guess rather than a guarantee, though — a worker stalled by a slow model call or a long garbage-collection pause can have a live lease expire underneath it, and then two workers are driving the same run — so the state write carries the claim's attempt value as a fencing token and is refused outright if anybody else has re-claimed the run in the meantime. Without that predicate the lapsed-lease path is a genuine lost-update race, not a theoretical one. And the attempt counter increments on claim, not on retry-within-a-claim, making it a direct measure of how many times a run has crashed a worker — the number you need for poison pills later.

Finally, the replay helper. This is about fifteen lines and it is where "resume from where we left off" actually happens.

import hashlib, json
from psycopg.types.json import Json


class NonDeterministicReplay(Exception):
    """The code path changed underneath a run that was already in flight."""


def step(conn, run_id: int, index: int, tool_name: str, args: dict):
    """Replay-safe step. A recorded result is returned, never re-executed.

    Returns (result, record_or_None). The caller persists the record in
    the same transaction as the new run state.
    """
    args_hash = hashlib.sha256(
        json.dumps(args, sort_keys=True, default=str).encode()
    ).hexdigest()

    row = conn.execute(
        "SELECT tool_name, args_hash, result FROM agent_steps"
        "  WHERE run_id = %s AND step_index = %s",
        (run_id, index),
    ).fetchone()

    if row is not None:
        if (row[0], row[1]) != (tool_name, args_hash):
            raise NonDeterministicReplay(
                f"run {run_id} step {index}: recorded {row[0]}/{row[1][:8]}, "
                f"replay wants {tool_name}/{args_hash[:8]}"
            )
        return row[2], None                       # replayed — no side effect

    # First execution. The key is derived from run and step, so a crash
    # between the tool call and the INSERT produces the SAME key on retry.
    result = TOOLS[tool_name](**args, idempotency_key=f"{run_id}:{index}")
    return result, (run_id, index, tool_name, args_hash, Json(result))

Read the last three lines again, because they are the whole idempotency argument in miniature. There is an unavoidable window between the tool executing and its result being committed, and a crash inside that window means the tool will be called again on resumption. Because the key is f"{run_id}:{index}" rather than a fresh UUID, the second call carries the identical key and the downstream service recognises it as a duplicate. Without that, a crash in a one-millisecond window sends the email twice — and one-millisecond windows are hit constantly at fleet scale.

The args_hash comparison earns its place too. If somebody changes the planning logic and redeploys while runs are mid-flight, a replayed run may want a different tool at step twelve than the one recorded there. Failing loudly with NonDeterministicReplay beats silently returning a result from a different tool.

Pausing for a human without holding a process open

This is the pattern that separates a real agent job runner from a fancy background task. Something mid-run needs a person: approve this refund, confirm this migration touches the right nine hundred files, sign off before the agent emails a customer. The naive implementation blocks — the worker sits in a polling loop or an await waiting for the approval. That worker is now unavailable for hours, dies with the next deploy, and twenty parked runs mean twenty occupied workers and a queue that stops moving.

The durable pattern releases the worker entirely. The run transitions to awaiting_approval, persists what it is asking for and everything needed to continue, and the worker returns to the pool. A webhook, a UI action or an expiry timer later transitions the run back to runnable with the human's decision written into state, and the next free worker picks it up as an ordinary claim.

From	Trigger	To	What gets persisted
running	Agent plan requires approval	awaiting_approval	The request, the proposed action, an expiry timestamp; lease cleared
awaiting_approval	Human approves via UI or webhook	runnable	Decision, approver identity, decision timestamp, written into state
awaiting_approval	Human rejects	runnable or cancelled	Rejection and reason; the agent decides whether an alternative exists
awaiting_approval	Expiry sweeper finds an overdue request	timed_out	Terminal reason; notification to the requester
runnable	A worker claims it	running	New lease, incremented attempt counter

The timeout row is the one people forget, and it is not optional. An approval nobody ever answers must expire into a terminal state rather than sitting in awaiting_approval until someone finds it during an unrelated investigation eight months later. Run a sweeper that moves overdue approvals to timed_out, notify whoever asked, and pick the window deliberately: a refund approval might expire in two working hours, a migration sign-off in three days. Store the expiry on the run so it survives a restart of the sweeper.

The interaction design — what you show the approver, how much context they need, how the decision travels back — is a separate and well-covered problem. For an in-framework treatment see the guide to LangGraph agent state, tool calling and human-in-the-loop; for the protocol-level equivalent, where a tool server itself asks the user a question mid-call, see MCP sampling and elicitation. Both assume the substrate described here exists underneath them.

Cancellation, timeouts and budgets

A run that can be started must be stoppable, and "stoppable" has to mean something better than killing the pod. The right mechanism is cooperative cancellation. A cancel request sets cancel_requested = TRUE on the run row — a cheap, durable write any API handler can make. The worker checks that flag at every step boundary and, when it sees it, finishes cleanly: it does not start the next step, it writes a terminal cancelled status, and it stops. Crucially it never interrupts a tool call mid-flight, because a tool killed halfway through has already had its side effect but has no recorded result — exactly the inconsistent state everything else here is designed to avoid.

Budgets work the same way and are checked at the same boundary. A wall-clock deadline, stored as deadline_at, catches the agent that has been going in circles since half past nine. A step budget catches loops that would otherwise run until that deadline. A token budget catches the run whose steps are individually reasonable and collectively ruinous. All three live on the run row rather than in worker configuration, so they survive a restart, they can differ per tenant or plan tier, and the reason a run stopped is recorded on the run itself rather than inferred from logs.

One refinement for teams running untrusted or generated code inside agent steps: cooperative cancellation only works when the step itself is well behaved. A step that has spawned a subprocess needs a hard kill path too, which argues for executing such steps inside an isolated sandbox with its own resource limits — the approach in the guide to sandboxing AI agents with microVMs and least privilege.

Observing a fleet you cannot watch

With synchronous agents you watch a request. With a background fleet there is nothing to watch, so you instrument or you fly blind. The signals below are the ones that actually tell you something; the general instrumentation mechanics — spans, trace context, exporters — are covered in the guide to agent observability with OpenTelemetry and are not repeated here.

Signal	What it tells you	Alert when
Queue depth	How much work is waiting for a worker	Rising steadily across several intervals rather than spiking and draining
Age of the oldest pending run	Whether anything is being starved	It exceeds your promised turnaround — the truest "we are behind" signal
Run duration percentiles	Whether runs are getting slower or longer	The 95th percentile moves without a matching change in input volume
Step retry and claim counts	Which runs and which tools are unstable	Any run reaching two claims; any tool's retry rate stepping up
Lease expiries and worker restarts	Whether workers are dying rather than finishing	Expiries appear at all outside a deploy window
Terminal-state distribution	What actually happens to your runs	The share of failed, timed_out or dead_letter shifts

Queue depth and oldest-pending age are the pair to put on the wall. Depth alone is misleading: a deep queue that drains fast is healthy, and a shallow queue with one run stuck at the front for six hours is not. Together they answer the only question that matters operationally — is the fleet keeping up, and is anything being left behind?

One tracing detail is specific to this architecture. A durable run may be executed by three different workers across two deploys, and most instrumentation will produce three unrelated traces. Store the trace identifier on the run row at admission and have each worker resume that trace context when it claims the run, so a replayed run remains one story rather than three fragments. When the numbers do go bad, the diagnostic path is the one described in the guide to LLM incident response and runbooks, with one advantage: a run table lets you query the blast radius directly rather than estimating it from logs.

Failure modes worth designing against

Five failure modes account for most of the pain in background agent systems, and every one of them has a structural fix that costs less than the incident.

The poison pill is a single run whose state or input reliably crashes any worker that claims it. Without a counter it cycles forever: claimed, crash, lease lapses, reclaimed, crash. Left alone it will take down a whole pool through repeated restarts. The fix is the attempt counter incremented on claim, a dead_letter terminal state once max_attempts is exceeded, and an alert when anything lands there. Dead-lettered runs are a queue a human works through, not a bin.

The thundering retry happens after an upstream model provider recovers from an outage. Several hundred runs failed during it, every one retries the instant the provider is reachable, and the synchronised burst either knocks the provider over again or blows through your rate limit and produces a second outage of your own making. The fix is exponential backoff with jitter — the jitter is the part that matters, because unjittered backoff simply synchronises retries into neat waves, as the Amazon Builders' Library article on timeouts, retries and backoff with jitter explains. Pair it with a circuit breaker at your model gateway so runs fail fast and park rather than queueing behind a dead provider; the gateway side is covered in the guide to a resilient LLM gateway with failover and rate limits.

Unbounded state growth is the slow one. Nobody reaps completed runs, the state blob includes the full text of every document the agent fetched, and eighteen months later agent_runs is hundreds of gigabytes, the claim query has quietly become slow, and a migration on that table is a project. The fix should be written on day one: keep large artefacts in object storage and store references in the state blob, set a retention policy per terminal state, and run a reaper on a schedule. Partitioning by month makes the deletion cheap.

The last is the non-idempotent tool that a replay double-fires, and it appears here again because it deserves to. The defect is invisible until a crash lands in a specific millisecond window, so it survives code review, staging and weeks of production traffic before showing up mid-incident, when everything else is already going wrong. Make idempotency-key support a requirement of your tool interface rather than a convention: if the registry will not accept a side-effecting tool without a key parameter, the failure mode cannot be introduced by accident.

Where to start

The path is incremental, and each step is useful on its own. First, give runs an identity: one table, one row per run, a status column and a caller-supplied idempotency key. Even with execution unchanged, you can now answer "what happened to that job?", which is more than most teams can do. Second, move admission to a queue and let a worker pool pull from it, so a slow run stops occupying a web process. Third, persist state after every step and add the step log — the point at which restarts stop costing you work. Fourth, add idempotency keys to every side-effecting tool, before you scale the fleet rather than after. Fifth, add cancellation, wall-clock deadlines, step budgets and token budgets, because a fleet you cannot stop is a fleet you cannot safely grow. Only then evaluate a durable execution engine, with a real workload and a clear list of the problems you want it to take off your hands.

There is a career argument here as well as an engineering one. Plenty of people can wire an agent to a model API; considerably fewer can point at a job runner that survives a deploy, resumes from step forty-one without re-sending the email, pauses for a human approver without pinning a worker, and stops when told to. That is platform work, it is in short supply in Bengaluru and Chennai as much as in London and Manchester, and it reads far better than another framework demo. If you have built one — even a small open-source one — put it on your Builder profile with the design decisions written out, because the design decisions are the part that shows you understand the problem.