Managed Agent Runtime or Your Own Harness? A Decision Ladder

What the loop actually does, and what you are outsourcing

Most agent architecture arguments are conducted at the wrong altitude. Teams debate frameworks and model providers, and skip the one structural decision that determines almost everything downstream: who owns the loop? There are three answers. Your own application code owns it, a provider owns it and runs it on their servers, or you own the code and run it on infrastructure you control. Every product in this market is a variation on one of those three, and the choice quietly fixes your residency posture, your recovery story, your observability ceiling and your exit cost for the next two years.

So it is worth being precise about what the loop is. Strip away the branding and an agent loop performs six jobs, over and over, until the goal is met or a budget is exhausted.

• Plan. Render current state into a prompt and ask the model what to do next.
• Call. Dispatch the chosen tool with the chosen arguments, into whatever execution environment that tool lives in.
• Observe. Capture the result — including failures, where most agent behaviour is actually decided — and fold it back into state.
• Compact. When the transcript outgrows the window, summarise, prune or offload without destroying what the next step needs.
• Checkpoint. Persist enough state that the run is resumable from a known-good point.
• Recover. On a crash, timeout, rate limit or bad tool result, decide whether to retry, re-plan, escalate or stop.

Four of those six — plan, compact, checkpoint, recover — are pure infrastructure. They have nothing to do with your product, and they are tedious to get right, which is precisely why a market grew to sell them. When OpenAI opened its Agents API in public beta on 10 September 2026, the framing was unusually honest about the split: the managed Codex harness handles sessions, orchestration, context compaction and recovery, while the developer supplies the tools and picks the execution environment. Agents can run in OpenAI-hosted, self-hosted or partner sandboxes, and compaction, tool search, programmatic tool calling and subagents come built in. Anthropic offers managed agents on comparable lines, a trade we looked at when comparing managed agents against LangGraph and the Agent SDK.

That split is the whole article in one sentence. The provider takes the four infrastructure jobs; you keep the two that are yours, the tools and where they execute. It is a good trade for a great many teams and an unacceptable one for some. A ladder tells you which you are in an afternoon, without a bake-off.

The decision ladder: the first hard constraint wins

A ladder is not a scorecard. Scorecards invite teams to weight twelve criteria, average them, and arrive at the runtime the loudest engineer already preferred. A ladder is ordered, and you stop at the first rung that gives a hard answer. The ordering principle: questions whose answers cannot be changed by spending engineering time come first, and questions that are really about effort come last.

The six questions, in order

1. Can your data leave your trust boundary? If a contract, a regulator or a client's security schedule says the payload stays inside accounts you control, the fully managed layer is eliminated before any feature comparison begins. It is first because no amount of cleverness reverses it. The answer is not a feeling — it is a clause, and someone should be able to quote it.
2. Does the agent execute untrusted code? If the model writes and runs code, or tool inputs derive from user-supplied content, you need hard isolation: a microVM or equivalent, default-deny egress, and a short-lived disposable filesystem. This rung points you at sandbox and serverless runtimes, or a managed runtime whose sandbox story you have actually read. Our guide to sandboxing agents with microVMs and least privilege covers what "isolated" has to mean before you accept the word.
3. Does a single run exceed a few minutes, or need to survive a restart? Short synchronous runs fit almost anything, including a request handler. Long autonomous runs need durable execution as a first-class feature rather than a bolt-on — the mechanics are in our piece on background agents and durable execution.
4. Do you need to swap model providers? Single-provider commitments are cheaper to build and more expensive to leave. If procurement, pricing risk or a client mandate means you must be able to move, a provider-managed loop is a structural liability rather than a configuration setting. Write the exit before you need it — see our LLM vendor exit plan.
5. How many engineers do you have to operate this? Not to build it — to operate it, on a Sunday, in eighteen months. A self-hosted harness is a service with an on-call rota, a state store, an upgrade path and a capacity plan. Below roughly two engineers who can own it, self-hosting accumulates debt rather than gaining control.
6. What does your audit and evidence obligation look like? If you must produce, on request, a per-run record of what the agent saw, decided, called and got back, then trace export and step-level accounting need to be contractual rather than best-effort. This rung frequently reverses a decision the first five made look comfortable.

The same ladder as a table

Note what the ladder does not ask: which framework has the nicer API, which vendor shipped the better demo, or which runtime the team already knows. Those are real considerations and they belong in the tie-break, after the ladder has narrowed you to a layer. Letting them in at the top is how organisations discover a residency problem in a security review three weeks before launch.

Where India and UK constraints bite first

Rung one is rarely theoretical in either of our markets. An Indian financial services team typically works to sectoral supervisory expectations that pin processing and storage in-country — in cloud terms the Mumbai or Hyderabad regions, and in some institutions still on-premise hardware with no public-cloud egress at all. That is not a preference to be negotiated with an architect; it arrives as an audit finding. A UK supplier bidding into public sector or NHS-adjacent work meets the same constraint through a different door: the contract, the impact assessment and the security schedule typically fix processing to UK regions such as London and require named subprocessors — which a managed agent runtime, with its own downstream infrastructure, may or may not satisfy.

Treat India's DPDP Act and UK GDPR as a prompt to get the requirement written down precisely, not as the source of your answer. Obligations in India are being phased in over a period running into 2027, and the operative constraint in front of most teams is a specific contract clause or sectoral rule rather than a general reading of the statute. Both failure modes are common: assuming cross-border processing is forbidden when your own contract permits it, and assuming it is fine when the client's schedule says otherwise. Our guide to data residency and routing for AI apps turns that into an actual routing policy.

The three layers, scored

As of September 2026 the market has settled into roughly three layers, and layers age better than vendor names. Managed hyperscaler runtimes — AWS Bedrock AgentCore, Google Vertex AI Agent Engine, Azure AI Foundry Agent Service — sell governance and integration with the enterprise controls an organisation already runs. Framework-native platforms — LangGraph Platform, OpenAI AgentKit — sell a well-shaped programming model with hosting attached. Sandbox and serverless runtimes — E2B, Modal, Daytona, Fly.io Machines, Cloudflare Agents, Vercel Sandbox — sell execution primitives and leave the loop to you.

Cutting across all three layers are the deployment models, and in regulated settings these matter more than the layer itself. Fully managed gives the fastest start, with your agents and data on vendor infrastructure. Bring-your-own-cloud puts the vendor's control plane inside your own cloud accounts, so orchestration is theirs and the data plane is yours — what many regulated organisations end up requiring, and what turns an otherwise disqualified vendor into a candidate. Fully self-hosted gives the most control and the most operational burden, in that order and without exception.

Match the runtime to the agent pattern rather than to the org chart. MicroVM sandboxes for code-executing agents. Durable-execution platforms for long-running autonomous agents — LangGraph's 1.2 milestone was reported as focused on durable execution and stateful workflows, a reasonable signal of where that layer is investing. Hyperscaler runtimes where governance is strict and the organisation's controls, identity and logging already live in that cloud. Where two patterns coexist in one product, run two runtimes rather than forcing one to do both badly. If you are still choosing the programming model underneath, our comparison of LangGraph, CrewAI and the OpenAI Agents SDK is the companion piece.

The portability tax and the driver pattern

Every choice here carries a portability tax; the only question is whether you pay it deliberately or discover it mid-migration. Four things do the actual locking, and none is the API surface everyone worries about.

• Session format. The vendor's representation of a conversation, its message types and its notion of a turn. Once your application reasons about that object directly, the vendor is in your domain model.
• Tool-definition schema. How tools are declared, arguments validated, results and errors represented. Superficially similar across vendors, different in exactly the details that break.
• Compaction behaviour. The least portable item and the least discussed. What survives a summarisation is a vendor product decision, and your agent's behaviour depends on it. Two runtimes with identical APIs produce different agents because they forget differently.
• Trace format. If your evaluations, dashboards and audit evidence are built on a proprietary trace shape, migrating the runtime silently means migrating your quality apparatus with it.

The pattern that keeps the loop swappable is unglamorous and it works: define your tools and your state schema in your own code, and treat the runtime as a driver behind a thin interface. The rule that makes it real is that no vendor object crosses the interface — not a session identifier you treat as meaningful, not a vendor message type, not a vendor tool result.

from dataclasses import dataclass, field
from typing import Any, Protocol

@dataclass
class AgentState:
    """Your schema. Never the vendor's session object."""
    run_id: str
    goal: str
    steps: list[dict[str, Any]] = field(default_factory=list)
    scratch: dict[str, Any] = field(default_factory=dict)
    cursor: int = 0

@dataclass
class ToolSpec:
    name: str
    description: str
    json_schema: dict[str, Any]

@dataclass
class StepResult:
    state: AgentState
    finished: bool
    output: str | None = None
    usage: dict[str, int] = field(default_factory=dict)

class AgentRuntime(Protocol):
    """The only runtime surface the application is allowed to see."""
    def start(self, state: AgentState, tools: list[ToolSpec]) -> str: ...
    def step(self, session_id: str, state: AgentState) -> StepResult: ...
    def checkpoint(self, session_id: str, state: AgentState) -> None: ...
    def resume(self, run_id: str) -> tuple[str, AgentState]: ...

Two implementations, sketched. The hosted driver translates in both directions and keeps the vendor's session identifier opaque; the self-hosted driver runs the six verbs itself.

class HostedRuntime:
    """Provider owns the loop: sessions, compaction, recovery."""

    def __init__(self, client, model: str, store):
        self.client, self.model, self.store = client, model, store

    def start(self, state: AgentState, tools: list[ToolSpec]) -> str:
        session = self.client.sessions.create(
            model=self.model,
            tools=[self._to_vendor_tool(t) for t in tools],
            metadata={"run_id": state.run_id},
        )
        self.store.put_session(state.run_id, session.id)
        return session.id

    def step(self, session_id: str, state: AgentState) -> StepResult:
        turn = self.client.sessions.advance(session_id)
        # Translate the vendor turn into OUR state. Nothing vendor-shaped escapes.
        state.steps.append(self._to_our_step(turn))
        state.cursor += 1
        return StepResult(
            state=state,
            finished=turn.status == "completed",
            output=turn.output_text,
            usage={"in": turn.usage.input_tokens, "out": turn.usage.output_tokens},
        )


class SelfHostedRuntime:
    """You own the loop: plan, call, observe, compact, checkpoint, recover."""

    def __init__(self, model_client, sandbox, store, compactor):
        self.model, self.sandbox = model_client, sandbox
        self.store, self.compactor = store, compactor

    def step(self, session_id: str, state: AgentState) -> StepResult:
        plan = self.model.complete(self._render(state))
        if not plan.tool_call:
            return StepResult(state=state, finished=True, output=plan.text)

        observation = self.sandbox.invoke(plan.tool_call.name, plan.tool_call.args)
        state.steps.append({"call": plan.tool_call.as_dict(), "obs": observation})
        state.cursor += 1
        state = self.compactor.maybe_compact(state)   # your policy, your summary
        self.checkpoint(session_id, state)
        return StepResult(state=state, finished=False, usage=plan.usage)

Durable execution: what "survives a restart" really requires

"Durable" is the most oversold word in this market, because it is usually taken to mean only "we persist state". Persistence is necessary and nowhere near sufficient. A run survives a restart when three properties hold together, and teams routinely ship the first without the other two.

Checkpoint boundaries must be explicit. A checkpoint written mid-tool-call is worse than no checkpoint, because resumption starts from a state that never coherently existed. Put the boundary after an observation is folded into state and before the next plan, write it transactionally, and treat the step cursor as the authoritative position in the run.

Idempotent tool calls are the part with teeth. If a run is resumed and a step replayed, every effectful tool in that step fires again unless you have made it safe. That means deriving a stable key from the run identifier, the step index and the arguments, passing that key downstream as its own idempotency token, and recording the intent to act before acting rather than the result afterwards.

import hashlib, json

def idempotency_key(run_id: str, step: int, name: str, args: dict) -> str:
    payload = json.dumps({"r": run_id, "s": step, "n": name, "a": args},
                         sort_keys=True, separators=(",", ":"))
    return hashlib.sha256(payload.encode()).hexdigest()


class IdempotentTool:
    """Wrap any effectful tool so replay cannot double-fire it."""

    def __init__(self, fn, store, name: str):
        self.fn, self.store, self.name = fn, store, name

    def __call__(self, state, **args):
        key = idempotency_key(state.run_id, state.cursor, self.name, args)

        record = self.store.get(key)
        if record and record["status"] == "committed":
            return record["result"]          # replay: return, do not re-execute
        if record and record["status"] == "reserved":
            # Crashed mid-effect. Ask the downstream system, do not guess.
            return self.fn.reconcile(request_id=key)

        self.store.reserve(key)              # write-ahead: intent before effect
        result = self.fn(**args, request_id=key)
        self.store.commit(key, result)
        return result

Replay safety is the third property and the easiest to break by accident. Anything non-deterministic inside a replayed region — a clock read, a random identifier, a live configuration lookup, a model call without a recorded response — must be captured at first execution and read back on replay. Otherwise the replayed run diverges, and a divergent replay is not recovery, it is a fresh run wearing the same identifier.

This is the rung where the buy case is strongest. All three properties are achievable in your own harness, and none is interesting to build. If rung three is your binding constraint and rungs one and two are not, a durable-execution platform is usually the right purchase.

Observability when the loop is not yours

The moment the loop runs on someone else's machine, you lose the ability to instrument it from the inside. What you get back is whatever the vendor chooses to surface, and the gap between a good dashboard and usable evidence is wide. Three things are non-negotiable, and all are easier to obtain during procurement than after.

• Trace export in an open format. Not a dashboard, not a CSV of summaries — spans you can ship into your own backend and keep after the contract ends. OpenTelemetry-shaped export is the realistic ask; our guide to instrumenting agents with OpenTelemetry covers the span conventions worth standardising on.
• Token accounting per step. Aggregate monthly spend tells you nothing actionable. You need input, output and cached-read counts attributed to a step, so a cost regression traces back to the prompt change or retry loop that caused it.
• Tool-call-level spans. Every dispatch, argument set, result and retry, with the parent run as the trace root. Without this an agent failure is a narrative rather than a diagnosis.

Standardise the span attributes in your own code even when the runtime emits its own, because your attribute names are what survives a runtime change. A minimal, portable shape costs nothing to adopt on day one.

# Emit these on every tool span, regardless of which runtime is driving.
span.set_attributes({
    "agent.run_id":        state.run_id,
    "agent.step":          state.cursor,
    "agent.runtime":       runtime_name,        # "hosted" | "self_hosted"
    "agent.tool.name":     tool_name,
    "agent.tool.attempt":  attempt,
    "agent.tokens.input":  usage["in"],
    "agent.tokens.output": usage["out"],
    "agent.compacted":     compacted_this_step,
})

On cost, reason about structure rather than chasing prices, which change constantly. The three layers bill on different axes: per-session or per-agent-invocation at the managed end, per-token where the loop is inference-bound, and per-compute-second where a sandbox is held open. The two line items that surprise teams are egress — moving traces, artefacts and tool outputs out of the vendor's network, which BYOC often reduces sharply — and idle time, where a sandbox billed per second waits on a slow model call or a human approval. An agent that waits is an agent that bills.

Migrating from managed to self-hosted without a rewrite

Plenty of teams should start managed and move later. That is a legitimate strategy, not a planning failure, provided the move is designed for at the start rather than improvised. Done properly it is a sequence, and the order matters.

1. Externalise state first. Mirror every checkpoint into your own store from day one, in your own schema, even while the vendor holds the authoritative copy. Highest-value item, and nearly free early.
2. Own the tool definitions. Tools live in your repository as ToolSpec objects, translated at the boundary. Never author a tool in a vendor console.
3. Normalise traces at emission. Map vendor spans into your attribute names as they arrive, so evaluations and dashboards rest on names you control.
4. Characterise compaction. Record what the managed runtime keeps and drops on real runs. That becomes the specification your own compactor must match; without it your self-hosted agent behaves differently for reasons nobody can explain.
5. Run both drivers in shadow. Same inputs, both runtimes, comparing tool-call sequences and outcomes on a fixed evaluation set before any traffic moves.
6. Cut over by cohort. Low-stakes workflows first, with the hosted driver retained as a live fallback until the self-hosted path has a quarter of boring behaviour behind it.

Be honest about the cost. With the driver pattern in place from the start, migration is a well-scoped project of a few engineer-weeks, dominated by re-tuning compaction and rebuilding whatever evaluation infrastructure was quietly coupled to the vendor's trace format. Without it you are not migrating, you are rewriting the agent and re-earning its reliability from scratch — and the reliability, not the code, is what took the time originally.

Five ways this goes wrong

1. Choosing on developer experience and discovering residency later. A prototype on a fully managed runtime is fast and delightful, right up to the security review that rules out the whole layer. Ask rung one in week one, in writing.
2. Assuming "managed" includes isolation. A managed loop and a hardened sandbox are separate products, and a managed runtime that lets you pick the execution environment has handed the isolation decision back to you. If the agent writes and runs code, read the sandbox documentation rather than inferring safety from the word "managed".
3. Persisting state and calling it durable. Checkpoints without idempotent tools mean every crash-and-resume risks a duplicated side effect. The characteristic symptom is a support ticket about two identical emails or two payments, traced to a restart nobody noticed.
4. Letting compaction be a black box. Agent behaviour degrades on long runs in ways that look like model regression and are actually summarisation policy. If you cannot inspect what was dropped, you cannot debug it — or reproduce the run for an auditor.
5. Coupling evaluations to a proprietary trace format. The runtime turns out to be the cheap part to replace; the quality apparatus built on its trace shape is not. Normalise at emission and the problem never arises.
6. Self-hosting with nobody to operate it. A harness built by one enthusiastic engineer who then changes team is a liability with a bus factor of one. Rung five is about the rota, not the build.

How to decide in an afternoon

Book three hours with the two or three people who can actually answer the questions, and work the ladder in order. Stop at the first hard answer.

1. Can the data leave your trust boundary? No closes the fully managed layer: BYOC or self-hosted.
2. Does the agent execute untrusted code? Yes means microVM-class isolation with default-deny egress, whoever runs the loop.
3. Do runs outlive a few minutes or need restart survival? Yes means durable execution, bought or built — with idempotent tools either way.
4. Must you swap model providers? Yes means a driver interface, and no provider-owned loop in your domain model.
5. Two engineers to operate this in eighteen months? No means managed or a framework platform, whatever the diagram prefers.
6. Do you owe per-run evidence? Yes puts trace export and step-level accounting in the contract, not the backlog.

Write the answers down with the clause or constraint that produced each one, and revisit the page every six months rather than every time a vendor ships a launch post. The products in this market will churn — new layers, renamed services, features that migrate from one tier to another. The ladder will not, because it is made of constraints rather than features. That is the point of building the decision out of questions vendors cannot change.