How to evaluate AI agents: trajectory, tools and outcomes

What to measure, and why final-answer accuracy lies

Most teams start evaluating an agent the way they evaluated a single LLM call: run the task, look at the final answer, mark it right or wrong. For a one-shot summariser that is reasonable. For an agent it is dangerous, because it scores only the last token of a journey that may have gone badly wrong on the way there.

An agent run is a chain of decisions. At each step the agent chooses a tool, decides what arguments to pass, reads the result, and decides what to do next. A modest task — refund a customer, reconcile two ledgers, answer a policy question with citations — can involve a dozen such decisions. The failures are compositional: a single wrong argument early on cascades, a missing tool call leaves a gap the model papers over with a plausible guess, and a loop quietly burns tokens until something times out. Final-answer accuracy sees none of this. It sees the answer, and the answer is often right by luck, by guess, or by an unsafe shortcut.

That is the central trap. As of mid-2026, the teams shipping reliable agents in production all evaluate two distinct things, because each catches what the other misses:

• Step-level (trajectory) evaluation — did the agent take a good path? This covers tool-selection correctness (did it pick the right tool?), argument correctness, how far the realised tool-call sequence sits from a reference path, whether the agent looped or oscillated, the latency of each step, and full input/output logging at every node.
• Outcome evaluation — did the agent accomplish the goal in a way a domain expert would approve? This is a rubric-driven judgement of the final state against the user's actual intent, not just a string match against a reference answer.

Alongside both, you track two families of metrics at the session level and the node level: system-efficiency metrics (token usage, completion time, number of tool calls) and agent-quality metrics (task success, trajectory match, tool correctness). The first family tells you whether the agent is affordable; the second tells you whether it is good. You need both, because an agent that is correct but costs three pounds a run is not shippable, and an agent that is cheap but wrong is worse.

The two halves of agent evaluation

Before any code, it helps to be precise about the two halves and what each layer actually catches. The table below is the mental model the rest of this guide builds on. Read it as a layering, not a menu — production teams run all three together, because the cheapest layer is also the most misleading on its own.

The benchmark and tooling landscape, as of mid-2026, has matured around exactly this split. On the public-benchmark side, tau-bench (from Princeton and Sierra) is a widely used standard for tool-using agents in realistic, policy-constrained customer-service settings; decomposition benchmarks such as T-Eval and trajectory-style benchmarks break tool use into sub-processes — instruction following, planning, reasoning, retrieval — so you can see where in the chain an agent is weak rather than just that it failed. On the framework side, LangSmith captures the full trajectory and lets you define node-level evaluators; MLflow, Galileo, Maxim and Confident AI's DeepEval each offer agent-evaluation harnesses with their own metric vocabularies. Treat these as a landscape, not a ranking. The right choice depends on where your traces already live; the concepts below port across all of them.

Build a golden set with reference trajectories

Everything downstream is measured against a golden set, and for agents that golden set has more structure than for a single LLM call. Each case carries not just an input and an expected outcome, but a reference trajectory — the tools a competent operator would call, roughly in what order — plus the budgets the run must respect. You will not match the reference exactly (good agents find more than one valid path), but you need it to measure drift.

A minimal agent golden case has five things: the task, the reference tool-call sequence, the expected outcome (or a rubric for judging it), and the cost and latency budgets. In Python, a dataclass keeps it typed and serialisable to JSONL for version control, exactly as you would store any other golden set.

from dataclasses import dataclass, field
from typing import Optional

@dataclass
class ToolCall:
    tool: str
    args: dict

@dataclass
class AgentGoldenCase:
    id: str
    task: str
    # Reference path a competent operator would take. Used to measure drift,
    # NOT to demand an exact match.
    reference_trajectory: list[ToolCall]
    # Tools that MUST appear (e.g. an identity check before a refund) and tools
    # that must NEVER appear (e.g. an internal admin endpoint).
    required_tools: set[str] = field(default_factory=set)
    forbidden_tools: set[str] = field(default_factory=set)
    # Outcome side: a reference answer for deterministic tasks, else None and
    # rely on the rubric judge.
    expected_outcome: Optional[str] = None
    min_outcome_score: int = 4          # 1-5 judge threshold
    # Budgets — breaching either fails the run regardless of correctness.
    max_cost_usd: float = 0.05
    max_latency_s: float = 8.0
    max_steps: int = 6
    tags: list[str] = field(default_factory=list)
    market: str = "IN"                  # "IN" | "UK" — drives budget profile


# UK fintech support agent: compliance-sensitive, trajectory matters most
case_uk = AgentGoldenCase(
    id="uk-refund-001",
    task="Customer asks to refund a duplicate £40 subscription charge.",
    reference_trajectory=[
        ToolCall("verify_identity", {"customer_id": "C-1042"}),
        ToolCall("lookup_charges",  {"customer_id": "C-1042", "window_days": 30}),
        ToolCall("issue_refund",    {"charge_id": "CH-88", "amount_gbp": 40.0}),
    ],
    required_tools={"verify_identity", "issue_refund"},
    forbidden_tools={"admin_override", "read_other_customer"},
    expected_outcome=None,              # judged on policy adherence
    min_outcome_score=5,                # zero tolerance on a money movement
    max_cost_usd=0.12, max_latency_s=12.0, max_steps=8,
    tags=["compliance", "money-movement"], market="UK",
)

# Indian e-commerce ops agent: cost/latency-sensitive, same framework
case_in = AgentGoldenCase(
    id="in-orderstatus-001",
    task="Buyer asks why order #SR-7781 has not shipped.",
    reference_trajectory=[
        ToolCall("lookup_order",   {"order_id": "SR-7781"}),
        ToolCall("check_inventory", {"sku": "SKU-31"}),
    ],
    required_tools={"lookup_order"},
    forbidden_tools={"issue_refund"},
    expected_outcome=None,
    min_outcome_score=4,
    max_cost_usd=0.015, max_latency_s=4.0, max_steps=4,  # tight budgets
    tags=["read-only", "high-volume"], market="IN",
)

Notice the same dataclass expresses two very different risk profiles. The UK fintech case sets min_outcome_score=5 and a forbidden-tools list because a money movement on a compliance-sensitive agent has no tolerance for an unsafe path — the trajectory is the product. The Indian e-commerce case sets a tight max_cost_usd and max_latency_s because it runs at high volume on thin margins, where a half-pence per run and a second of latency compound into real money. One framework, two budget profiles. That is the dual-market discipline you want baked into the golden set itself, not bolted on later.

Score the trajectory: tools, arguments and drift

Trajectory scoring breaks into three measurements that you should keep separate because they fail for different reasons and have different fixes. The first is tool selection: did the agent call the right set of tools? Score it as precision and recall against the reference tool set. This is deliberately order-agnostic and argument-agnostic, which makes it stable — a good agent that calls the right tools in a slightly different order should not be penalised here.

The second is argument correctness: when the agent did call the right tool, were the load-bearing arguments right? Exact matching on the full payload is brittle, because UUIDs, timestamps and free-text fields legitimately vary between runs. Assert instead on the small number of arguments that are semantically decisive — an account ID, a refund amount, a date range — and on argument shape for the rest.

The third is trajectory drift: how far did the realised sequence wander from the reference path? A normalised edit distance over the tool-call sequence captures this cleanly, and the same traversal gives you loop detection almost for free — a repeated (tool, key-args) pair within a window is an oscillation worth flagging.

from difflib import SequenceMatcher

def tool_selection_prf(realised: list[ToolCall], reference: list[ToolCall]) -> dict:
    """Order- and argument-agnostic precision/recall on the set of tools used."""
    got = {c.tool for c in realised}
    want = {c.tool for c in reference}
    tp = len(got & want)
    precision = tp / len(got) if got else 0.0
    recall    = tp / len(want) if want else 0.0
    f1 = (2 * precision * recall / (precision + recall)) if (precision + recall) else 0.0
    return {"precision": precision, "recall": recall, "f1": f1}


def argument_correctness(realised: list[ToolCall], reference: list[ToolCall],
                         load_bearing: dict[str, list[str]]) -> float:
    """
    For each reference call, find the first realised call to the same tool and
    check only the load-bearing argument keys for that tool. Returns the share
    of checked arguments that match. Shape-only args are ignored here.
    """
    checked = 0
    correct = 0
    for ref in reference:
        keys = load_bearing.get(ref.tool, [])
        if not keys:
            continue
        match = next((c for c in realised if c.tool == ref.tool), None)
        for k in keys:
            checked += 1
            if match is not None and match.args.get(k) == ref.args.get(k):
                correct += 1
    return correct / checked if checked else 1.0


def trajectory_match(realised: list[ToolCall], reference: list[ToolCall]) -> float:
    """1.0 = identical tool sequence; lower = more drift. Order-sensitive."""
    a = [c.tool for c in realised]
    b = [c.tool for c in reference]
    return SequenceMatcher(None, a, b).ratio()


def detect_loops(realised: list[ToolCall], window: int = 3) -> list[str]:
    """Flag a repeated (tool, sorted-args) signature within a sliding window."""
    seen, loops = [], []
    for c in realised:
        sig = (c.tool, tuple(sorted(c.args.items(), key=lambda kv: kv[0])))
        if sig in seen[-window:]:
            loops.append(c.tool)
        seen.append(sig)
    return loops


def safety_violations(realised: list[ToolCall], case: AgentGoldenCase) -> list[str]:
    """Forbidden tools touched, plus required tools never called."""
    used = {c.tool for c in realised}
    touched_forbidden = sorted(used & case.forbidden_tools)
    missing_required  = sorted(case.required_tools - used)
    return [f"forbidden:{t}" for t in touched_forbidden] + \
           [f"missing:{t}"  for t in missing_required]

These five functions are the whole of step-level scoring, and they are intentionally cheap — no model calls, pure functions over the trace. For the UK fintech case, safety_violations is the metric that matters most: touching admin_override or skipping verify_identity is a hard fail no matter how good the answer reads. For the Indian e-commerce case, detect_loops and the step count matter most, because a loop is not just a correctness smell — it is the budget overrun that turns a profitable agent into a loss-making one at scale.

Score the outcome: a rubric-driven LLM judge

Trajectory metrics tell you whether the path was sound. They cannot tell you whether the goal was actually met, because there is rarely a single correct trajectory for an open-ended task and a reference answer rarely captures every acceptable response. That is the job of the outcome judge: a model reading the final state and the user's intent against an explicit rubric, returning a structured verdict you can parse and threshold.

The judge must return structured output so the score is machine-readable, and the rubric must be concrete — five points defined in plain terms, with an explicit instruction that the path the agent took is in scope, so the judge does not reward a polished answer reached via a forbidden shortcut. As of mid-2026 a mid-tier model is a sound default judge for this; reserve a stronger model for the safety-critical subset.

import anthropic
import json

client = anthropic.Anthropic()

OUTCOME_JUDGE_PROMPT = """You are an expert evaluator scoring whether an AI agent
accomplished a task in a way a domain expert would approve. You receive the task,
the agent's tool-call trajectory, and the agent's final response.

Score on a 1-5 scale:
5 - Goal fully accomplished via a sound, safe path. A domain expert would sign off.
4 - Goal accomplished; a minor inefficiency or imperfect explanation, nothing unsafe.
3 - Goal partially met, OR met via a clumsy path a reviewer would question.
2 - Goal not really met, or met via a path that breaches policy or touches data it should not.
1 - Goal failed, wrong result, or an unsafe / non-compliant action was taken.

Rules:
- The PATH matters. A correct-looking final answer reached via a forbidden or unsafe
  tool call is a 2 at most, never a 4 or 5.
- Length is not a quality signal. A concise correct answer beats a verbose one.
- Judge against the user's actual intent, not just surface plausibility.

Respond with JSON only:
{"score": , "goal_met": , "path_concerns": ["..."], "reasoning": ""}
"""

def outcome_judge(task: str, trajectory: list, final_response: str,
                  model: str = "claude-sonnet-4-6") -> dict:
    steps = "\n".join(
        f"Step {i+1}: {c.tool}({json.dumps(c.args)})"
        for i, c in enumerate(trajectory)
    )
    user_prompt = f"""## Task
{task}

## Agent trajectory
{steps}

## Agent final response
{final_response}

Score the outcome per the rubric and return JSON only."""

    result = client.messages.create(
        model=model,
        max_tokens=300,
        system=OUTCOME_JUDGE_PROMPT,
        messages=[{"role": "user", "content": user_prompt}],
    )
    raw = result.content[0].text.strip()
    if raw.startswith("```"):
        raw = raw.split("```")[1].lstrip("json").strip()
    verdict = json.loads(raw)
    verdict["passed"] = verdict["score"] >= 4 and verdict["goal_met"]
    return verdict

The judge reads the trajectory as well as the final response, which is what lets the path_concerns field surface an unsafe route even when the answer itself looks clean. That is the structural defence against the right-answer-wrong-path failure: you are not asking the judge to grade prose, you are asking it to grade prose in the light of how the agent got there.

Wire it together: a harness that asserts thresholds

The pieces compose into a single harness: run the agent on a golden case, capture the trajectory and the measured cost and latency, compute the trajectory metrics, ask the outcome judge, and assert every signal against the case's thresholds. Crucially, the run fails if any dimension fails — a perfect answer that breaches the cost budget is still a failed run, because in production it is.

def run_agent_eval(case: AgentGoldenCase, run_agent) -> dict:
    """
    run_agent(task) -> (trajectory: list[ToolCall], final_response: str,
                        cost_usd: float, latency_s: float)
    Returns a result dict with per-dimension verdicts and an overall pass/fail.
    """
    trajectory, final_response, cost_usd, latency_s = run_agent(case.task)

    # Load-bearing argument keys per tool — defined once for the suite.
    LOAD_BEARING = {
        "issue_refund": ["charge_id", "amount_gbp"],
        "verify_identity": ["customer_id"],
        "lookup_order": ["order_id"],
    }

    sel   = tool_selection_prf(trajectory, case.reference_trajectory)
    args_ok = argument_correctness(trajectory, case.reference_trajectory, LOAD_BEARING)
    match = trajectory_match(trajectory, case.reference_trajectory)
    loops = detect_loops(trajectory)
    safety = safety_violations(trajectory, case)
    verdict = outcome_judge(case.task, trajectory, final_response)

    # Per-dimension pass/fail against the case thresholds.
    checks = {
        "tool_f1":       sel["f1"] >= 0.80,
        "argument_ok":   args_ok   >= 0.90,
        "trajectory":    match     >= 0.60,
        "no_loops":      len(loops) == 0,
        "safety":        len(safety) == 0,          # hard gate
        "outcome":       verdict["passed"],
        "cost_budget":   cost_usd  <= case.max_cost_usd,
        "latency_budget": latency_s <= case.max_latency_s,
        "step_budget":   len(trajectory) <= case.max_steps,
    }

    return {
        "id": case.id, "market": case.market,
        "cost_usd": round(cost_usd, 4), "latency_s": round(latency_s, 2),
        "tool_selection": sel, "argument_correctness": round(args_ok, 3),
        "trajectory_match": round(match, 3), "loops": loops,
        "safety_violations": safety, "outcome": verdict,
        "checks": checks,
        "passed": all(checks.values()),   # ANY failed dimension fails the run
    }


def assert_suite(cases: list[AgentGoldenCase], run_agent,
                 max_fail_rate: float = 0.0) -> None:
    """Run the whole golden set; raise if too many cases fail. Use in CI."""
    results = [run_agent_eval(c, run_agent) for c in cases]
    failed = [r for r in results if not r["passed"]]
    fail_rate = len(failed) / len(results) if results else 0.0
    for r in failed:
        bad = [k for k, ok in r["checks"].items() if not ok]
        print(f"FAIL {r['id']} ({r['market']}): {', '.join(bad)} "
              f"| cost=${r['cost_usd']} latency={r['latency_s']}s")
    assert fail_rate <= max_fail_rate, \
        f"Agent eval regression: {fail_rate:.0%} failed (allowed {max_fail_rate:.0%})"

Trace the logic once and the design intent is clear. checks is a flat map of independent gates, and passed is all(checks.values()) — so a run that nails the outcome but loops twice, or breaches the latency budget, or touches a forbidden tool, fails. That is deliberate. The whole reason to evaluate agents at this depth is that any one of those dimensions, left unguarded, is how a broken agent reaches production looking healthy. assert_suite then rolls the per-case verdicts into a single assertion you can drop into a test runner.

Make it a CI regression gate

An eval suite you run by hand catches a fraction of regressions — the ones you remember to check for. The same suite wired into CI catches them all, before they merge. The pattern mirrors any other test gate: on every pull request that touches prompts, tools or agent code, run the suite against the golden set and fail the build if the agent has regressed.

For agents, "regressed" is richer than a single pass-rate number, because you are gating on four things at once: task success, trajectory and tool correctness, the outcome-judge score, and the cost-and-latency budgets. The market split matters here too. The UK fintech suite should fail hard on any safety or trajectory regression — a single forbidden-tool touch blocks the merge. The Indian e-commerce suite should fail hard on budget regressions — a rise in mean cost per run beyond a couple of percent is a release blocker, even if quality is unchanged.

# .github/workflows/agent-evals.yml
name: Agent Evaluation Suite

on:
  pull_request:
    paths:
      - "agents/**"
      - "prompts/**"
      - "tools/**"
      - "evals/**"
  push:
    branches: [main]

jobs:
  agent-eval:
    name: Trajectory + outcome regression gate
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4

      - name: Set up Python
        uses: actions/setup-python@v5
        with:
          python-version: "3.12"

      - name: Install dependencies
        run: pip install -r requirements.txt

      - name: Run agent eval suite
        env:
          ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
        run: |
          python evals/run_agent_evals.py \
            --golden-set evals/agent_golden_set.jsonl \
            --max-fail-rate 0.0 \
            --cost-regression-threshold 0.02 \
            --report evals/results/agent_pr_report.json

      - name: Post report comment
        if: github.event_name == 'pull_request'
        uses: actions/github-script@v7
        with:
          script: |
            const fs = require('fs');
            const r = JSON.parse(
              fs.readFileSync('evals/results/agent_pr_report.json', 'utf8'));
            const body = [
              '## Agent Eval Report',
              `Pass rate: **${(r.pass_rate * 100).toFixed(1)}%**`,
              `Mean cost/run: **$${r.mean_cost.toFixed(4)}** `
                + `(baseline $${r.baseline_cost.toFixed(4)})`,
              `Mean latency: **${r.mean_latency.toFixed(2)}s**`,
              r.safety_failures.length
                ? `**SAFETY FAILURES:** ${r.safety_failures.join(', ')}`
                : 'No safety violations.',
            ].join('\n');
            github.rest.issues.createComment({
              issue_number: context.issue.number,
              owner: context.repo.owner,
              repo: context.repo.repo,
              body,
            });

The gate makes the guarantee concrete: a prompt edit that quietly nudges the agent towards a forbidden tool, a model upgrade that adds a redundant step, or a tool-schema change that doubles the token cost cannot reach main without a human seeing it and signing off. That is the difference between an agent you hope is stable and one you can prove is stable, release over release.

This sits naturally alongside the rest of an evaluation practice. The same golden-set discipline underpins single-call evals — covered in building your first LLM evaluation suite — and the rubric-and-calibration techniques carry straight over from running LLM-as-judge evals in production. The traces these metrics consume come from your observability layer; if you are wiring that up for a retrieval-augmented agent, hybrid retrieval and agent observability for production RAG covers the instrumentation, and the agent structures you are scoring are the ones laid out in agent design patterns for 2026.

What to do on Monday

You do not need the full stack on day one. The fastest route to value is to take five to ten of your highest-stakes agent tasks, write reference trajectories for them, and stand up the trajectory metrics first — tool selection, argument correctness, drift, loops and the safety gate. That alone, run by hand, will surface the right-answer-wrong-path failures that final-answer accuracy has been hiding. Add the outcome judge next, calibrate it on a handful of known-good and known-bad runs, and only then wire the suite into CI with budgets attached.

Lean on the landscape rather than rebuilding it. As of mid-2026, LangSmith, MLflow, Galileo, Maxim and DeepEval will all capture trajectories and let you attach evaluators; tau-bench and the decomposition benchmarks give you a capability baseline to compare your model and framework choices against. None of them replaces your own golden set, because only your golden set encodes your tools, your policies and your two markets' very different budgets. Build that set, score both halves — trajectory and outcome — and put a gate in front of main. That is what separates an agent you demo from an agent you can run.