Evaluating Multi-Turn Conversational Agents in Production

What you need to know

Most evaluation suites for conversational AI test the wrong thing. They send one prompt, read one reply, score it, and move on — and then the product ships and falls over the moment a real user asks a follow-up. That is because almost nothing interesting about a conversational agent happens in a single turn. The interesting behaviour is between the turns: whether the assistant asks a clarifying question instead of guessing, whether it remembers the constraint you gave it four messages ago, whether it stays on the task you set or quietly drifts onto a different one, and whether it can recover gracefully once it has made a mistake. A single request-and-response test cannot see any of that, because none of it exists yet at turn one.

The benchmarks make the size of the gap embarrassingly clear. On Scale AI's MultiChallenge — a 2026-relevant test of realistic multi-turn behaviour — frontier models that score near-perfectly on older multi-turn benchmarks drop below 50% accuracy in its 2025 evaluation once instruction retention and inference memory are properly probed. If the best models in the world halve their score when you actually test the multi-turn capabilities, an evaluation suite that only checks single replies is, in effect, grading the easy fifth of the work and waving the hard four-fifths through to production untested.

This guide is the playbook for closing that gap. It catalogues the failure modes that only appear across turns and how to detect each one, shows how to build an evaluation dataset from real production threads in a closed loop, walks through simulated-user evaluation, and lays out the frameworks — DeepEval, RAGAS and MLflow as of June 2026 — that score conversations rather than turns. There are two complete code snippets: a simulated-user loop that drives your agent and collects a transcript, and a conversation-level metric that judges the whole thing and returns a pass or fail with a reason. Whether you run a customer-support copilot for a Bengaluru fintech or a voice assistant for a London council, the mechanics are the same.

Why single-turn evals miss most real behaviour

Picture the simplest possible evaluation: a fixed list of questions, the agent's answers, and a score for each answer in isolation. This is how most teams start, and for a stateless tool — a one-shot summariser, a single classification call — it is perfectly adequate. The trouble begins the moment the product has a memory and a conversation, because the things that make conversational agents useful are exactly the things a one-shot test cannot represent.

Consider the four behaviours that define a good conversational agent and notice that none of them fit in a single turn. The first is clarifying questions: when a request is ambiguous, the right move is often to ask rather than answer, and you can only score that if your test allows a second turn in which the user supplies the missing detail. The second is follow-ups: a user who asks "and what about the refund?" relies on the agent to know what "the refund" refers to from earlier in the thread. The third is context carried across turns: a constraint stated once — "I only want vegetarian options", "keep it under two hundred pounds", "I am in Chennai so use IST" — must hold for the rest of the conversation, not just the next reply. The fourth is recovery from earlier mistakes: when the agent gets something wrong and the user corrects it, the measure of quality is whether it absorbs the correction or stubbornly repeats the error.

Every one of those is a property of a sequence, not a message. You cannot test instruction retention with one instruction and one reply, because retention is by definition about whether the instruction survives later turns. This is the same lesson that the agent-evaluation literature reached for tool-using agents — that you have to judge the whole trajectory, not the final answer — applied to dialogue. Our companion guide on evaluating AI agents by trajectory, tool calls and outcome makes that case for tool use; multi-turn conversation is the dialogue equivalent, where the "trajectory" is the sequence of turns and the carried context between them.

A failure-mode catalogue

If you are going to test multi-turn behaviour, it helps to know precisely what you are testing for. The failures below are the ones that recur across production conversational systems, and the reason to name them is that each one demands a different detection strategy — a single overall "quality" score will quietly average them away. Treat this table as the menu of things your eval suite should be able to catch, then map each row to a concrete check.

These map closely to the failure categories that recent benchmarks isolate — MultiChallenge tests instruction retention and inference memory; MT-Eval categorises multi-turn interaction into recollection, expansion, refinement and follow-up. Use those as references for what to test, not as your own product metric.

The discipline this table enforces is to test each row deliberately. Drift needs a turn-by-turn topic check against the goal; attrition needs you to extract the user's stated facts and assert they are never contradicted; loops need repetition detection over consecutive turns; latency abuse needs nothing cleverer than logging tokens and response time per turn. Lump them into one judge that "rates the conversation out of ten" and you get a number that hides exactly which mode is hurting you.

Building eval datasets from production threads

The best source of multi-turn test cases is not a benchmark or a synthetic generator — it is your own production traffic, because that is where the real personas, the real ambiguity and the real failure modes already live. The mistake teams make is logging conversations the way they log API calls: one row per request, each response divorced from the turn before it. To evaluate multi-turn behaviour you must log conversations as threads: the full ordered sequence of user and assistant turns, with a stable thread id, so the whole exchange can be reconstructed and replayed later.

With threads logged, the workflow becomes a closed loop. First, replay: periodically run a sample of real threads back through your current agent and capture the new transcripts. Second, judge: have a human reviewer or an LLM judge mark which replayed threads now fail — drifted, forgot a constraint, looped. Third, curate: take the failing threads, trim them to the minimal sequence that reproduces the failure, label the expected behaviour, and add them to a regression set. Fourth, gate: every future change to the prompt, the model or the retrieval layer must pass that regression set before it ships. That is the loop closing — a failure that happened once in production becomes a permanent test that the same bug can never ship through twice.

This is the same golden-set discipline that underpins all serious LLM evaluation, extended from single examples to whole conversations. If you have not built golden sets before, our guide to building an LLM evaluation suite with golden sets and judges covers the foundations; the only change for multi-turn is that each golden example is a thread rather than a row. Seed the set with a handful of hand-written hard cases — a user who changes their mind halfway, one who gives a constraint then contradicts it, one who tests whether a turn-one instruction still holds at turn ten — and let production grow it from there.

Simulated-user evaluation

Curating production threads is essential, but it has one limit: you can only test the conversations that have already happened. To probe a persona you have never seen — the adversarial user, the one who switches goals mid-conversation, the impatient one who refuses to repeat themselves — you need to generate the conversation. That is what simulated-user evaluation does: a second LLM plays the user, driven by a persona and a goal, and converses with your agent for N turns, producing a complete multi-turn transcript you can then score.

The loop is simple. You define a persona and a goal — say, "a non-technical customer in Manchester trying to change a delivery address, who becomes impatient if asked the same question twice" — and a stopping condition, usually a turn cap or the goal being met. On each iteration the simulator-LLM reads the conversation so far and emits the next user message; your agent replies; the pair is appended to the transcript. Frameworks such as DeepEval ship a conversation simulator that orchestrates exactly this, but the loop is short enough to write by hand, which is the clearest way to understand it. Here it is end to end — the simulated user is prompted with a persona and the running history, your agent is a single function you provide, and the result is a transcript of alternating turns.

"""Simulated-user multi-turn evaluation loop.

A second LLM plays the user against a persona + goal, drives your agent
for up to max_turns, and returns the full transcript for scoring.
Code is US English by convention. Uses the Anthropic Messages API.
"""
from anthropic import Anthropic

client = Anthropic()
USER_MODEL = "claude-haiku"   # the simulated user can be a cheaper model

PERSONA = (
    "You are a non-technical customer in Manchester trying to change the "
    "delivery address on an existing order. You get impatient if asked the "
    "same question twice. Stop once the address is changed or you give up."
)

def simulated_user_turn(history):
    """Ask the user-LLM for the next user message, given the transcript."""
    convo = "\n".join(f"{t['role']}: {t['content']}" for t in history)
    resp = client.messages.create(
        model=USER_MODEL,
        max_tokens=256,
        system=PERSONA + " Reply ONLY as the customer, one short message. "
                         "If your goal is met or hopeless, reply exactly DONE.",
        messages=[{"role": "user",
                   "content": f"Conversation so far:\n{convo}\n\nYour next message:"}],
    )
    return resp.content[0].text.strip()

def run_simulation(agent_fn, max_turns=12):
    """Drive the agent with a simulated user; return the transcript."""
    transcript = []
    user_msg = "Hi, I need to change my delivery address."   # opening turn
    for _ in range(max_turns):
        transcript.append({"role": "user", "content": user_msg})
        agent_reply = agent_fn(transcript)          # YOUR agent under test
        transcript.append({"role": "assistant", "content": agent_reply})

        user_msg = simulated_user_turn(transcript)  # user-LLM responds
        if user_msg.strip() == "DONE":
            break
    return transcript

# `agent_fn` takes the running transcript and returns the next assistant reply.
transcript = run_simulation(my_agent)
print(f"{len(transcript)} turns collected")

Three details in that loop are load-bearing. The simulated user is given the full running history on every turn, because a user who cannot see what was already said cannot ask a coherent follow-up — that is what makes the test multi-turn rather than a sequence of unrelated questions. The persona carries an explicit stopping signal (DONE), so the conversation ends when the goal is met or abandoned rather than running to the turn cap every time. And agent_fn receives the whole transcript, so your agent under test is exercised exactly as it would be in production, with full context, not fed one isolated message. Swap the persona and you have a new test: the adversarial user, the goal-switcher, the one who states a constraint in turn one purely to see whether turn ten still honours it.

Metrics and frameworks: scoring the whole conversation

Once you have transcripts — replayed from production or generated by simulation — you need to score them, and the crucial distinction is between turn-level and conversation-level metrics. A turn-level metric judges a single assistant response within its context, typically over a sliding window of recent turns; it answers "was this reply relevant and faithful?". A conversation-level metric judges the whole exchange holistically; it answers "did the agent retain what the user told it, stay in role, and actually complete the task across all the turns?". Multi-turn failure modes are overwhelmingly conversation-level, so that is where most of your scoring belongs.

As of June 2026, three frameworks dominate, and they are complementary rather than competing. DeepEval has the richest multi-turn toolkit: a conversation simulator plus a set of conversation-level metrics — KnowledgeRetentionMetric for attrition, RoleAdherenceMetric and TopicAdherenceMetric for drift, ConversationCompletenessMetric for task completion, and ConversationalGEval for any custom criterion expressed in natural language. RAGAS provides MultiTurnSample together with AspectCritic, which lets you write a conversation-level pass/fail criterion in plain English and have an LLM judge it as a binary outcome. MLflow's mlflow.genai.evaluate() runs judges over full execution traces and multi-turn conversations and ties the whole thing into experiment tracking, and it can run RAGAS metrics as scorers, so many teams simulate and score with DeepEval or RAGAS and orchestrate runs and history through MLflow.

As of June 2026. All three projects move quickly — class names, imports and judge defaults change between releases — so confirm the current API against each project's own documentation before you build around it.

Underneath all of these sits the same engine you should already understand: an LLM acting as judge over the transcript. The conversation-level metrics above are, in the main, structured LLM-as-judge calls with the whole conversation in context and a specific rubric. If you have not set up a judge before — calibrated it against human labels, controlled for its biases — read our guide to running LLM-as-judge evals in production first, because a multi-turn judge inherits every pitfall of a single-turn one and adds the difficulty of reasoning over a long, stateful transcript.

A conversation-level pass/fail metric

The most useful custom metric for multi-turn work is a conversation-level criterion that returns a clear pass or fail with a reason — the dialogue equivalent of RAGAS's AspectCritic. You write the criterion in plain language ("the assistant never contradicts a constraint the user stated earlier in the conversation"), hand the whole transcript to a judge model, and get back a binary verdict and a short justification you can log. Below is a self-contained version you can drop into any pipeline; it judges the entire transcript at once, which is what makes it conversation-level rather than turn-level.

"""Conversation-level pass/fail metric (AspectCritic-style).

Judges a WHOLE transcript against a natural-language criterion and
returns pass/fail + reason. Conversation-level: the full transcript
is in context, not a single turn. Code is US English by convention.
"""
import json
from anthropic import Anthropic

client = Anthropic()
JUDGE_MODEL = "claude-opus"   # use a strong model as the judge

def conversation_criterion(transcript, criterion):
    """Return {'passed': bool, 'reason': str} for a conversation criterion."""
    convo = "\n".join(f"{t['role'].upper()}: {t['content']}" for t in transcript)
    system = (
        "You are a strict evaluator of multi-turn conversations. "
        "Read the ENTIRE conversation, then judge it against the criterion. "
        "Reply ONLY with JSON: {\"passed\": true|false, \"reason\": \"...\"}. "
        "Fail if the criterion is violated on ANY turn."
    )
    prompt = f"CRITERION:\n{criterion}\n\nCONVERSATION:\n{convo}\n\nVerdict as JSON:"
    resp = client.messages.create(
        model=JUDGE_MODEL,
        max_tokens=512,
        system=system,
        messages=[{"role": "user", "content": prompt}],
    )
    return json.loads(resp.content[0].text)

# Example: the constraint-retention criterion that catches knowledge attrition.
CRITERION = (
    "The assistant must never contradict or forget any constraint the user "
    "stated earlier in the conversation (e.g. budget, language, city, "
    "dietary rule). If the user set a constraint and a later assistant turn "
    "violates it, the conversation FAILS."
)

result = conversation_criterion(transcript, CRITERION)
print(result["passed"], "—", result["reason"])
assert result["passed"], f"Constraint retention failed: {result['reason']}"

The shape of that function is deliberately the same as a RAGAS AspectCritic over a MultiTurnSample, or a DeepEval ConversationalGEval: a natural-language criterion, the whole conversation in context, and a binary verdict with a reason. Writing it by hand once shows you what those frameworks are doing under the bonnet, and the reason string is what makes it usable in practice — when the metric fails in CI you want to read why, not just see a red cross. Keep each criterion narrow and single-purpose (one for constraint retention, one for staying in role, one for task completion) so that a failure points at exactly one capability, mirroring the per-row discipline of the failure-mode catalogue.

Async evaluation and the continuous-improvement loop

None of this needs to run in the request path. Conversation-level judging is naturally an offline, asynchronous activity: you log threads in production, then on a schedule — nightly, or per deploy — replay and judge them in the background, well away from the latency budget your users feel. That offline posture is what lets you afford a strong, slow judge model over a long transcript without anyone waiting on it, and it is the natural home for batch-style evaluation jobs over thousands of logged conversations.

The loop that ties it together is the same closed loop from earlier, now automated. Production threads are logged; an async job replays a sample through the current agent; conversation-level metrics and custom criteria judge each transcript; failures are curated into the regression set; and that regression set runs in continuous integration so the next change cannot ship a known multi-turn failure. Putting these evals in CI is the step that turns evaluation from a one-off audit into a ratchet that only ever tightens — our guide to putting evals in CI for prompt and agent regression testing covers the mechanics, and the only multi-turn-specific change is that each test case is a thread and each assertion is a conversation-level metric rather than a single-turn one.

One practical note for teams whose agents retrieve: a conversational RAG system stacks two evaluation problems — retrieval quality on each turn, and conversational behaviour across turns. Score them separately. Use turn-level faithfulness and context-precision checks for the retrieval, as our guide to RAG evaluation with RAGAS, faithfulness and context precision describes, and layer the conversation-level metrics on top for drift, retention and completeness. Conflating the two is how teams end up with a "RAG is broken" ticket that turns out to be a multi-turn memory bug.

Benchmarks to know

You will not use academic benchmarks as your product metric — they test general capability, not your specific agent — but two are worth knowing as references for what good multi-turn evaluation tests for. MultiChallenge (Scale AI, arXiv 2501.17399) probes four realistic challenges: instruction retention, inference memory of user information, reliable versioned editing, and self-coherence, and uses an LLM-as-judge with instance-level rubrics. Its headline finding is the one that should change how you build: frontier models that ace older multi-turn tests scored below 50% on it in the benchmark's 2025 release, with even the strongest models around the low forties (leaders have since improved). MT-Eval (arXiv 2401.16745, EMNLP 2024) categorises multi-turn interaction into recollection, expansion, refinement and follow-up, builds matched single-turn versions of each multi-turn query, and shows significant degradation in the multi-turn setting that is not explained by the model's raw capability — error propagation and the distance to relevant earlier content are the culprits.

The reason to read these is not to copy their scores but to copy their taxonomy. MultiChallenge's four challenges and MT-Eval's four interaction types are, between them, a ready-made checklist of the capabilities your own eval suite should exercise — and they confirm the central claim of this guide from the research side: the multi-turn setting is where models fail, in ways the single-turn setting cannot predict. Use them as a map of what to test; use your production threads and simulated users to test it on your actual agent.

Conclusion and next steps

The argument of this guide is simple: evaluate the conversation, not the turn. Single-turn evals pass the easy fifth of the behaviour and ship the hard four-fifths blind, and the benchmarks prove it — the same models that look flawless on one-shot tests fall apart when instruction retention and memory are actually probed. The fix is a system, not a metric: catalogue the multi-turn failure modes and detect each one explicitly; log conversations as threads and curate failing ones into a closed-loop regression set; generate the conversations you have not seen yet with simulated users; and score everything with conversation-level metrics from DeepEval, RAGAS or MLflow, judged offline and gated in CI.

Start this week with one agent and one failure mode. Pick constraint retention, write the simulated-user persona that tests it, run the loop, score the transcript with a single conversation-level criterion, and add the failing case to a regression set in your CI. That is the whole loop in miniature, and it scales out from there to every persona and every mode that matters for your product. If you are the engineer who built the eval harness that caught a multi-turn memory bug before it reached users, that is exactly the kind of shipped, measurable work the people hiring in AI across India and the UK want to see — and a Verified Builder profile on AI Tech Connect is where you put it in front of them.

Sources

• DeepEval — Multi-turn evaluation metrics
• DeepEval — Conversation Simulator
• DeepEval — Conversational G-Eval
• RAGAS — Evaluating multi-turn conversations (MultiTurnSample, AspectCritic)
• MLflow — LLM and agent evaluation (mlflow.genai.evaluate)
• MultiChallenge — A realistic multi-turn benchmark (arXiv 2501.17399)
• MT-Eval — A multi-turn capabilities benchmark (arXiv 2401.16745)