Evaluating RAG with RAGAS: Faithfulness, Context Precision and Recall

What you need to know

• RAGAS scores RAG without answer labels. The Retrieval-Augmented Generation Assessment framework is reference-free for most of its metrics: it uses an LLM to judge retrieval and generation quality straight from the question, the retrieved context and the generated answer, so you can start measuring before you have a hand-built set of correct answers.
• Four core metrics, split into two halves. Faithfulness and answer relevancy judge the generation; context precision and context recall judge the retrieval. The split is the whole point — it tells you which half of the pipeline to fix.
• Start with faithfulness and context recall. Faithfulness catches hallucination. Context recall tells you whether your retrieval architecture is fundamentally sound. Get those two stable, then add the other two.
• Thresholds are starting points, not laws. The commonly cited defaults are useful anchors, but the right bar depends on your domain and the cost of being wrong. Calibrate against a human-labelled slice of your own traffic before gating releases.
• Wire it into CI and tracing. RAGAS is the framework; DeepEval gives you pytest-style assertions for merge gates; Langfuse runs the same metrics over live traffic. Mix them per need.

Every team that ships retrieval-augmented generation hits the same wall. The demo is flawless. The first fifty internal questions get good answers. Then the system goes to real users in Bengaluru and London, and within a week someone forwards a screenshot where the assistant confidently cited a refund policy that does not exist, or answered a question about the wrong product because the right document never made it into the context window. The answer was fluent. It was well structured. It was wrong. And nothing in your logs flagged it, because a wrong answer and a right answer look identical to a system that only checks whether the response parsed.

That gap — between fluent and correct — is what RAG evaluation exists to close. You cannot improve what you do not measure, and "it seems fine when I try it" is not measurement. This guide is about RAGAS, the open-source framework that has become the common vocabulary for talking about RAG quality, and specifically about its four core metrics: faithfulness, answer relevancy, context precision and context recall. By the end you will know what each one measures, what a low score is actually telling you, which two to start with, and how to turn the whole thing into a CI gate that stops a regression before it merges.

This is the evaluation companion to the retrieval guides we have published before. If you have read our production RAG hybrid-retrieval guide or our piece on chunking and embedding strategies, this is the part that tells you whether any of those choices actually helped. It also sits directly underneath today's sibling on adaptive RAG and query routing — the metrics here are exactly the signal that drives a router to escalate a hard query from naive to hybrid to agentic retrieval.

Why RAG needs its own evaluation

A RAG pipeline is two systems bolted together. First a retriever takes the user's question and pulls a handful of chunks out of a vector store or a hybrid index. Then a generator — the LLM — reads those chunks and writes an answer. When the answer is bad, the bad could have come from either stage, and the two require completely different fixes. A retrieval failure means the model never saw the right information; you fix it with better chunking, better embeddings, a reranker, or a larger top-k. A generation failure means the model had the right information and ignored it, padded around it, or invented something on top of it; you fix it with a better prompt, a stronger model, or firmer grounding instructions.

A single blended quality score hides this. If you only know "the answer was 6 out of 10", you do not know whether to spend the next sprint on your retriever or your prompt. The genius of the RAGAS metric set is that it deliberately separates the two halves of the pipeline so the number tells you where to look. That is the practical value of measuring all four metrics rather than collapsing them into one figure that looks tidy on a dashboard and tells you nothing actionable.

The second thing that makes RAG hard to evaluate is that you usually do not have ground-truth answers. Standard machine-learning evaluation assumes a labelled test set: input, expected output, compare. For a question-answering system over your own corpus, hand-writing the correct answer to thousands of questions is expensive, and the corpus changes underneath you. RAGAS sidesteps much of this by being reference-free for several metrics — it uses an LLM to judge quality directly, rather than comparing against a fixed gold answer. That is what lets you start evaluating on day one instead of waiting months for a labelled set.

The four core metrics

RAGAS exposes more than four metrics, but four carry most of the diagnostic load. Two judge the generation, two judge the retrieval. Read them as two pairs and the whole picture clicks into place.

Faithfulness — does the answer stay grounded?

Faithfulness asks a deceptively simple question: of the factual claims the answer makes, how many are actually supported by the retrieved context? The judge decomposes the answer into individual claims and checks each one against what was retrieved. An answer that invents a figure, a date or a policy that nowhere appears in the context scores low. This is your primary hallucination detector, and it is a generation-side metric — the context was there; the model went beyond it. A commonly cited default threshold is around 0.75, meaning roughly three in four claims should be traceable to the context before you are comfortable. For a regulated assistant you would push that far higher.

Answer relevancy — is the answer actually responsive?

An answer can be perfectly faithful and still useless if it talks around the question. Answer relevancy measures how directly the response addresses what was asked, penalising padding, hedging, evasion and off-topic detours. It is the metric that catches the model that produces three confident paragraphs that never quite answer the question. This is also generation-side. A common default lands around 0.8. Low answer relevancy with high faithfulness usually means a prompting problem — the model is grounded but unfocused.

Context precision — is the retrieved context clean and well-ranked?

Context precision moves to the retrieval side. Of everything the retriever pulled in, how much was actually relevant to the question, and is the relevant material ranked near the top? A retriever that returns ten chunks where only two matter, with the two buried at positions eight and nine, scores poorly even if the answer eventually came out right. Precision matters because irrelevant context costs tokens, dilutes the signal the generator sees, and on a bad day actively misleads it. A typical default sits around 0.7.

Context recall — did retrieval bring back everything needed?

Context recall is the one that tells you whether your retrieval architecture is fundamentally sound. It asks whether the retrieved context contains all the information required to answer the question completely. If the answer needs three facts and retrieval surfaced only two, recall is low and no amount of prompt engineering will save you — the generator simply never saw the third fact. This is the metric that most often needs a reference answer, because to know whether everything required was retrieved you generally need to know what a complete answer looks like. A common default sits around 0.8.

Metric	What it measures	A low score means	Common default
Faithfulness (generation)	Share of answer claims supported by the retrieved context	The model is hallucinating — inventing claims the context does not back	~0.75
Answer relevancy (generation)	How directly the answer addresses the actual question	The answer is padded, evasive or off-topic — a prompting problem	~0.8
Context precision (retrieval)	How much retrieved context is relevant, and how well it is ranked	The retriever returns noise or buries the relevant chunks — tune ranking/top-k	~0.7
Context recall (retrieval)	Whether retrieval brought back all the context needed to answer	The retriever misses required information — fix chunking/embeddings/coverage	~0.8 (usually needs a reference answer)

Reading the metrics: retrieval problem or generation problem?

This is where the four-metric split earns its keep. Because the metrics are paired by pipeline stage, the pattern of scores points straight at the broken component. You stop guessing and start diagnosing.

If context precision or context recall is low, you have a retrieval problem. The generator is not the issue — it is being handed bad raw material. The levers are all upstream: your chunking strategy may be splitting facts across boundaries, your embedding model may not capture the domain's vocabulary, you may need a reranker to lift the relevant chunks to the top, or your top-k may be too small to bring back everything required. Our chunking and embedding guide and the hybrid-retrieval guide are the playbooks for exactly these fixes.

If context precision and recall are healthy but faithfulness or answer relevancy is low, you have a generation problem. Retrieval did its job — the right information was in the context window — but the model hallucinated on top of it, ignored it, or wandered off the question. The levers here are the prompt (are your grounding instructions firm enough?), the model (is it strong enough for this task?), and the way you present the context (are you asking it to cite, to answer only from the provided text, to say "I do not know" when the context is silent?).

Running RAGAS over an evaluation dataset

The mechanics are straightforward. You build a small evaluation dataset — questions, the contexts your retriever returned, the answers your generator produced, and reference answers where a metric needs them — and you hand it to ragas.evaluate() with the metrics you care about. The dataset can be hand-curated, which is the most reliable, or LLM-synthesised and then human-checked, which scales better. Even fifty well-chosen questions covering your real query distribution will tell you more than a vague sense that things "seem fine".

Here is a worked example using a Hugging Face Dataset with the columns RAGAS expects, scored on all four core metrics. See the RAGAS documentation for the current column names and metric imports, which shift between releases.

from datasets import Dataset
from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

# One row per evaluation question. `contexts` is the list of chunks the
# retriever returned; `ground_truth` is a reference answer (needed for recall).
samples = {
    "question": [
        "What is the refund window for an annual plan?",
        "Which AWS region hosts the London tenant?",
    ],
    "answer": [
        "Annual plans can be refunded within 14 days of purchase.",
        "The London tenant runs in the eu-west-2 region.",
    ],
    "contexts": [
        ["Refunds: annual subscriptions are eligible for a full refund "
         "within 14 days of the original purchase date."],
        ["Tenant routing: UK customers are served from AWS London "
         "(eu-west-2); India customers from AWS Mumbai (ap-south-1)."],
    ],
    "ground_truth": [
        "Annual plans are refundable within 14 days of purchase.",
        "The London tenant is hosted in AWS London, eu-west-2.",
    ],
}

dataset = Dataset.from_dict(samples)

result = evaluate(
    dataset,
    metrics=[
        faithfulness,        # generation: are claims grounded?
        answer_relevancy,    # generation: is the answer on-point?
        context_precision,   # retrieval: is the context clean and ranked well?
        context_recall,      # retrieval: did we retrieve everything needed?
    ],
)

print(result)
# {'faithfulness': 0.98, 'answer_relevancy': 0.91,
#  'context_precision': 0.88, 'context_recall': 0.84}

The output is a score per metric, averaged across the dataset. Track those four numbers per commit. A drop in context recall after a chunking change, or a drop in faithfulness after a model swap, is exactly the kind of regression that is invisible to manual spot-checking but obvious on a tracked metric. The point is to make quality a number that moves with your code, not a feeling that moves with your mood.

Gating merges in CI with DeepEval

A tracked number is good; a number that blocks a bad merge is better. The cleanest way to turn RAGAS-style metrics into a release gate is DeepEval, which wraps the same metrics in pytest-style assertions so a quality regression fails the build exactly like a unit test would. You write test cases, set thresholds, and your CI pipeline refuses to merge a change that drops faithfulness below the bar. See the DeepEval documentation for the current metric classes and runner.

# test_rag_quality.py — run in CI with `deepeval test run test_rag_quality.py`
import pytest
from deepeval import assert_test
from deepeval.test_case import LLMTestCase
from deepeval.metrics import (
    FaithfulnessMetric,
    ContextualRecallMetric,
)

# Thresholds are starting points — calibrate them to YOUR domain.
faithfulness = FaithfulnessMetric(threshold=0.75, model="gpt-5.2")
context_recall = ContextualRecallMetric(threshold=0.80, model="gpt-5.2")

# Pull these from your golden set; one LLMTestCase per evaluation question.
golden_cases = load_golden_set("eval/rag_golden.jsonl")


@pytest.mark.parametrize("case", golden_cases)
def test_rag_pipeline(case):
    test_case = LLMTestCase(
        input=case["question"],
        actual_output=run_pipeline(case["question"]),   # your live RAG call
        expected_output=case["reference_answer"],
        retrieval_context=case["retrieved_chunks"],
    )
    # Fails the build if either metric drops below threshold.
    assert_test(test_case, [faithfulness, context_recall])

Wire that into your pull-request pipeline and a change that quietly breaks retrieval can no longer reach production. Pair it with online evaluation on live traffic — tools such as Langfuse trace real requests and can run RAGAS metrics over a sample of production calls, so you catch drift that your offline golden set never anticipated. The pattern is the same one we recommend for any LLM feature in our LLM evaluation-suite guide: a golden set for the merge gate, live tracing for the long tail.

The 2026 tooling ecosystem

By mid-2026 the RAG-evaluation landscape has settled into a small set of tools that each do one thing well, and the sensible move is to mix them rather than pick one. RAGAS itself is the conceptual framework and the metric definitions — the shared vocabulary everyone reaches for. DeepEval brings pytest-style assertions and slots straight into CI/CD. Langfuse handles production tracing and can run RAGAS metrics inside your live pipeline for online evaluation. Hallucination-detection specialists such as Patronus and its Lynx model focus narrowly on the faithfulness problem when you need a dedicated, hardened detector for high-stakes answers.

None of these are mutually exclusive. A common production setup uses RAGAS metrics as the definition, DeepEval to gate merges in CI, and Langfuse to trace and sample live traffic — with a hallucination detector layered on top for the answers that carry real risk. The judging philosophy underneath all of them is LLM-as-judge, which we cover in depth in our guide to LLM-as-a-judge evals that hold up in production. RAG evaluation is one application of that broader technique; agent evaluation is another, which our guide to evaluating AI agents on trajectory, tool calls and outcome takes further for systems where retrieval is just one step in a longer chain.

Pitfalls that quietly invalidate your scores

RAGAS will hand you four clean-looking numbers whether or not they mean anything, so the discipline is in trusting them only when they have earned it. A handful of pitfalls catch almost everyone the first time.

The judge is itself a noisy model

LLM-judge metrics are model-dependent and noisy. The same dataset scored with a different judge model, or even the same model on a different day, can move by a few points. Pin the judge model and its version, run multiple samples per item and average them, and accept that small movements are noise rather than signal. A two-point wobble between runs is not a regression; a ten-point drop after a chunking change probably is.

Treating thresholds as universal law

The default thresholds in this guide — faithfulness around 0.75, the rest around 0.7 to 0.8 — are anchors to start from, not bars handed down from on high. A documentation bot and a medical-advice assistant should not share a faithfulness threshold. Calibrate against a human-labelled slice of your own traffic: have people score a few dozen answers, see where the RAGAS judge agrees and disagrees with them, and set your gate where the metric and human judgement line up for your domain and your tolerance for a wrong answer.

Optimising a single metric

The fastest way to ship a worse system that looks better is to optimise one metric in isolation. Push faithfulness to 0.99 by making the model refuse to answer anything not spelled out verbatim, and you will tank answer relevancy as it stops being helpful. This is Goodhart's law in miniature: when a measure becomes a target, it stops being a good measure. Track the whole set and watch them move together. A change that lifts one metric while sinking another has not improved your system; it has moved the failure somewhere your dashboard was not looking.

Conclusion: make RAG quality a number you can trust

The progression for any serious RAG system is the same. Stop relying on the feeling that answers "seem fine". Build a small evaluation dataset that reflects your real query distribution. Score it with RAGAS on faithfulness and context recall first — hallucination and retrieval soundness — and read the two halves of the pipeline separately so you always know whether to fix the retriever or the prompt. Add answer relevancy and context precision to refine once the first two are stable. Gate your merges in CI with DeepEval so a regression cannot reach production, and trace live traffic with Langfuse so the failures your golden set never imagined still surface.

Do that and RAG quality stops being a mystery you debug by screenshot and becomes a number that moves with your code — one you can defend to a stakeholder, calibrate to your domain, and feed back into routing so the system works harder only where the evaluation says it must. The teams across India and the UK shipping RAG that holds up are not the ones with the cleverest prompts. They are the ones who measured the two halves of the pipeline, calibrated their judge against real people, and let the numbers tell them where the work was.

For the retrieval side of these fixes, start with our hybrid-retrieval guide; for the judging machinery underneath every metric here, read our LLM-as-a-judge guide. Evaluation is the part of a RAG system that turns "it seems to work" into "we know it works, and here is the proof".