What are RAG evaluation metrics?
A RAG pipeline has two places it can fail: the retriever can pull back the wrong chunks (or miss the right ones), and the generator can drift off-topic or invent facts that sound plausible but aren't in the context it was given. RAG evaluation metrics exist to measure each edge separately, because "the final answer looked reasonable" doesn't tell you which part of the pipeline to fix when it doesn't.
Most frameworks converge on the same four signals — two for retrieval, two for generation — computed by an LLM-as-judge over a question / context / answer triple, usually without needing a human-written reference answer for at least the generation-side metrics (RAGAS docs summary; TruLens RAG triad).
The RAG triad: context relevance, groundedness, answer relevance
TruEra (now folded into TruLens) coined the RAG triad as a compact way to reason about hallucination: context relevance (is each retrieved chunk relevant to the query?), groundedness (is the answer's content actually supported by that context?), and answer relevance (does the final answer address what was asked?). If all three score well, the app is hallucination-free up to the limits of its own knowledge base — it can still be wrong if the knowledge base itself is wrong (TruEra — What is the RAG Triad; TruLens docs).
This is the same idea RAGAS implements under different names — groundedness ≈ faithfulness, context relevance ≈ context precision — which is why the two frameworks' scores are directly comparable even though the terminology doesn't match exactly.
The four core RAGAS metrics
RAGAS is the most widely cited reference implementation of RAG metrics. Its four load-bearing scores split cleanly along the retrieval/generation line:
| Metric | Side | Question it answers | Needs ground truth? |
|---|---|---|---|
| Faithfulness | Generation | Is every claim in the answer supported by the retrieved context? | No |
| Answer Relevancy | Generation | Does the answer actually address the question asked? | No |
| Context Precision | Retrieval | Are the retrieved chunks relevant, and are relevant ones ranked first? | No |
| Context Recall | Retrieval | Did retrieval pull back everything needed to answer? | Yes |
Faithfulness is computed by decomposing the answer into atomic claims with an LLM, checking each against the retrieved context, and scoring supported claims over total claims — one of two unsupported claims gives 0.5, not zero, which is why claim-level decomposition catches partial hallucination that a holistic "is this grounded, yes/no" judgment tends to round up and miss (RAGAS metrics explained; RAG evaluation deep dive). A score below roughly 0.7–0.8 is a commonly cited threshold for meaningful hallucination worth investigating — treat it as a rule of thumb, not a hard industry standard, since the exact cutoff is workload-dependent.
Context precision penalizes irrelevant chunks, especially ones ranked high — it's an average-precision-style calculation over an LLM's relevant/irrelevant verdict on each retrieved chunk. Context recall is the one metric here that needs a ground-truth answer: it checks whether the information required to answer correctly was retrieved at all, independent of whether the generator used it well.
RAGAS vs TruLens vs DeepEval: which framework for which stage
The three metrics are the same in spirit across tools; what differs is workflow fit. All three use an LLM-as-judge and support the core RAG triad, but they're built for different points in the lifecycle (Atlan — RAGAS vs TruLens vs DeepEval; framework selection guide).
| Aspect | RAGAS | TruLens | DeepEval |
|---|---|---|---|
| Core idea | Batch metrics library for RAG pipelines | Live tracing + RAG-triad feedback functions | Pytest-style assertions for CI |
| Ground truth needed | Only for context recall | No (reference-free triad) | Optional, both modes supported |
| Best fit | Offline benchmarking, comparing configs | Production monitoring on live traffic | Gating merges / regression tests in CI |
| Output | Aggregate scores over a dataset | Per-call scores attached to traces | Pass/fail per test case |
A pattern that shows up repeatedly in practitioner write-ups: use RAGAS (or TruLens) during development to compare pipeline configurations, add DeepEval as a pytest-style gate in CI once the eval set is stable, and run TruLens or Arize Phoenix in production to watch scores on live traffic rather than only at deploy time — the three don't conflict and are commonly run together (Atlan comparison; RAGAS vs TruLens vs DeepEval).
Get the weekly AI engineering brief
RAG, agents, evals, and the tools worth using — one practical email a week. Plus the free roadmap PDF.
By subscribing you agree to receive emails from AI Engineer Insights. Unsubscribe anytime. See our Privacy Policy.
Building an eval set that's actually worth running
- • Start from real queries, not made-up ones. Pull from support logs, product feedback, or early user sessions — synthetic-only test sets miss the messy phrasing real users produce.
- • Cover the failure modes you actually care about, not just "does it work": ambiguous questions, questions with no answer in the knowledge base, multi-hop questions that need two chunks combined.
- • Label ground truth only where you need it — faithfulness and answer relevancy don't require it, so spend labeling effort on context recall and answer correctness, the metrics that do.
- • Re-run 2–3 times and average on borderline cases; LLM-judge scores are reported to vary by roughly ±0.05 between runs on the same input due to judge non-determinism.
Where RAG eval scores lie to you
A high score is not proof of correctness. Independent benchmarking has found that none of the major frameworks can distinguish a factually wrong retrieved context from a correct one — the metrics check whether the answer is consistent with what was retrieved, not whether what was retrieved is true. A pipeline can score 0.95 on faithfulness and still confidently deliver the wrong business answer if the underlying document was stale or incorrect (Atlan — independent benchmark findings).
The judge model matters more than the framework you pick around it: since all three tools defer scoring to an LLM, a poorly calibrated or biased judge produces misleading dashboards regardless of which library computed the number. Treat eval scores as a triage signal that tells you where to look — not a certificate of correctness.
Frequently Asked Questions
What is RAG evaluation?
RAG evaluation is measuring whether a retrieval-augmented generation pipeline retrieves the right context and generates an answer that's grounded in it. It splits into two failure edges: retrieval quality (did we find the right chunks?) and generation faithfulness (did the model use them correctly, without inventing anything?). Standard frameworks like RAGAS and TruLens score both with an LLM-as-judge.
What is the RAG triad?
The RAG triad, coined by TruEra/TruLens, is three evaluations that together catch hallucination: context relevance (are retrieved chunks relevant to the query?), groundedness (is the answer supported by the retrieved context?), and answer relevance (does the answer address the question?). Passing all three gives confidence the app is hallucination-free up to the limits of its knowledge base.
What is faithfulness in RAG evaluation?
Faithfulness measures what fraction of an answer's claims are actually supported by the retrieved context. RAGAS computes it by decomposing the answer into atomic statements with an LLM, checking each against the context, and scoring supported/total. A score below roughly 0.7–0.8 is commonly used as a threshold that flags meaningful hallucination worth investigating.
RAGAS vs TruLens vs DeepEval — which should I use?
They fit different stages. RAGAS is the fastest path to scored, standardized RAG metrics for offline comparison of pipeline configurations. TruLens adds OpenTelemetry-based tracing so you can watch faithfulness and relevance drift on live production traffic, not just an eval set. DeepEval is built pytest-style, so RAG quality becomes a CI gate that can block a merge. Many teams run more than one: RAGAS or TruLens in development, DeepEval in CI, TruLens (or Arize Phoenix) in production.
Does a high faithfulness score mean the RAG system is correct?
No. Faithfulness only checks that the answer's claims trace back to the retrieved context — it can't tell you if that context was itself wrong or stale. A RAG system can score 0.95 on faithfulness and still give a confidently wrong business answer if the underlying documents were outdated or incorrect. RAG evaluation frameworks operate at the inference layer; they don't substitute for keeping the knowledge base accurate.
Do I need labeled ground truth to evaluate a RAG pipeline?
Not for the core generation-side metrics. Faithfulness and answer relevancy are reference-free — they only need the question, retrieved context, and generated answer. Context recall is the exception: it requires a ground-truth answer to check whether retrieval pulled back everything necessary, so it's typically added once you have a labeled eval set.
References
- • RAGAS — RAG evaluation metrics documentation
- • RAG Evaluation Metrics Explained (RAGAS breakdown)
- • Ragas Metrics Explained: Faithfulness, Context Precision and Recall
- • TruLens — The RAG Triad
- • TruEra — What is the RAG Triad?
- • Atlan — RAGAS vs TruLens vs DeepEval, 2026 Guide
- • RAG Evaluation Frameworks Compared: RAGAS vs TruLens vs DeepEval
- • RAG Evaluation: RAGAS, Faithfulness, and LLM-as-Judge