RAG Evaluation Metrics: Faithfulness, Precision, Recall

What are RAG evaluation metrics?

A RAG pipeline has two places it can fail: the retriever can pull back the wrong chunks (or miss the right ones), and the generator can drift off-topic or invent facts that sound plausible but aren't in the context it was given. RAG evaluation metrics exist to measure each edge separately, because "the final answer looked reasonable" doesn't tell you which part of the pipeline to fix when it doesn't.

Most frameworks converge on the same four signals — two for retrieval, two for generation — computed by an LLM-as-judge over a question / context / answer triple, usually without needing a human-written reference answer for at least the generation-side metrics (RAGAS docs summary; TruLens RAG triad).

The RAG triad: context relevance, groundedness, answer relevance

TruEra (now folded into TruLens) coined the RAG triad as a compact way to reason about hallucination: context relevance (is each retrieved chunk relevant to the query?), groundedness (is the answer's content actually supported by that context?), and answer relevance (does the final answer address what was asked?). If all three score well, the app is hallucination-free up to the limits of its own knowledge base — it can still be wrong if the knowledge base itself is wrong (TruEra — What is the RAG Triad; TruLens docs).

This is the same idea RAGAS implements under different names — groundedness ≈ faithfulness, context relevance ≈ context precision — which is why the two frameworks' scores are directly comparable even though the terminology doesn't match exactly.

The four core RAGAS metrics

RAGAS is the most widely cited reference implementation of RAG metrics. Its four load-bearing scores split cleanly along the retrieval/generation line:

Metric	Side	Question it answers	Needs ground truth?
Faithfulness	Generation	Is every claim in the answer supported by the retrieved context?	No
Answer Relevancy	Generation	Does the answer actually address the question asked?	No
Context Precision	Retrieval	Are the retrieved chunks relevant, and are relevant ones ranked first?	No
Context Recall	Retrieval	Did retrieval pull back everything needed to answer?	Yes

Faithfulness is computed by decomposing the answer into atomic claims with an LLM, checking each against the retrieved context, and scoring supported claims over total claims — one of two unsupported claims gives 0.5, not zero, which is why claim-level decomposition catches partial hallucination that a holistic "is this grounded, yes/no" judgment tends to round up and miss (RAGAS metrics explained; RAG evaluation deep dive). A score below roughly 0.7–0.8 is a commonly cited threshold for meaningful hallucination worth investigating — treat it as a rule of thumb, not a hard industry standard, since the exact cutoff is workload-dependent.

Context precision penalizes irrelevant chunks, especially ones ranked high — it's an average-precision-style calculation over an LLM's relevant/irrelevant verdict on each retrieved chunk. Context recall is the one metric here that needs a ground-truth answer: it checks whether the information required to answer correctly was retrieved at all, independent of whether the generator used it well.

RAGAS vs TruLens vs DeepEval: which framework for which stage

The three metrics are the same in spirit across tools; what differs is workflow fit. All three use an LLM-as-judge and support the core RAG triad, but they're built for different points in the lifecycle (Atlan — RAGAS vs TruLens vs DeepEval; framework selection guide).

Aspect	RAGAS	TruLens	DeepEval
Core idea	Batch metrics library for RAG pipelines	Live tracing + RAG-triad feedback functions	Pytest-style assertions for CI
Ground truth needed	Only for context recall	No (reference-free triad)	Optional, both modes supported
Best fit	Offline benchmarking, comparing configs	Production monitoring on live traffic	Gating merges / regression tests in CI
Output	Aggregate scores over a dataset	Per-call scores attached to traces	Pass/fail per test case

A pattern that shows up repeatedly in practitioner write-ups: use RAGAS (or TruLens) during development to compare pipeline configurations, add DeepEval as a pytest-style gate in CI once the eval set is stable, and run TruLens or Arize Phoenix in production to watch scores on live traffic rather than only at deploy time — the three don't conflict and are commonly run together (Atlan comparison; RAGAS vs TruLens vs DeepEval).

Faithful but wrong: why you also need an answer correctness metric

A faithfulness score of 0.95 doesn't mean the answer is right — it means the answer is consistent with whatever was retrieved. If the retrieved chunk itself contains a stale price or a wrong number, a perfectly faithful answer still repeats the error. Answer correctness closes that gap by scoring the generated answer against a human-written reference answer, not against the retrieved context. RAGAS computes it as a weighted blend of a factual F1 score (claims the answer and reference agree on, versus false positives and false negatives) and the semantic similarity between the two texts — commonly weighted around 0.75 factual / 0.25 semantic (RAGAS answer correctness docs). One practitioner checklist for production RAG puts a concrete number on the gap: a pipeline that scored 0.91 on answer relevancy and 0.88 on faithfulness still shipped a wrong account number to a user, because the model faithfully repeated a chunk that was itself incorrect — the exact failure mode faithfulness cannot catch (RAG evaluation metrics — a 12-point checklist).

The practical rule: ship both. Gate on faithfulness (commonly ≥0.9–0.95) to catch ungrounded claims, and gate separately on answer correctness against a small labeled gold set (commonly ≥0.85) to catch the faithful-but-wrong case. The two metrics fail independently and point at different fixes — a faithfulness miss means tighten the generation prompt, an answer-correctness miss with high faithfulness means go fix the retrieved document or the knowledge base, not the model.

Taking RAG eval into production: Arize Phoenix

RAGAS and DeepEval answer "is this pipeline good on my eval set" offline. The open question in production is different: is quality drifting on live traffic, and which span in a multi-step trace caused a bad answer? Arize Phoenix, built on OpenTelemetry/ OpenInference, captures spans for each pipeline step — embedding, retrieval, reranking, generation — and runs LLM-as-judge evaluators (hallucination, QA correctness, relevance) against those spans after the fact, so a bad answer can be traced to the exact step that produced it rather than treated as one opaque failure (Arize — LLM evaluation methods and metrics guide; Arize Phoenix — open-source LLM observability and eval guide). It runs fully locally with Apache-2.0 licensing, which matters for teams whose data-governance review blocks sending every production trace to a third-party SaaS.

The resulting production stack most teams converge on: RAGAS or TruLens during development to compare configurations, DeepEval as a pytest-style CI gate once the eval set stabilizes, and Phoenix (or TruLens) in production to watch faithfulness and answer-correctness for drift on a sampled slice of real traffic — one documented pattern samples roughly 5% of production traffic weekly and alerts on faithfulness drift before users report problems.

Get the weekly AI engineering brief

RAG, agents, evals, and the tools worth using — one practical email a week. Plus the free roadmap PDF.

By subscribing you agree to receive emails from AI Engineer Insights. Unsubscribe anytime. See our Privacy Policy.

Building an eval set that's actually worth running

• Start from real queries, not made-up ones. Pull from support logs, product feedback, or early user sessions — synthetic-only test sets miss the messy phrasing real users produce.
• Cover the failure modes you actually care about, not just "does it work": ambiguous questions, questions with no answer in the knowledge base, multi-hop questions that need two chunks combined.
• Label ground truth only where you need it — faithfulness and answer relevancy don't require it, so spend labeling effort on context recall and answer correctness, the metrics that do.
• Re-run 2–3 times and average on borderline cases; LLM-judge scores are reported to vary by roughly ±0.05 between runs on the same input due to judge non-determinism.

When context relevance scores are low, the usual culprits are upstream: revisit your chunking strategy and your embedding model before tuning the prompt.

Where RAG eval scores lie to you

A high score is not proof of correctness. Independent benchmarking has found that none of the major frameworks can distinguish a factually wrong retrieved context from a correct one — the metrics check whether the answer is consistent with what was retrieved, not whether what was retrieved is true. A pipeline can score 0.95 on faithfulness and still confidently deliver the wrong business answer if the underlying document was stale or incorrect (Atlan — independent benchmark findings).

The judge model matters more than the framework you pick around it: since all three tools defer scoring to an LLM, a poorly calibrated or biased judge produces misleading dashboards regardless of which library computed the number. Treat eval scores as a triage signal that tells you where to look — not a certificate of correctness.

Frequently Asked Questions

What is RAG evaluation?

RAG evaluation is measuring whether a retrieval-augmented generation pipeline retrieves the right context and generates an answer that's grounded in it. It splits into two failure edges: retrieval quality (did we find the right chunks?) and generation faithfulness (did the model use them correctly, without inventing anything?). Standard frameworks like RAGAS and TruLens score both with an LLM-as-judge.

What is the RAG triad?

The RAG triad, coined by TruEra/TruLens, is three evaluations that together catch hallucination: context relevance (are retrieved chunks relevant to the query?), groundedness (is the answer supported by the retrieved context?), and answer relevance (does the answer address the question?). Passing all three gives confidence the app is hallucination-free up to the limits of its knowledge base.

What is faithfulness in RAG evaluation?

Faithfulness measures what fraction of an answer's claims are actually supported by the retrieved context. RAGAS computes it by decomposing the answer into atomic statements with an LLM, checking each against the context, and scoring supported/total. A score below roughly 0.7–0.8 is commonly used as a threshold that flags meaningful hallucination worth investigating.

RAGAS vs TruLens vs DeepEval — which should I use?

They fit different stages. RAGAS is the fastest path to scored, standardized RAG metrics for offline comparison of pipeline configurations. TruLens adds OpenTelemetry-based tracing so you can watch faithfulness and relevance drift on live production traffic, not just an eval set. DeepEval is built pytest-style, so RAG quality becomes a CI gate that can block a merge. Many teams run more than one: RAGAS or TruLens in development, DeepEval in CI, TruLens (or Arize Phoenix) in production.

Does a high faithfulness score mean the RAG system is correct?

No. Faithfulness only checks that the answer's claims trace back to the retrieved context — it can't tell you if that context was itself wrong or stale. A RAG system can score 0.95 on faithfulness and still give a confidently wrong business answer if the underlying documents were outdated or incorrect. RAG evaluation frameworks operate at the inference layer; they don't substitute for keeping the knowledge base accurate.

Do I need labeled ground truth to evaluate a RAG pipeline?

Not for the core generation-side metrics. Faithfulness and answer relevancy are reference-free — they only need the question, retrieved context, and generated answer. Context recall is the exception: it requires a ground-truth answer to check whether retrieval pulled back everything necessary, so it's typically added once you have a labeled eval set.

What is answer correctness, and how is it different from faithfulness?

Faithfulness checks whether the answer is consistent with the retrieved context — internal consistency, not truth. Answer correctness checks the answer against a human-written reference answer, combining a factual F1 score over claims with semantic similarity between the two texts. A response can score high on faithfulness and still be wrong if the retrieved chunk itself was outdated or incorrect; answer correctness is the metric that catches that faithful-but-wrong case, at the cost of needing a labeled gold set.

Should I use Arize Phoenix instead of RAGAS?

They solve different problems, not competing ones. RAGAS is an offline metrics library for scoring a pipeline against a fixed eval set during development. Arize Phoenix is a production observability platform: it traces every span (embedding, retrieval, reranking, generation) on OpenTelemetry and runs LLM-as-judge evaluators against live traffic after the fact, so you can localize a bad answer to the exact step that caused it. Most production RAG stacks run both — RAGAS or TruLens in development, Phoenix (or TruLens) watching sampled production traffic for drift.

Want 1:1 help? Book a session

5.0 · 18 reviews

Career guidance, resume & interview prep, or tech consulting — 1:1 with a Lead AI Engineer. Start with a free 30-min quick chat.

RAG Evaluation Metrics: Faithfulness, Context Precision/Recall, and How to Actually Measure a RAG Pipeline