AI series · Part 3: Evals and Accuracy
Measuring AI Accuracy: Retrieval Metrics vs Answer Metrics
Measure retrieval and answers separately, because the fix depends on which one failed. A plain guide to Recall@K, MRR, faithfulness, correctness and LLM-as-judge.
Measure two things separately:
- 1Retrieval: did we find the right information?
- 2Answer: did the LLM answer correctly, using only that information?
It's like checking why a student got a question wrong. Did they open the wrong page (a retrieval problem), or misread the right page (an answer problem)? The fix is completely different.
The four metrics
| Metric | Simple meaning | Example |
|---|---|---|
| Recall@K | How often the right chunk is in the top K | Found in 24 of 25 → 96% |
| MRR | How high it is ranked (1st = 1, 2nd = 0.5, 4th = 0.25), averaged | Ranks 1, 2, 4 → 0.58 |
| Faithfulness | Everything in the answer comes from the documents; nothing is made up | |
| Correctness | The answer matches the expert's answer |
How to score answers
- LLM-as-judge: a strong model marks each answer against the expert answer, like a teacher.
- Human review of a sample, to check the judge is fair.
- User feedback: track thumbs up and thumbs down in production.
Common tools are RAGAS, DeepEval and LangSmith. In our reference build, one flag turns on Claude as the judge:
python -m evals.run_evals --judgeI measure retrieval with Recall and MRR, and answers with faithfulness and correctness. If the right chunk was found but the answer is wrong, I fix the prompt or model. If the chunk wasn't found, I fix retrieval.
- The short version
Next in the GenAI series: Which Model for Which Job: Picking LLMs by Task, Cost and Evals
Start from the top: GenAI Quick Answers and Series Guide
We deploy these patterns across production engagements in fintech, healthcare, and eCommerce. Talk to a senior engineer about how they apply to your platform.