AI series · Part 3: Evals and Accuracy
Golden Evals: The Answer Key Every AI Feature Needs
A golden eval is a fixed set of real questions with expert-approved answers. Re-run it after every change, keep what improves the score and reject what doesn't. Here's how to build one.
A golden eval is a fixed set of real questions with expert-approved correct answers. It is the answer key of an exam. After every change (new model, new chunking, new prompt), the system takes the same exam again. Score goes up: keep the change. Score goes down: reject it.
How to build one
- 1Collect real user questions from chat logs or support tickets.
- 2Have experts (for example, HR) write the correct answer and mark the source document.
- 3Add tricky ones: different wording ("dad"), exact codes ("POL-TRV-07"), and questions that have no answer in the documents.
- 4Re-run it automatically on every change.
What it looks like in practice
A bank's chatbot team keeps 300 golden questions. A developer changes the chunk size, and accuracy drops from 92% to 85%. The change is rejected before a single customer sees it.
Our reference HR assistant ships with 25 golden questions. Each line records the question, the expected answer and the source it should come from.
I build a golden dataset from real user questions with expert-verified answers and sources, and run it on every change to catch regressions.
- The short version
Next in the GenAI series: Measuring AI Accuracy: Retrieval Metrics vs Answer Metrics
Start from the top: GenAI Quick Answers and Series Guide
We deploy these patterns across production engagements in fintech, healthcare, and eCommerce. Talk to a senior engineer about how they apply to your platform.