AI series · Part 5: The Top-K Problem
Top-5 Misses, Top-20 Bloats: Search Wide, Then Pick Narrow
Too few chunks and the right one is missing. Too many and the LLM drowns in noise. A six-step retrieval pipeline that matched top-20 accuracy with about 7× fewer tokens.
Every RAG system hits this wall:
- Top-5 misses the right chunk. Vector search is rough, so the right chunk may come back 8th, and top-5 never sees it.
- Top-20 causes context bloat. The right chunk is included, but with 19 useless ones. That means higher cost, slower answers and a confused LLM.
LLMs suffer from "lost in the middle": they pay less attention to the middle of a long context, just as people skim the middle of a long email. More context isn't automatically better.
The librarian analogy
You ask a librarian for "maternity rules for interns".
- Top-5: you get 5 general maternity books, and the intern rule is missing.
- Top-20: you get 20 books and drown in them.
- The smart way: the librarian collects 20, checks them, and hands you the best 3.
The fix: search wide, then pick narrow
| Step | What it does |
|---|---|
| 0. Rewrite the question | "dad" → "paternity leave", "PF" → "provident fund" |
| 1. Filter by category | Search only the Leave and Intern documents (taxonomy) |
| 2. Hybrid search, wide | Keyword + vector search, top 20, then merge |
| 3. Re-rank | A careful model keeps only the best 3 |
| 4. Compress | Keep only the relevant sentences |
| 5. Knowledge graph | For relationship questions |
| 6. Measure | Prove it with golden evals |
The results
Measured on the 25 golden questions of our reference HR assistant:
| Method | Found right chunk | Ranked #1 (MRR) | Tokens to LLM |
|---|---|---|---|
| Simple top-3 | 96% | 0.92 | 140 |
| Simple top-20 | 100% | 0.93 | 912 |
| Smart pipeline → top-3 | 100% | 1.00 | 137 |
I'd separate finding from choosing. First I search wide: rewrite the question, filter by category, and use hybrid keyword plus vector search to collect 20 to 50 candidates, so the right chunk is almost always there. Then I choose narrow: a re-ranker keeps only the best 3 to 5, and I compress them. The LLM gets a small, highly relevant context, so there's no missing context and no bloat. I prove it with golden evals, comparing Recall, MRR and faithfulness before and after.
- The short version
Start from the top: GenAI Quick Answers and Series Guide
We deploy these patterns across production engagements in fintech, healthcare, and eCommerce. Talk to a senior engineer about how they apply to your platform.