AI series · Part 2: Retrieval (RAG)
Re-ranking: The Second Opinion That Picks the Best Chunks
A re-ranker reads the question and each candidate together and scores how relevant it really is. It's slow but accurate, so you use it on a shortlist. Plus a real lesson on where re-rankers fail.
A re-ranker is a second, more careful model. It reads the question and each candidate chunk together and scores how relevant the chunk really is.
The hiring analogy
- 1HR quickly scans 500 CVs and shortlists 20. That's vector search: fast, but rough.
- 2The manager interviews those 20 and picks the best 3. That's the re-ranker: slow, but accurate.
Google works the same way: fetch thousands of pages quickly, then rank the top ones carefully. Common tools are Cohere Rerank and bge-reranker (a cross-encoder).
A real lesson: re-rankers aren't magic
In our HR assistant, a user asked about leave for a new dad. The re-ranker didn't connect "dad" with "paternity", and it pushed the correct chunk from #1 down to #4. It made the result worse.
The fix was simple: a small synonym list that rewrites "dad" → "paternity leave" before searching. We only found the problem because we measured every change against a fixed question set.
I retrieve wide with fast search, then a cross-encoder re-ranker picks the best few. Re-rankers aren't magic, so I always measure them.
- The short version
Next in the GenAI series: Golden Evals: The Answer Key Every AI Feature Needs
Start from the top: GenAI Quick Answers and Series Guide
We deploy these patterns across production engagements in fintech, healthcare, and eCommerce. Talk to a senior engineer about how they apply to your platform.