Reranking in RAG: Improve Retrieval Accuracy

Contents

Your vector search returns the right answer somewhere in the top 20 results, but your LLM only sees the top 3, and the actual answer got buried at position 14. That's not a hypothetical failure mode — it's one of the most common, most fixable problems in RAG systems, and reranking is the fix that consistently punches above its complexity weight.

I started paying real attention to reranking after noticing my retrieval "felt off" on certain queries despite technically working — the right document was in there, just not near the top. Ever had a RAG bot give a partial or slightly wrong answer even though you know the source material had the full answer? This is usually exactly why.

By the end of this guide, you'll understand what reranking actually does, why it's different from your initial retrieval step, and which reranker fits your latency and budget constraints. IMO, this is one of the cheapest, highest-leverage upgrades you can make to an existing RAG pipeline :)

Reranking in RAG Improve Retrieval Accuracy

Figure 1: Reranking improves RAG retrieval precision by re-scoring candidate documents

Why First-Stage Retrieval Alone Isn't Enough

Here's the core issue: vector search (and even hybrid search) is built for speed at scale, not maximum precision. It encodes your query and documents separately into vectors, then compares them using cosine similarity — fast, but a genuinely lossy approximation of "how relevant is this document to this specific query."

Think of it like a classroom analogy: retrieval is everyone submitting a homework answer; reranking is the teacher actually checking them and picking the best five. The first pass is about casting a wide, efficient net. The second pass is about genuinely judging quality.

Dropping a reranker between your vector store and your LLM commonly delivers an NDCG@10 lift in the 5–15 point range, and 20+ points on lexically hard datasets — for under 200ms of added latency. That's not a marginal tweak. That's the difference between a RAG system that hallucinates and one that reliably cites the right source.

How Cross-Encoders Actually Work

The dominant reranking architecture today is the cross-encoder, and understanding why it's more accurate (and more expensive) than your initial retrieval matters for using it correctly.

A bi-encoder (what powers your initial vector search) encodes the query and each document separately into vectors, then compares them afterward. Fast, but the model never actually sees the query and document together.
A cross-encoder feeds the query and a candidate document into the same transformer together, so every query token can "see" every document token directly. This produces a single, far more precise relevance score.

That joint attention is exactly why cross-encoders are more accurate — and exactly why they're too slow to run against your entire corpus. Scoring 10,000 documents pairwise for a clustering task would take a cross-encoder roughly 65 hours. That's precisely why rerankers only ever operate on a small shortlist, not your full index.

The Two-Stage Pipeline

The pattern that's become standard in production RAG looks like this:

First-stage retrieval (vector search, hybrid search, or both) pulls back a broader candidate set — typically the top 20–100 documents.
Reranking runs a cross-encoder over just that shortlist, producing a much more precise final ordering.
The top 3–10 reranked results get passed to the LLM as context.

Retrieval gets you started. Reranking makes you sharp. The division of labor matters: don't ask your fast first-stage retriever to be perfectly precise, and don't ask your precise reranker to search your entire corpus. Each does what it's actually good at.

The Major Reranking Options

Cohere Rerank

Cohere's rerank-v3.5 is a hosted API offering, with strong multilingual coverage and long-context support.

Priced around $2.00 per 1,000 search units, where one unit covers a query plus up to 100 documents.
Documents over 500 tokens get auto-chunked internally, so you don't need to pre-process for length.
Also available through AWS Bedrock and Oracle's Generative AI service for teams with existing cloud procurement constraints.

For most teams, this is genuinely the fastest path to measurable retrieval improvement. No infrastructure to manage, no model to fine-tune — just an API call inserted into your existing pipeline.

BGE Reranker (Self-Hosted, Open Source)

BAAI publishes the BGE reranker v2 family under Apache 2.0, giving you a genuinely solid self-hosted option.

bge-reranker-v2-m3 — the multilingual workhorse and the default starting point for most self-hosted setups.
bge-reranker-v2-gemma — LLM-based, higher quality on English text, but slower.
bge-reranker-v2-minicpm-layerwise — lets you pick a specific layer to balance latency against quality directly.

Choose this path when data sovereignty matters, or when API costs at scale start outweighing the operational overhead of self-hosting.

Lightweight Cross-Encoders (MS MARCO Family)

For simpler, lower-latency needs, the classic cross-encoder/ms-marco-MiniLM models remain genuinely solid options.

MiniLM-L6-v2 — the best balance of speed and quality; use this as your default for most reranking tasks.
MiniLM-L12-v2 — slightly more accurate but slower; reach for this when ranking quality is worth the extra latency.
Processes roughly 1,800 documents per second, making it fast enough for real-time applications even without a hosted API.

LLM-Based Reranking

Instead of a dedicated scoring model, this approach directly prompts a large language model to rank candidate documents.

Can achieve higher accuracy by leveraging an LLM's broader contextual understanding.
The tradeoff is steep: significantly slower and more expensive than cross-encoders — some benchmarks show LLM rerankers running hundreds of times slower for only marginal accuracy gains over faster alternatives.

Reserve this for situations where result quality genuinely matters more than latency or cost — this isn't your default choice, it's a specialized tool for specific high-stakes retrieval tasks.

Quick Comparison Table

|----------|------|---------|----------|

A Practical Implementation Pattern

Here's roughly how reranking slots into an existing pipeline using a lightweight cross-encoder.

from sentence_transformers import CrossEncoder

reranker = CrossEncoder("cross-encoder/ms-marco-MiniLM-L6-v2")

def rerank(query, candidates, top_n=5):
    pairs = [[query, doc] for doc in candidates]
    scores = reranker.predict(pairs)
    ranked = sorted(zip(candidates, scores), key=lambda x: x[1], reverse=True)
    return [doc for doc, score in ranked[:top_n]]

initial_results = vector_search(query, top_k=25)
final_results = rerank(query, initial_results, top_n=5)

Notice the shape here: retrieve broadly first, rerank narrowly second. That top_k=25 followed by top_n=5 pattern is genuinely the whole architecture in miniature.

Domain-Specific Fine-Tuning Matters More Than You'd Think

Here's something that trips up teams working in specialized fields: most pre-trained cross-encoders are generalists, trained on datasets like MS MARCO — essentially a massive collection of Bing search queries paired with generic web passages.

If your domain is legal contracts, medical records, or security incident reports, that generalist model might not rank your content correctly. It doesn't inherently know that "force majeure" is a specific contract term rather than a military phrase, for instance. Fine-tuning on domain-specific data can meaningfully close this gap.

Don't assume a top-benchmark reranker will automatically perform well on your specialized content. Test it against your actual documents before trusting the leaderboard score blindly.

Common Mistakes People Make

I've seen these repeated across enough implementations to flag them as patterns.

Running a reranker over your entire index instead of a shortlist. Cross-encoders are too slow for full-corpus search — always narrow with fast first-stage retrieval first.
Assuming a generalist reranker works perfectly for specialized domains. Legal, medical, and technical content often needs fine-tuning or domain-aware evaluation before trusting the ranking.
Skipping reranking because "vector search already works." Retrieval technically returning the right document somewhere in the top 20 isn't the same as your LLM actually seeing it in the top 5 it gets fed.
Choosing LLM-based reranking by default. The latency and cost overhead rarely justifies the marginal accuracy gain over a good cross-encoder for most use cases.

So, Which One Should You Actually Use?

Here's the honest, no-fluff breakdown: start with Cohere's rerank API if you want the fastest measurable improvement with zero infrastructure overhead. For most teams, this is genuinely the highest-leverage, lowest-effort change available.

Move to a self-hosted option like BGE-reranker-v2-m3 or MiniLM-L6-v2 once cost at scale or data sovereignty requirements make a hosted API less appealing. Reserve LLM-based reranking for genuinely quality-critical, latency-tolerant scenarios — it's a specialized tool, not a default choice.

Frequently Asked Questions

What is reranking in RAG?

Reranking is a second-stage precision step that re-scores initial retrieval results using a cross-encoder model. It takes the top 20-100 candidates and produces a more accurate final ranking for the LLM.

Why is reranking better than vector search alone?

Vector search encodes query and documents separately, making it fast but approximate. Reranking feeds query and document together into a cross-encoder, producing much more precise relevance scores at the cost of some latency.

Which reranker should I use?

Start with Cohere rerank-v3.5 for fastest improvement with no infrastructure. Use BGE-reranker-v2-m3 for self-hosted needs. Use MiniLM-L6-v2 for real-time low-latency applications.

How much does reranking improve retrieval?

Reranking typically delivers 5-15 point NDCG@10 lift over vector search alone. Adding reranking on top of hybrid fusion can push Recall@5 to 0.816, a 39% improvement over dense-only retrieval.

How does reranking affect latency?

Reranking adds roughly 50-200ms latency depending on the model and shortlist size. This is acceptable for most RAG applications where accuracy matters more than millisecond response times.

Can I fine-tune a reranker for my domain?

Yes, cross-encoders can be fine-tuned on domain-specific data. Legal, medical, and technical domains often benefit from fine-tuning since generalist models may not understand specialized vocabulary.

Wrapping This Up

Reranking exists to fix a real gap: fast first-stage retrieval finds the right document somewhere in a broad candidate pool, but "somewhere" isn't good enough when your LLM only reads the top few results. Cross-encoders solve this by jointly analyzing query and document together, trading some speed for meaningfully better precision.

Remember to keep your reranker scoped to a shortlist, not your full corpus, and don't assume a generalist model automatically understands your domain's specific vocabulary. FYI, if your RAG system currently skips reranking entirely, adding it is genuinely one of the cheapest, most measurable upgrades available — often for under 200ms of added latency :)

Now go test this against your own retrieval setup instead of trusting a single benchmark number — your actual query mix and document domain are the only evaluation that really matters here.