Retrieval Strategies Beyond Vector Search: BM25, Hybrid Search, and Sparse Retrieval

Introduction

Dense vector search — embedding a query, then finding the nearest neighbours in a vector database — has become the default retrieval strategy for RAG systems. It is easy to set up, works well with modern embedding models, and integrates cleanly with LLM pipelines. But “just use embeddings” is not a retrieval strategy. It is one tool in a toolbox that includes BM25, learned sparse retrieval, hybrid fusion, late interaction models, graph traversal, and cross-modal search.

The choice of retrieval strategy is the highest-leverage architectural decision in any RAG system. A pure dense vector approach returns semantically similar chunks, but it struggles with exact keyword matches, out-of-domain queries, rare terms, and documents where language differs from the query. BM25 handles exact matching robustly but ignores semantics entirely. Hybrid search combines both. Learned sparse retrieval and ColBERT offer different points on the accuracy-latency-cost frontier.

This post surveys the full landscape of retrieval strategies available in 2026, with practical guidance on when to use each one, how to combine them, and how to evaluate retrieval quality. We include production-ready code examples, benchmarks, and links to deeper dives on related topics like chunking, embeddings and vector databases, and RAG architectures.

Sparse Retrieval with BM25

BM25 (Best Matching 25) is a bag-of-words retrieval function that ranks documents by their relevance to a query term-by-term. It builds on TF-IDF with two key improvements: a saturation function that limits the impact of term frequency, and document length normalisation. The formula scores each document as a sum over query terms:

Score(D, Q) = sum over t in Q of IDF(t) * (tf(t, D) * (k1 + 1)) / (tf(t, D) + k1 * (1 - b + b * |D| / avgdl))

The parameter k1 controls term frequency saturation (typical range 1.2–2.0, default 1.5). Higher values give more weight to repeated terms. The parameter b controls length normalisation (0–1, default 0.75). At b=0, document length is ignored; at b=1, it is fully normalised. For code and technical documentation, lowering b to 0.5 often improves results because longer documents are not inherently less relevant.

Advantages over dense retrieval.

• Exact matches: BM25 finds documents containing the exact query terms. If a user searches for “Kubernetes pod eviction timeout”, BM25 finds documents that contain those exact phrases, even if the embedding model has never seen those terms in training.
• Out-of-domain robustness: BM25 requires no training data and no embedding model. It works on any text in any language. It is the only retrieval strategy that generalises zero-shot to entirely new domains without any model update.
• Interpretability: Every BM25 score can be explained by which terms matched and how often. This simplifies debugging and relevance tuning through field boosting – you can weight a title match at 3x a body match.
• Speed: Inverted index lookups are O(1) per term and complete in single-digit milliseconds on commodity hardware, even for millions of documents.

BM25 is not a replacement for dense retrieval. It misses synonyms, paraphrases, and semantically related concepts. But it is an essential component of any robust retrieval pipeline and the foundation for hybrid search. Most search infrastructure — Elasticsearch, Lucene, Meilisearch, and Qdrant — includes BM25 as a built-in retrieval option.

Learned Sparse Retrieval

Learned sparse retrieval bridges the gap between BM25 and dense embeddings by using transformer models to produce sparse, interpretable term weights. Instead of embedding text into a dense 768- or 1024-dimensional vector, models like SPLADE and uniCOIL produce a sparse vector where each dimension corresponds to a vocabulary term and the value represents the term's importance.

SPLADE (Sparse Lexical and Expansion Model).

SPLADE uses a BERT encoder to expand the input text into a weighted set of terms. The model takes a query or document, passes it through BERT, and projects the final hidden states to vocabulary logits. A FLOPS regularisation loss encourages sparsity during training, forcing the model to use as few terms as possible while maintaining retrieval accuracy. The output is a vector with typically 50–300 non-zero entries out of a 30K-term vocabulary — sparse enough to index with an inverted index, dense enough to capture semantic expansion.

SPLADE-v3, released in 2025, achieves 97% of the retrieval accuracy of the best dense models on BEIR while using a standard inverted index for storage and search. This means you get near-dense retrieval quality with BM25 infrastructure — no vector database required. The practical implication is significant for teams that already operate Elasticsearch or Solr: they can upgrade retrieval quality without adding a new storage system.

uniCOIL.

uniCOIL (unified Contextualised Inverted List) takes a similar approach but produces a single weight per vocabulary term regardless of the input length. It converts the BERT output into a single term-weight vector per document. UniCOIL is simpler and faster than SPLADE at the cost of slightly lower accuracy. It is a good choice when latency matters more than recall.

Both SPLADE and uniCOIL combine the semantic understanding of transformers with the efficiency of inverted index retrieval. They excel at capturing synonym expansion and query-document term mismatch without the storage overhead of dense vector databases. For deployment, the splade Python library provides pre-trained models and efficient indexing pipelines.

Hybrid Search

Hybrid search runs a dense (vector) and a sparse (BM25 or SPLADE) retrieval in parallel, then fuses the results into a single ranked list. The dense arm captures semantic similarity — synonyms, paraphrases, conceptual matches. The sparse arm captures exact keyword matches, rare terms, and out-of-domain queries that the embedding model was not trained on. Together, they consistently outperform either method alone.

Reciprocal Rank Fusion (RRF).

RRF is the most widely used fusion strategy. It computes a combined score for each document by summing reciprocal ranks across all result lists. The formula is: RRF score = sum over systems of 1 / (k + rank(system, doc)). The constant k (typically 60) dampens the impact of high ranks from individual systems. RRF is simple, deterministic, and requires no training or calibration.

def reciprocal_rank_fusion(
    results: list[list[tuple[str, float]]],
    k: int = 60,
    top_n: int = 10,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = {}
    for system_results in results:
        for rank, (doc_id, _score) in enumerate(system_results):
            scores[doc_id] = scores.get(doc_id, 0) + 1 / (k + rank)

    ranked = sorted(scores.items(), key=lambda x: -x[1])
    return ranked[:top_n]

# Usage: fuse BM25 + dense vector results
dense_results = vector_db.query(query, top_k=20)
bm25_results = bm25_index.search(query, top_k=20)
fused = reciprocal_rank_fusion([dense_results, bm25_results])

Weighted sum interpolation.

An alternative to RRF is interpolating normalised scores: hybrid score = alpha * dense_score + (1 - alpha) * sparse_score. The alpha parameter controls the blend. Alpha = 0.7 (70% dense, 30% sparse) is a common starting point, but the optimal value depends on your domain. For legal or medical search where exact terminology matters, alpha should be lower (0.4–0.5). For conversational or creative search, alpha should be higher (0.8–0.9). The alpha parameter should be tuned on your validation set.

Production implementation.

Most vector databases now support hybrid search natively, eliminating the need to manually fuse results. Qdrant, Weaviate, Elasticsearch, and Pinecone all offer built-in hybrid search with configurable fusion.

# Qdrant hybrid search configuration
from qdrant_client import QdrantClient
from qdrant_client.models import (
    HybridFusion,
    FusionQuery,
)

client = QdrantClient(url="http://localhost:6333")

results = client.query_points(
    collection_name="syntave_docs",
    prefetch=[
        {"query": dense_vector, "limit": 20},
        {"query": sparse_vector, "limit": 20},
    ],
    query=FusionQuery(fusion=HybridFusion.RRF),
    limit=10,
)

# Alternative: weighted alpha interpolation
# alpha = 0.7 (70% dense, 30% BM25)
# hybrid_score = alpha * norm_dense_score + (1 - alpha) * norm_bm25_score

In our production benchmarks, hybrid search with RRF improves recall@10 by 8–15 percentage points over pure dense search across standard BEIR datasets. The improvement is largest for queries with named entities, technical jargon, or domain-specific abbreviations — exactly the queries that real users ask in enterprise settings.

Reranking

Reranking decouples retrieval efficiency from accuracy through a two-stage pipeline. The first stage (dense, sparse, or hybrid) retrieves a broad set of candidates — typically top 20–100 — using fast approximate methods. The second stage applies a more expensive but more accurate model to reorder these candidates by true relevance.

Cross-encoders versus bi-encoders.

The embedding models used in first-stage retrieval are bi-encoders: they encode the query and each document independently into fixed vectors, then compare them with cosine similarity. This is fast (documents are pre-encoded) but loses query-document interaction information. Cross-encoders, by contrast, concatenate the query and document and process them through a full transformer forward pass. They see the interaction between every query token and every document token, producing a more accurate relevance score at the cost of O(N) transformer forward passes for N candidates.

A cross-encoder reranking top 20 documents adds 100–400ms of latency depending on model size and hardware, but improves nDCG@10 by 10–18 points in standard benchmarks. The latency impact is manageable because only the top candidates are reranked. The retrieval budget framework is: spend 50ms to recall 20 candidates, spend 200ms to rerank 5 into the final context.

Reranker options.

• Cohere Rerank: API-based, strong multilingual support, 256 token limit per query-document pair. Best for teams that want zero infrastructure.
• BGE-Reranker-v2 (BAAI): Open-source, multiple sizes (base, large). Comparable accuracy to Cohere. Self-hostable on CPU or GPU. Our default recommendation for production.
• Jina Reranker: 8192 token context window, ideal for long document reranking. Strong performance on financial and legal domains.
• monoT5: Sequence-to-sequence reranker based on T5. Casts reranking as a classification task: is document A more relevant than document B? Best accuracy on MS MARCO. Requires GPU inference.

ColBERT and Late Interaction

ColBERT (Contextualized Late Interaction over BERT) occupies a unique niche between bi-encoders and cross-encoders. Instead of collapsing the document into a single vector or computing full query-document attention, it stores a bag of token-level embeddings for each document and computes relevance as the sum of maximum similarities between query token embeddings and document token embeddings.

Score(Q, D) = sum over query token i of max over document token j of cosine_sim(E_Q[i], E_D[j])

This late interaction design preserves fine-grained token-level matching while keeping inference fast: documents are pre-encoded into token vectors (stored by the PLAID indexing system), and at query time only the query must be encoded (a single forward pass). The maximum-similarity step is a matrix multiplication that runs efficiently on GPU or with approximate indexing on CPU.

Efficiency-performance trade-off.

ColBERT-v2 achieves 97% of cross-encoder accuracy on BEIR while being 100x faster at query time (because documents are pre-encoded). Storage requirements are higher than dense embeddings — each document stores one vector per token — but PLAID compression reduces this to roughly 48 bytes per token, comparable to dense vector storage at 768-dim float32.

The ColBERT ecosystem has matured significantly since 2024. The colbert-ai library provides a production-ready implementation with PLAID indexing, on-the-fly compression, and integration with Hugging Face models. Several vector databases, including Weaviate and Vespa, support ColBERT late interaction natively. For teams that need cross-encoder accuracy with near-bi-encoder latency, ColBERT is the strongest option.

Graph-Based Retrieval

Vector search treats documents as independent points in embedding space. Graph-based retrieval treats them as nodes in a knowledge graph, where edges represent semantic relationships — entity co-occurrence, citation links, hierarchical structure, or ontological connections. Graph retrieval is not a replacement for vector search but a complement that handles multi-hop reasoning and structured knowledge queries that vector search handles poorly.

GraphRAG patterns.

The Microsoft GraphRAG pattern (2024) extracts entities, relationships, and communities from a document corpus using an LLM, builds a knowledge graph, and answers queries by traversing graph communities. For global sensemaking questions (“What are the main themes in this research area?”), GraphRAG significantly outperforms naive vector search because it reasons over entity clusters rather than individual document chunks.

Production deployment of graph-augmented retrieval involves three stages: (1) entity extraction using an LLM or NER model, (2) entity linking to a canonical knowledge base (Wikipedia, Wikidata, or a custom ontology), and (3) graph traversal to find relevant entities and their associated documents. The retrieved entities are then used to filter or augment the vector search results.

In practice, graph-based retrieval is most valuable in regulated domains — legal, finance, healthcare — where documents reference entities (contracts, clauses, regulations) that have explicit relationships. A query like “What are the data retention requirements under GDPR for EU subsidiaries?” benefits from entity traversal (GDPR → EU subsidiaries → data retention policies) before document retrieval. The combination of vector and graph retrieval is an active research area; see our RAG architectures guide for the latest patterns.

Multi-Modal Retrieval

Modern retrieval systems must handle multiple modalities: text, images, audio, and video. A user searching for “red sports car dashboard” expects results that include both text descriptions and relevant images. Multi-modal retrieval solves this by mapping all modalities into a shared embedding space.

CLIP and joint embedding spaces.

CLIP (Contrastive Language-Image Pre-training) from OpenAI trains a text encoder and an image encoder jointly on 400 million image-text pairs. The result is a shared embedding space where the embedding of the text “a dog playing fetch” is close to the embedding of an image of a dog playing fetch. At retrieval time, you can query with text and retrieve images, or query with an image and retrieve text — cross-modal retrieval in both directions.

OpenCLIP (open-source CLIP replications) and SigLIP (Google, 2024) provide comparable quality with more permissive licenses. SigLIP uses a sigmoid loss function that enables training with larger batch sizes, resulting in better accuracy on fine-grained retrieval tasks. For production, most teams use OpenCLIP-ViT-L/14 or SigLIP-400M as the default vision-language model.

Cross-modal search in practice.

Multi-modal retrieval requires a vector database that supports indexing multiple embedding fields per document. Qdrant and Weaviate support multi-vector indexing natively. Each document stores a text embedding (from an LLM embedding model) and an image embedding (from CLIP). At query time, a text query is encoded with both the text encoder and CLIP; the vector database performs separate searches and fuses the results. For image queries, only the CLIP embedding is used.

We cover multi-modal retrieval in depth in our guide to multimodal AI and embedding and vector database guide.

Evaluating Retrieval Quality

The choice of retrieval strategy is meaningless without systematic evaluation. Retrieve-and-compare is a cycle: choose a strategy, measure it on a labelled dataset, diagnose failures, adjust, and repeat. The following metrics and datasets define the standard evaluation framework.

Offline metrics.

• Recall@K: What fraction of relevant documents appears in the top K results? The most important metric for RAG, because missed relevant documents are permanently excluded from the LLM context. Target: > 90% at K=10.
• MRR (Mean Reciprocal Rank): The average of 1/rank(first relevant). Measures how quickly the first correct answer appears. Important for single-answer Q&A.
• nDCG@K (Normalised Discounted Cumulative Gain): Accounts for graded relevance — highly relevant documents contribute more than somewhat relevant ones. The standard IR metric for web search evaluation.
• Precision@K: What fraction of retrieved results is relevant? Important for token budget management — irrelevant results consume context window capacity.

Standard datasets.

• BEIR (Benchmarking IR): 18 datasets covering diverse domains (bio-medical, finance, legal, news). The standard benchmark for retrieval model comparison.
• MS MARCO: 1M queries from Bing search logs with human-judged relevance. The largest and most widely used passage retrieval dataset.
• LOTTE (Library of Online Texts for Text Evaluation): 6 datasets focused on technical documentation and code search. Most relevant for developer-tool RAG systems.

Comparison table.

Approximate benchmarks on BEIR datasets. Latency measured on single GPU (A10G) for dense and ColBERT, single CPU core for BM25 and SPLADE. Actual numbers vary by model size, hardware, and document length.

For a deeper treatment of RAG evaluation, see our guides on RAG best practices and LLM evaluation metrics.

Conclusion and References

Vector search is a powerful retrieval strategy, but it is not the only one — and it is rarely the best in isolation. The most robust RAG systems combine multiple retrieval strategies through hybrid fusion, reranking, and domain-specific augmentation. BM25 provides exact-match robustness. SPLADE offers semantic expansion with inverted index efficiency. Hybrid search fuses the best of both worlds. ColBERT pushes the accuracy frontier with late interaction. Graph retrieval handles structured multi-hop queries. Multi-modal search extends retrieval beyond text.

The right strategy depends on your domain, latency budget, and infrastructure. Start with dense vector search because it is the easiest to set up. Add BM25 hybrid search for your second iteration — the improvement is immediate and the implementation cost is minimal. Layer on reranking when recall plateaus. Evaluate with BEIR or LOTTE at every change. Only reach for learned sparse retrieval, ColBERT, or graph retrieval when you have data showing that simpler approaches are insufficient.

At Syntave, our unified retrieval API supports all of these strategies behind a single interface, allowing teams to switch between them without rewriting their application. For more on how retrieval fits into the broader RAG pipeline, see our guides on RAG architectures, abstracting the RAG pipeline, and RAG best practices.