Best RAG Practices: Building Reliable Retrieval-Augmented Generation Systems

Introduction

Retrieval-Augmented Generation (RAG) has become the standard architecture for grounding LLM outputs in external knowledge. Every major AI application in 2026 — customer support bots, internal search, code assistants, compliance tools — uses some form of RAG. The idea is simple: retrieve relevant documents from a knowledge base at query time and inject them into the model's context as evidence for generation.

But simplicity in theory masks complexity in practice. A production RAG system involves document ingestion pipelines, embedding models, vector databases, re-ranking, query transformation, context window management, and evaluation. Each component has its own failure modes, and the system is only as strong as its weakest link. A broken chunking strategy can make your best embedding model useless. A poorly tuned re-ranker can degrade latency by 500ms while actually lowering accuracy.

This post collects the engineering practices we have developed at Syntave across dozens of production RAG deployments. We cover every stage of the pipeline, from document ingestion through evaluation, with concrete recommendations, code examples, and the rationale behind each choice.

The RAG Pipeline: Indexing, Retrieval, Generation

Every RAG system follows the same three-stage architecture, originally formalised by Lewis et al. in 2020:

1. Indexing: Parse, clean, chunk, and embed source documents into a vector database.
2. Retrieval: At query time, embed the user's question and search the vector database for the most semantically similar chunks.
3. Generation: Construct a prompt containing the retrieved chunks and the user's question, then feed it to an LLM to produce a grounded answer.

This sounds straightforward, but each stage conceals a dozen design decisions that determine whether the system works in production or fails at 2 AM.

from typing import List, Dict
import chromadb
from openai import OpenAI

class RAGPipeline:
    def __init__(self, collection_name: str):
        self.client = chromadb.PersistentClient()
        self.collection = self.client.get_or_create_collection(collection_name)
        self.llm = OpenAI()

    def ingest(self, documents: List[str], ids: List[str]):
        self.collection.add(documents=documents, ids=ids)

    def query(self, question: str, top_k: int = 3) -> str:
        results = self.collection.query(query_texts=[question], n_results=top_k)
        context = "\n---\n".join(results["documents"][0])
        prompt = f"Answer using this context:\n{context}\n\nQ: {question}"
        response = self.llm.chat.completions.create(
            model="gpt-4o",
            messages=[{"role": "user", "content": prompt}]
        )
        return response.choices[0].message.content

The code above shows a minimal but complete RAG pipeline. It ingests documents, queries by semantic similarity, and generates an answer. In practice, every line hides complexity: how are documents chunked? What embedding model is used? What if the retrieved documents are irrelevant? What if the context exceeds the model's window? The following sections address each of these questions.

Document Ingestion: Parsing, Cleaning, Chunking

Ingestion is the most underestimated phase of RAG. Every error introduced here propagates through the entire pipeline. A PDF with mis-parsed tables, an HTML page with boilerplate extracted as content, a code file with comments separated from their functions — these are not edge cases. They are the norm.

The ingestion pipeline should handle at least three formats: plain text, Markdown, and HTML. For PDFs, use a proper document parser (like Amazon Textract, Azure Document Intelligence, or unstructured.io) rather than naive text extraction. For code, respect the file structure — each function or class is a natural semantic unit.

Cleaning

Raw documents contain noise that degrades embedding quality: HTML tags, navigation bars, headers and footers, repeated boilerplate, special characters, and inconsistent whitespace. Build a cleaning pipeline that strips these elements before chunking. For Markdown, preserve the heading structure — it is valuable semantic signal for chunking boundaries.

Chunking Strategies

Chunking is the single most impactful retrieval quality lever, yet it is the most commonly neglected. The goal is to produce chunks that are semantically self-contained: each chunk should contain enough context to answer a question without needing surrounding chunks.

The three main strategies, in order of sophistication:

• Fixed-size chunking: Split by character or token count with overlap. Simple, fast, and usually wrong. A boundary may fall mid-sentence or mid-paragraph, producing chunks that start or end in semantic no-man's-land.
• Recursive chunking: Split on natural boundaries — first at paragraph breaks, then at sentence boundaries, then at character limits. LangChain's RecursiveCharacterTextSplitter is the canonical implementation. Better than fixed-size, but still format-aware, not semantics-aware.
• Semantic chunking: Use an embedding model to detect natural topic boundaries. Compute embeddings for overlapping sentence windows, detect sharp changes in the embedding trajectory (using cosine distance), and split at those points. This is the state of the art and what we recommend for production.

For an exhaustive treatment, see our dedicated post on chunking strategies for RAG.

Embedding Models: Choosing the Right One

The embedding model determines what “similar” means in your RAG system. A poor embedding model — or one poorly matched to your domain — will retrieve irrelevant documents even with perfect chunking.

As of 2026, the embedding model landscape has converged on a few top contenders:

• text-embedding-3-small/large (OpenAI): The default choice for general-purpose English RAG. 1,536 dimensions (small) or 3,072 (large). Strong retrieval accuracy, good multilingual support, low latency via the API. The small model is sufficient for most use cases.
• BGE (BAAI): Open-source embeddings that rival OpenAI on the MTEB benchmark. BGE-M3 supports multiple languages and multiple granularities (dense, sparse, multi-vector). Excellent for self-hosted deployments where you control the infrastructure.
• E5 and E5-mistral (Microsoft): Instruction-tuned embeddings that let you prefix queries with a task description. Especially strong for retrieval tasks where the query and document distributions differ (e.g., short query, long document).
• Voyage AI: Domain-specific embedding models for code, finance, legal, and medical. If your RAG system operates in a specialised domain, Voyage's fine-tuned models can outperform general-purpose embeddings by 5-15 points on recall.
• Cohere Embed v3: Strong multilingual support with 1,024 dimensions. Good integration with Cohere's re-ranking API, making it a natural choice if you already use Cohere for re-ranking.

The best embedding model for your use case depends on your language, domain, and infrastructure constraints. Evaluate at least three candidates on a representative set of queries before committing. MTEB scores are useful baselines, but they do not predict performance on your specific data.

Vector Databases: Pinecone, Weaviate, Qdrant, Milvus, pgvector

The vector database stores the embeddings and performs approximate nearest neighbour (ANN) search at query time. The choice of vector DB is one of the most durable infrastructure decisions in a RAG system, because migrating between databases is costly.

Here is our assessment of the major options as of 2026:

• Pinecone: The easiest to get started with. Fully managed, no ops overhead, excellent scaling to billions of vectors. Serverless mode eliminated minimum spend for small deployments. The primary trade-off is cost at scale — Pinecone is the most expensive option per vector stored.
• Qdrant: The best balance of performance and developer experience. Strong filtering support (crucial for multi-tenant RAG), binary quantization for 32x memory reduction, and a well-designed gRPC API. Qdrant Cloud is competitive on price. Our default recommendation for most production deployments.
• Weaviate: Strong multi-modal support (text, images, objects) and built-in generative search modules. Weaviate handles more than vector search — it offers hybrid search, re-ranking, and LLM integration out of the box. Good choice if you want an all-in-one platform.
• Milvus (Zilliz Cloud): The high-performance option. Milvus handles billions of vectors with sub-10ms latency. Complex to self-host (requires Kubernetes and a deep understanding of the architecture), but Zilliz Cloud mitigates this. Best for very large-scale deployments.
• pgvector (PostgreSQL): The “good enough” option for small deployments. If you already use PostgreSQL, adding pgvector avoids infrastructure sprawl. Works well for up to 10 million vectors with HNSW indexing. Beyond that, dedicated vector DBs start to pull ahead on latency and recall.

At Syntave, we abstract vector DB selection behind a unified interface so you can start with one provider and switch without rewriting your application. See our post on abstracting the RAG pipeline for the architecture.

Retrieval Strategies: Semantic Search, Hybrid Search, Re-Ranking

Retrieval is not a single operation. It is a pipeline of increasingly refined filtering steps.

Semantic Search

The simplest retrieval strategy: embed the user's query, find the top-K nearest neighbours in the vector database by cosine similarity. This works well when the query is a natural language question and the documents use similar language to describe answers. It fails when the query uses different terminology than the documents — for example, a user asks “how do I cancel my subscription?” but the documentation uses the phrase “terminate your account.”

Hybrid Search (Dense + Sparse)

Hybrid search combines dense embeddings (semantic similarity) with sparse retrieval (BM25 keyword matching). The results are fused using weighted reciprocal rank fusion (RRF) or a learned weighting. Hybrid search consistently outperforms either method alone, particularly on queries that contain rare or domain-specific terms that the embedding model may not capture.

Most vector databases now support hybrid search natively. Qdrant, Weaviate, and Elasticsearch all offer built-in hybrid search with configurable alpha weighting. Our default recommendation: start with alpha = 0.7 (70% semantic, 30% keyword) and tune based on your evaluation set.

Re-Ranking

The initial retrieval step (ANN search) is fast but imprecise. It returns the top-K most similar chunks by embedding similarity, which is a good proxy but not a perfect measure of relevance. Re-ranking applies a more expensive but more accurate model to the top K results, re-scoring them by their actual relevance to the query.

The standard re-ranker is a cross-encoder: a model that takes a query-chunk pair as input and outputs a relevance score. Unlike bi-encoders (embedding models), cross-encoders process the query and chunk together, allowing them to capture interactions between the two texts. Cohere Rerank and BGE-Reranker-v2 are the most popular options, with BGE being competitive on accuracy while being open-source and self-hostable.

A common pattern: retrieve top 20 with ANN, re-rank to top 5 with a cross-encoder. This gives the recall of a broader search with the precision of a more expensive relevance model.

Query Transformation: Rewriting, HyDE, Multi-Query

Users rarely write ideal retrieval queries. They ask ambiguous, underspecified, or poorly-phrased questions. Query transformation techniques rewrite the user's query into one or more forms that are better suited for retrieval.

Query Rewriting

Use a small LLM to rewrite the user's query into a retrieval-optimised form. For example, “how do I connect?” becomes “step-by-step guide to connecting to the Syntave API.” The rewrite expands abbreviations, resolves pronouns, and adds context. This simple transformation can improve retrieval recall by 10-20% in our benchmarks.

HyDE (Hypothetical Document Embeddings)

HyDE takes query transformation a step further: instead of rewriting the query, it asks an LLM to generate a hypothetical document that would answer the query. It then embeds that hypothetical document and uses it for retrieval. The intuition: the embedding of a “good answer” is closer to real answer documents in embedding space than the embedding of the query itself — especially for queries that are short or ambiguous.

def hyde_query(query: str, llm) -> str:
    """Generate a hypothetical document from the query,
    then use that document (not the query) for retrieval."""
    prompt = f"Write a paragraph that would answer: {query}"
    hypothetical_doc = llm.generate(prompt)
    # Embed the hypothetical doc, not the original query
    embedding = embed_model.encode(hypothetical_doc)
    return vector_db.search(embedding, top_k=5)

Multi-Query

Generate multiple reformulations of the user's query, retrieve results for each, and merge the results. This covers different phrasings and perspectives, improving recall for complex or multifaceted questions. The merged results can be de-duplicated and re-ranked before generation.

Context Window Management: Stuffing, Filtering, Compression

Once you have retrieved documents, you must fit them into the LLM's context window — without exceeding token limits and without including irrelevant content that dilutes the signal.

Stuffing

The simplest approach: concatenate all retrieved chunks into the prompt. This works when retrieved chunks are few (<5) and the total fits comfortably within the context window. At Syntave, we use this as the default for queries with top-K ≤ 3 and chunks under 512 tokens each.

Filtering

Not every retrieved chunk is relevant. Apply a relevance threshold — either the embedding similarity score or the re-ranker score — and discard chunks below the threshold. This prevents the LLM from being distracted by irrelevant content. The threshold must be tuned per dataset: too aggressive and you lose relevant context; too permissive and you introduce noise.

Compression

LLMLingua (Jiang et al., 2023) and similar methods compress the retrieved context by removing tokens that are predicted to be unimportant for the generation task. The compression model runs in a single forward pass and achieves 2-5x compression with minimal loss in answer quality. We have found LLMLingua particularly useful for long-context RAG where multiple chunks must be included but the total exceeds 32K tokens.

Evaluation: RAGAS, Faithfulness, Answer Relevance

You cannot improve what you do not measure. RAG evaluation must happen at two levels: component-level (does the retriever find the right documents?) and system-level (does the final answer answer the user correctly?).

RAGAS Framework

RAGAS (Retrieval Augmented Generation Assessment) provides a standardised set of metrics for evaluating RAG systems. The four core metrics are:

• Faithfulness: Is every claim in the answer supported by the retrieved context? Measures hallucinations. Target: > 0.9.
• Answer Relevancy: Is the answer relevant to the question? Measures off-topic responses. Target: > 0.85.
• Context Precision: What fraction of retrieved chunks is actually relevant? Measures retrieval quality. Target: > 0.8.
• Context Recall: What fraction of relevant chunks was retrieved? Measures retrieval completeness. Target: > 0.8.

from ragas import evaluate
from ragas.metrics import (
    faithfulness,
    answer_relevancy,
    context_precision,
    context_recall,
)

results = evaluate(
    dataset=eval_dataset,
    metrics=[
        faithfulness,
        answer_relevancy,
        context_precision,
        context_recall,
    ]
)
print(results)  # {"faithfulness": 0.89, "answer_relevancy": 0.92, ...}

Beyond RAGAS, we recommend maintaining a curated evaluation set of 200-500 real user queries with human-annotated ideal answers and relevant document IDs. This is expensive to create but invaluable for regression testing. Every time you change your chunking strategy, embedding model, or re-ranker, run the full evaluation set and compare against the baseline.

Advanced RAG Patterns: Self-RAG, Corrective RAG, Adaptive RAG

Once your basic RAG pipeline is solid, you can layer on advanced patterns that handle specific failure modes.

Self-RAG (Asai et al., 2023)

Self-RAG trains the model to generate special reflection tokens that indicate: (1) whether retrieval is needed, (2) whether the retrieved passages are relevant, and (3) whether the generated response is supported by the passages. This gives the model fine-grained control over its own retrieval and generation behaviour. Self-RAG has been shown to reduce hallucinations by 30-50% compared to standard RAG pipelines.

Corrective RAG (CRAG)

CRAG (Yan et al., 2024) adds a retrieval evaluator that scores the relevance of retrieved documents. If the score is low, the system triggers a corrective step: it attempts a web search or a different retrieval strategy before generating. This prevents the model from generating answers based on irrelevant or missing context.

Adaptive RAG

Adaptive RAG (Shao et al., 2024) routes queries dynamically: simple factual queries go directly to the LLM (no retrieval), moderately complex queries use standard RAG, and complex multi-hop queries use iterative retrieval with reasoning. A lightweight classifier determines which route each query takes, reducing latency and cost for easy questions while maintaining accuracy for hard ones.

For a detailed comparison of these and other RAG variants, see our post on types of RAG: a comprehensive taxonomy.

Production Optimisation: Latency, Caching, Monitoring

A RAG pipeline that works in a notebook may take 5-10 seconds per query in production. Optimising for latency without sacrificing quality requires a systematic approach.

Caching

The highest-impact optimisation is caching at multiple levels. Cache the embedding of common queries (avoid re-embedding), cache the vector DB results for identical queries (avoid re-retrieval), and cache the LLM generation for identical query-context pairs (avoid re-generation). In our production deployments, a three-level cache hits 40-60% of repeated queries, reducing end-to-end latency from 1.5s to under 100ms for cached results.

Latency Budgeting

Break your end-to-end latency target into component budgets. If your target is 1 second, typical budgets might be: embedding (100ms), vector search (50ms), re-ranking (200ms), LLM generation (600ms), and overhead (50ms). Measure the actual distribution in production — you will find surprises. We have seen re-ranking take 800ms on CPU while the LLM finishes in 300ms.

Monitoring

Every RAG query should produce a trace with: latency per component, token usage, retrieved chunk IDs, re-ranker scores, and the final answer. Aggregate these into dashboards for retrieval recall (are users finding what they need?), latency percentiles (are p95 times acceptable?), and error rates (are any components failing?). At Syntave, this observability is built into every API call — see our documentation for details.

Our Approach at Syntave: Unified Retrieval API

The practices in this post are not theoretical for us. They are the engineering decisions we encode into the Syntave platform. Our unified retrieval API abstracts away the complexity of chunking strategy, embedding provider, vector database, and re-ranker into a single call that returns grounded answers with citations.

Rather than forcing developers to choose between Qdrant and Pinecone, or experiment with five different chunking strategies, we provide sensible defaults backed by production testing. When your requirements outgrow the defaults, you can override individual components without changing your application code.

This philosophy — maximum flexibility with minimum complexity — is the subject of our post on abstracting the RAG pipeline. If you are evaluating RAG versus fine-tuning for your use case, see our decision framework.