Retrieval-Augmented Generation in Production: Building RAG Systems That Return Accurate Results

The naive RAG implementation is deceptively simple: embed a collection of documents, store the embeddings in a vector database, embed the user's query, retrieve the top-k most similar chunks, and pass them to the LLM as context. In development, with clean test data and well-formulated queries, this works well. In production, with real user queries, messy documents, and edge cases, it fails in predictable but non-obvious ways that are difficult to debug without a systematic approach.

The most common failure mode is retrieval of technically similar but semantically irrelevant chunks. Vector similarity measures cosine distance between embedding vectors — it finds chunks that use similar vocabulary, not chunks that answer the specific question. A query about 'how to cancel a subscription' retrieves chunks about 'subscription management' even when those chunks don't contain cancellation information, because the vocabulary overlap is high. The retrieved context confuses the LLM, producing hallucinated or hedged answers.

The second common failure is context fragmentation. When a document is chunked at fixed size boundaries, a chunk often ends mid-sentence or mid-concept. The chunk retrieved for a user's question may contain the beginning of the answer but not the conclusion — or the conclusion without the prerequisite context. The LLM then either hallucinates the missing information or hedges with 'I don't have enough information,' even when the answer exists in an adjacent chunk that wasn't retrieved.

Query-document mismatch is the third structural problem. Documents are written as statements: 'the billing cycle resets on the first of each month.' User queries are phrased as questions: 'when does my billing reset?' The embedding space does not perfectly align question vectors with answer vectors for the same information. This mismatch causes high-relevance answers to rank below medium-relevance content that uses more similar phrasing to the query.

Chunking is the process of splitting source documents into the segments that will be embedded and stored for retrieval. The chunking strategy is one of the most impactful architectural decisions in a RAG system, and it's frequently underestimated. Chunks that are too large include so much irrelevant information that the embedding doesn't capture the specific meaning well. Chunks that are too small lose the context needed to answer questions accurately.

Fixed-size character splitting with overlap is the baseline chunking approach — split every N characters with an M-character overlap between adjacent chunks. This is simple to implement but semantically blind: it splits paragraphs, sentences, and concepts mid-way. The overlap reduces context fragmentation at chunk boundaries, but it doesn't fix the fundamental problem that splits are semantically arbitrary. Fixed-size chunking is appropriate only for uniform, structured documents where the content within any given N characters is self-contained.

Semantic chunking splits documents at natural boundaries — paragraph breaks, section headings, sentence endings — and merges small chunks until they reach a target size range. This produces chunks that are semantically coherent: each chunk represents a complete idea rather than an arbitrary slice of text. In LangChain, RecursiveCharacterTextSplitter with a separator hierarchy (['\n\n', '\n', '. ', ' ']) approximates semantic chunking by trying to split at paragraph breaks first, then sentence breaks, then word breaks. For structured documents with explicit sections, use the section structure as the primary chunking boundary.

Context-enriched chunks improve retrieval recall by embedding additional contextual information that helps the retrieval system find the right chunks even when the user's query doesn't match the chunk's literal vocabulary. Two effective techniques: document-level metadata injection (adding the document title, category, and date to every chunk's text before embedding) and hypothetical document embeddings (generating a synthetic question for each chunk and embedding both the chunk and the question). These techniques increase retrieval accuracy at the cost of storage and embedding computation.

The embedding model transforms text into a high-dimensional vector that represents its semantic meaning. The choice of embedding model determines the quality ceiling of your retrieval system — a weaker embedding model produces vectors that don't capture semantic nuance well, and no amount of retrieval or reranking optimization can compensate for fundamentally poor embeddings. The question is not which model is cheapest but which model's embedding space best captures the semantic relationships in your specific domain.

OpenAI's text-embedding-3-small and text-embedding-3-large are the most widely deployed embedding models, offering strong performance across a wide range of domains and languages. For domain-specific applications — medical, legal, financial — fine-tuned embedding models trained on domain data often outperform general-purpose models. MTEB (Massive Text Embedding Benchmark) provides a standardized evaluation of embedding model quality across multiple retrieval tasks and is the starting point for comparing models before making a commitment.

Supabase with the pgvector extension is the most developer-friendly production vector storage solution for web applications already using PostgreSQL. Storing embeddings in the same database as the application's relational data enables joins that are not possible with standalone vector databases — retrieve documents with their embeddings and filter by relational conditions (user ownership, access permissions, category membership) in a single query. The pgvector HNSW index achieves approximate nearest neighbor search performance within 5-10% of standalone vector databases at moderate scale.

Index configuration significantly impacts retrieval performance at scale. The HNSW index parameters — ef_construction (build-time quality vs. speed tradeoff) and m (connectivity between nodes) — should be tuned based on dataset size and query latency requirements. For most production applications with millions of documents, ef_construction=64 and m=16 provide a good balance. Monitor index size growth and query latency as the dataset grows — you may need to increase m for recall-critical applications as the dataset expands.

Pure vector search has a consistent weakness: it struggles with exact-match queries — proper nouns, product names, error codes, specific version numbers, and technical identifiers. When a user asks about 'error code 42P01' or 'LangChain version 0.2 breaking changes,' the vector similarity finds semantically related content but misses the exact match. Hybrid search combines vector similarity with keyword search to capture both semantic similarity and lexical precision.

The hybrid search architecture combines two retrieval pipelines: a vector search pipeline (semantic similarity over embeddings) and a keyword search pipeline (BM25 or full-text search over raw text). Each pipeline returns a ranked list of results, and a fusion algorithm combines the two rankings into a single list. Reciprocal Rank Fusion (RRF) is the standard fusion algorithm — it combines rankings without requiring score normalization and produces good results across a wide range of query types.

In Supabase with pgvector, hybrid search can be implemented with a single SQL function that computes both the vector similarity score and the full-text search score, then combines them using a weighted sum or RRF. The full-text search leverages PostgreSQL's built-in tsvector and ts_rank, which supports stemming, stop words, and phrase matching. The relative weight of vector versus keyword components should be tuned on your evaluation dataset — queries with technical identifiers benefit from higher keyword weight; conversational queries benefit from higher semantic weight.

Query expansion improves retrieval recall for ambiguous or short queries. Generate multiple versions of the user's query — semantic paraphrases, entity expansions, and hypothetical document forms — and retrieve results for each version. Merge the results using RRF and deduplicate. This technique significantly improves recall for queries that don't match the vocabulary of the target documents, at the cost of additional embedding computations and slightly higher latency.

The top-k results from hybrid search are candidates, not necessarily the best answers. Reranking applies a more powerful model to the retrieved candidates to produce a final ordering based on relevance to the specific query. The retrieval stage optimizes for recall — finding documents that might be relevant. The reranking stage optimizes for precision — identifying which of the retrieved documents are actually most relevant to answer the query.

Cross-encoder rerankers (Cohere Rerank, Jina Reranker, BAAI/bge-reranker) evaluate query-document pairs directly rather than comparing separate embeddings. This allows the model to reason about the relationship between the query and each candidate document in full context, producing significantly more accurate relevance rankings than embedding similarity alone. The tradeoff is latency: reranking 20 candidate documents requires 20 model evaluations, whereas retrieval is a single approximate nearest neighbor search.

Query decomposition breaks complex, multi-part questions into simpler sub-queries that can be answered independently. A question like 'what are the differences between LangChain's agent and chain patterns and when should I use each?' is better handled as two sub-queries. Each sub-query retrieves focused context, and the answers are synthesized by the LLM into a comprehensive response. LangChain's MultiQueryRetriever implements this pattern automatically.

Contextual compression post-retrieval removes irrelevant sentences from retrieved chunks before passing them to the LLM. Even after reranking, a retrieved chunk may contain significant irrelevant content that dilutes the signal for the LLM. LangChain's ContextualCompressionRetriever applies an LLM or embedding filter to each retrieved chunk, extracting only the sentences relevant to the query. This reduces context window usage and improves answer quality by increasing the signal-to-noise ratio of the context.

A RAG system without a systematic evaluation framework is impossible to improve reliably. Changes to chunking, embedding models, retrieval configuration, or prompts can improve performance on some queries while degrading it on others. Without measurement, you cannot distinguish improvements from regressions. Building an evaluation framework before optimizing is the most important architectural decision in a production RAG system.

The RAG evaluation framework consists of three components: a question set representative of real user queries, a set of ground truth answers or reference documents for each question, and metrics for measuring retrieval quality (retrieval recall, MRR) and answer quality (faithfulness, answer relevance, correctness). Tools like RAGAS (RAG Assessment) provide automated metrics that use LLMs to evaluate faithfulness and relevance without requiring manual human evaluation of every response.

Faithfulness measures whether the generated answer is supported by the retrieved context — it detects hallucination. Answer relevance measures whether the answer actually addresses the question asked. Context relevance measures whether the retrieved chunks contain information relevant to the question. These three metrics together give a complete picture of where the RAG pipeline is failing: low context relevance indicates a retrieval problem, low faithfulness indicates a generation problem, and low answer relevance indicates a prompt or decomposition problem.

Production monitoring for RAG extends beyond aggregate metrics. Log every query, the retrieved chunks, and the generated answer for a sample of production traffic. Set up a human review queue for low-confidence answers. Use this reviewed data to expand the evaluation dataset continuously. The evaluation set should grow with the production system — new query types, new document categories, and edge cases discovered in production become new evaluation examples.