Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
How much chunk overlap should be used?
There is no single chunk overlap value that is best for every RAG system. The right amount depends on chunk size, document structure, how often important ideas cross chunk boundaries, and how much duplication the system can tolerate. Start with the purpose of overlap Overlap exists to protect information near chunk boundaries. If a sentence, definition, or explanation is split between two chunks, a small repeated region can keep enough context in both chunks for retrieval. That means overlap should be large enough to preserve boundary context, but not so large that neighboring chunks become nearly identical. Think about overlap relative to chunk size Overlap is usually easier to reason about as a fraction of the chunk size rather than as an isolated number. For example: This means about 10% of each chunk is repeated into the next chunk. Another configuration might be: Again, the overlap is about 10% of the chunk size. These are examples, not universal rules. A practical starting range For many text based RAG pipelines, a modest overlap such as roughly 10% to 20% of the chunk size can be a reasonable starting point for experimentation. For example: 500 token chunks → try around 50–100 tokens of overlap. 1,000 token chunks → try around 100–200 tokens of overlap. The final value should be chosen through retrieval evaluation rather than by percentage alone. Use less overlap when document boundaries are already meaningful If chunks are created using: Complete paragraphs Section headings Semantic boundaries Well formed sentences then less overlap may be necessary because the splitter is already trying to keep related content together. In some structured documents, zero overlap may work well. Use more overlap when boundaries are artificial More overlap may be useful when chunks are created using fixed token or character limits and important sentences frequently cross those boundaries. For example: Too little overlap If overlap is too small: Important sentences may still be split. Definitions and examples may become separated. Retrieval may return incomplete evidence. Boundary information may be harder to match. Too much overlap If overlap is too large: Many chunks become very similar. The vector index stores repeated information. Embedding cost increases. Retrieval may return several near duplicate chunks. Repeated text can waste the generator's context window. At the extreme, if a 500 token chunk overlaps by 400 tokens, neighboring chunks are mostly duplicates. Tune overlap using retrieval evaluation A good workflow is: Useful signals include: Whether relevant passages appear in Top K results Whether answers miss information near boundaries How often retrieved chunks are nearly duplicates Index size and embedding cost How much duplicate text reaches the generator Do not tune overlap independently Chunk size and overlap should be evaluated together. A 100 token overlap may be large for a 300 token chunk but small for a 2,000 token chunk. The chunking method also matters. Fixed size chunking may need more overlap than structure aware chunking. For the reason overlap exists, see “Why is chunk overlap used?” . In short, use enough overlap to preserve context across chunk boundaries, but keep it small enough to avoid excessive duplication. A modest 10–20% of chunk size can be a useful starting point, but the correct value should be determined through retrieval evaluation on your own documents and queries.
What is parent-child chunking?
Parent child chunking is a RAG chunking pattern where the same document is represented at two related levels: Parent chunks contain larger sections of text. Child chunks contain smaller pieces taken from those parent sections. The child chunks are usually used for precise retrieval, while the larger parent chunks are used to provide richer context to the language model. Simple example Suppose a document contains a section about authentication. The parent contains the broader section, while each child focuses on a smaller topic. Why use two levels? Small chunks are often easier to retrieve accurately because they contain less unrelated text. But a very small chunk may not contain enough surrounding information for generation. Parent child chunking separates these two needs: What is stored? A parent child setup usually keeps a relationship between the two levels. For example: The child record may also store metadata such as: Document ID Section heading Page number Parent ID Source information This allows the retrieval system to find the larger parent after a child has matched the query. What gets embedded? A common design is to embed and index the smaller child chunks because they provide fine grained retrieval. The larger parent chunks may be stored separately and retrieved through their parent IDs rather than searched directly. Other implementations may index both levels, but the defining idea is the relationship between smaller children and larger parents. Example query Imagine the user asks: A small child chunk about access token expiration may match very closely. The RAG system can then use its parent id to recover the larger authentication section before sending context to the generator. How is this different from ordinary chunking? Ordinary flat chunking usually stores each chunk independently. Parent child chunking preserves relationships: This lets retrieval stay fine grained without losing access to broader context. Parent child chunking and hierarchical chunking Parent child chunking is a simple form of hierarchical organization. It typically focuses on two levels: parent and child. Hierarchical chunking can extend the idea to several levels such as document → section → subsection → paragraph. When is parent child chunking useful? It is especially useful for: Long technical documentation Manuals Policies Legal documents Reports Knowledge bases with large structured sections It is most valuable when small chunks improve retrieval but larger surrounding sections improve answer quality. In short, parent child chunking combines small retrieval units with larger context units. The child helps the system find the right passage, while the parent provides the broader information needed for a grounded answer.
How should code be chunked for RAG?
Code should usually be chunked for RAG using logical program structure rather than only fixed character or token counts. Source code has natural boundaries such as files, classes, functions, methods, and modules. Preserving those boundaries makes each retrieved chunk easier to understand and more useful for code related questions. Prefer semantic code units Useful chunk boundaries include: A complete function A complete method A class A small module A configuration block A related group of constants or definitions For example, instead of splitting this code every 500 tokens, keep the function intact: A complete function is usually a better retrieval unit than a fragment containing only half of its logic. Preserve the structural context A code chunk should retain enough surrounding information to explain where it belongs. Useful metadata can include: Repository name File path Programming language Class name Function or method name Module or package name Start and end line numbers For example: This metadata can improve filtering, navigation, and source attribution during retrieval. Keep related comments and documentation with the code Docstrings and comments often explain: What a function does Expected inputs Return values Constraints Edge cases When possible, keep them with the code they describe. For example: Separating the docstring from the function can remove useful semantic context. Handle large functions or classes carefully Some code units may be too large for a practical retrieval chunk. In that case, split them at smaller structural boundaries such as: Methods inside a class Logical blocks inside a very large function Groups of closely related helper functions The parent class or function identity should still be preserved in metadata. Preserve important dependencies A code chunk may depend on symbols defined elsewhere. Useful context can include: Imported modules Called functions Referenced classes Interfaces Parent classes You do not necessarily need to duplicate all dependency code inside every chunk, but preserving symbol names and relationships helps retrieval systems connect related code. Avoid arbitrary mid statement splits Blind fixed size chunking can produce fragments such as: These fragments may be syntactically incomplete and harder to interpret. If a size limit must be used, first split using code aware boundaries and then apply token limits within those units. Consider language aware parsing Code parsers, abstract syntax trees, or language specific splitters can identify structural elements more reliably than plain text delimiters. For example, a parser can detect: Functions in Python Methods and classes in Java Components and functions in JavaScript or TypeScript Structs and functions in Go This is especially useful for large repositories containing many files and languages. Add relationships when repository level questions matter For questions such as: or: chunk text alone may not be enough. The ingestion pipeline can additionally preserve relationships such as: Function calls Imports Class inheritance File dependencies These relationships can later support graph based or metadata assisted retrieval. A practical code chunking flow In short, code should be chunked around meaningful program structures such as functions, methods, and classes, while preserving file paths, symbol names, comments, and dependency context. Structure aware code chunks are generally more useful for RAG than arbitrary fixed size fragments.
How should tables be chunked for RAG?
Tables should be chunked for RAG in a way that preserves the relationship between headers, rows, columns, and the surrounding document context . Unlike normal prose, table meaning often depends on structure. A cell such as 42 is meaningless unless the system also knows which row and column it belongs to. Preserve headers with every table chunk If a large table is split into multiple chunks, repeat the relevant headers in each chunk. For example: If another chunk contains later rows, it should still carry: This keeps each chunk understandable on its own. Prefer row groups over arbitrary token splits Do not split a table in the middle of a row or separate a value from its associated columns. A safer approach is to group complete rows while staying within the target chunk size. For example: This preserves row integrity. Preserve multi level headers Some tables have grouped or hierarchical headers. For example: If this table is chunked, the hierarchy between 2025, 2026, Revenue, and Profit should remain explicit. Flattening or dropping one header level can make retrieved values ambiguous. Keep table title and surrounding context Many tables depend on nearby text for meaning. Useful context may include: Table title Section heading Caption Units Date range Footnotes For example: This context should be stored with the chunk or preserved as metadata. Small tables can remain as one chunk If a table is compact and fits comfortably within the target chunk size, keeping it intact is usually preferable. Splitting a small table unnecessarily can destroy useful relationships between rows and columns. Large tables may need logical partitioning Very large tables can be divided using meaningful dimensions such as: Row ranges Categories Time periods Geographic regions Product groups For example: This can make retrieval more focused than arbitrary row counts when the table has natural groups. Consider a text representation for embedding Many embedding models work better when table structure is converted into explicit text. For example, a row can be represented as: This makes the column meaning explicit in the text that is embedded. For wide tables, a row oriented or record oriented representation can often be easier to retrieve than raw visual formatting. Preserve the original table structure too Even if the embedding text is normalized, it is useful to keep the original table or structured representation alongside it. This allows the application to: Reconstruct the table for display Preserve formatting Cite the original source Answer questions requiring multiple cells Handle merged cells and repeated labels carefully PDF and spreadsheet tables may contain merged cells or visually implied labels. During parsing, those labels should be propagated so each row has complete meaning. For example, instead of: normalize it to: This makes each row self contained. Preserve table metadata Useful metadata may include: Document ID Page number Table number Table title Section heading Sheet name Row range Column names This helps retrieval filtering and source attribution. Avoid excessive overlap Table chunks usually do not need the same kind of token overlap used for prose. Instead, repeat the structural context that matters: Headers Labels Units Titles This provides continuity without duplicating many full rows. A practical table chunking flow In short, tables should be chunked around complete rows or meaningful logical groups while preserving headers, labels, units, captions, and metadata. The goal is to make every chunk understandable independently without breaking the relationships that give table cells their meaning.
What is an embedding in a RAG system?
An embedding in a RAG system is a numerical vector that represents the meaning of a piece of content, such as a document chunk or a user query. Instead of storing only the original text for retrieval, a RAG system also converts that text into a vector produced by an embedding model. Simple idea Suppose a document contains: An embedding model converts that text into a vector such as: The real vector may contain hundreds or thousands of numerical dimensions. Those numbers are not human readable words. Together, they encode patterns learned by the embedding model about the content's meaning. What gets embedded in RAG? A typical RAG system creates embeddings for: Document chunks during ingestion User queries during retrieval For example: and: The query vector can then be compared with stored document vectors. Embeddings represent semantic information Embeddings are designed so that pieces of text with similar meaning can be located near each other in vector space. For example, these two phrases may receive similar representations: and: They do not use the same words, but they express related meaning. Embeddings are not the original content An embedding is a mathematical representation of the content, not a replacement for the source text. A RAG system normally stores both: The original chunk text Its embedding vector It may also store metadata such as document ID, page number, section heading, or permissions. Where are embeddings stored? Document embeddings are commonly stored in a vector capable database or search index. A simplified record might look like: During retrieval, the system compares the query embedding with these stored vectors to find semantically related chunks. Embedding dimension Every embedding produced by a particular model has a fixed number of dimensions. For example, conceptually: The exact number depends on the embedding model. All vectors being compared in the same vector space must be compatible. Embeddings are model dependent Different embedding models learn different vector spaces. A vector produced by one model should not normally be compared directly with vectors produced by an unrelated model. This is why a RAG system should use compatible embedding models for indexed documents and incoming queries. That topic is covered separately in “Why should document and query embeddings use compatible embedding models?” . Embeddings can represent more than text Depending on the model, embeddings can also represent: Images Audio Code Multimodal content But in a standard text RAG pipeline, embeddings usually represent text chunks and user queries. The reasons RAG systems rely on these vectors for retrieval are covered separately in “Why are embeddings used in RAG?” . In short, an embedding is a dense numerical representation of content that places semantically related items near one another in vector space. In RAG, embeddings are created for document chunks and queries so the retrieval system can compare their meaning mathematically.
Why are embeddings used in RAG?
Embeddings are used in RAG because they let the retrieval system compare meaning mathematically instead of relying only on exact keyword matches. A RAG system needs to find document chunks that are relevant to a user's question. Embeddings make this possible by converting both stored content and incoming queries into vectors that can be compared in the same semantic space. They help retrieve semantically similar content Consider this query: The knowledge base may contain: These sentences do not use the same words, but they express related meaning. Keyword only retrieval may miss such matches if the wording differs significantly. Embedding based retrieval can identify them because their vectors can be close in semantic space. Documents are embedded during ingestion When documents are prepared for RAG, their chunks are converted into embeddings. Conceptually: The vector becomes a searchable representation of that chunk's meaning. The user query is embedded too When a user asks a question, the query is passed through a compatible embedding model. The retrieval system then compares the query vector with stored document vectors. Similar vectors indicate related meaning Vector similarity functions such as cosine similarity or dot product can estimate how close the query is to each stored chunk. A simplified flow is: The exact similarity method depends on the embedding model and vector index. Embeddings reduce dependence on exact wording This is important because users often ask questions differently from how source documents are written. For example: The wording is different, but an embedding model can still represent the semantic relationship between the two. Embeddings make large scale semantic search practical A RAG knowledge base may contain thousands or millions of chunks. Embeddings allow those chunks to be indexed in a vector search system so the application can retrieve a small set of likely relevant passages instead of sending the entire knowledge base to the language model. They support the retrieval stage, not generation itself Embeddings are mainly used to help locate relevant information. They do not directly produce the final natural language answer. The typical relationship is: Why use embeddings together with other retrieval methods? Embeddings are strong at semantic matching, but they are not perfect for every query. Exact identifiers, product codes, names, numbers, or rare terms may sometimes be better handled by keyword or sparse retrieval. This is why some RAG systems combine embedding based dense retrieval with keyword search in a hybrid retrieval system. That topic is covered later in “What is hybrid search in RAG?” . Compatible embeddings are essential The stored document chunks and incoming queries must be represented in a compatible vector space. If unrelated embedding models are used, their vectors generally cannot be meaningfully compared. This is covered separately in “Why should document and query embeddings use compatible embedding models?” . In short, embeddings are used in RAG to turn documents and queries into comparable numerical representations of meaning. This enables semantic search, helps retrieve relevant information even when wording differs, and allows large knowledge bases to be searched efficiently before the language model generates an answer.
How are document chunks converted into embeddings?
Document chunks are converted into embeddings by passing each chunk through an embedding model , which transforms the chunk's text into a fixed length numerical vector. This usually happens during the RAG ingestion pipeline after documents have been parsed and split into chunks. 1. Start with a document chunk After chunking, the system may have text such as: The chunk may also have metadata such as: Document ID Page number Section heading Source URL Permissions The metadata is usually stored alongside the embedding rather than being converted into the same vector automatically. 2. Prepare the text for the embedding model The chunk text is formatted according to the embedding model's requirements. This may include: Normalizing whitespace Preserving useful headings Adding selected contextual metadata to the text Ensuring the chunk fits within the model's input limit For example: Whether headings or other context should be included depends on the retrieval design. 3. Tokenize the chunk The embedding model does not directly process raw characters. The text is first converted into tokens by the model's tokenizer. Conceptually: The exact tokenization depends on the embedding model. 4. Pass the tokens through the embedding model The tokenized chunk is then processed by the embedding model. The model converts the semantic information in the chunk into a numerical representation. 5. Produce a fixed length vector The output is a vector with a fixed number of dimensions determined by the model. For example: A real embedding may contain hundreds or thousands of values. Every chunk embedded with the same model produces a vector with the same dimensionality. 6. Store the vector with the chunk The embedding is normally stored together with the original text and metadata. A simplified record may look like: The embedding can be stored in a vector database or another system that supports vector indexing and similarity search. 7. Repeat this for every chunk The ingestion pipeline processes all chunks in the knowledge base. Each vector represents one retrieval unit. Embeddings are often generated in batches For efficiency, production systems commonly send several chunks to the embedding model in one batch rather than embedding them one at a time. Conceptually: Batching can improve ingestion throughput while producing one embedding per chunk. What happens later during retrieval? When a user submits a query, the query is converted into an embedding using a compatible model. The query vector is compared with the stored chunk vectors to identify semantically similar chunks. The query side process is covered separately in “How is a user query converted into an embedding?” . Why model consistency matters Document chunks and queries should be embedded into the same compatible vector space. If document vectors were created with one unrelated embedding model and the query vector with another, similarity scores would generally not be meaningful. This is covered in “Why should document and query embeddings use compatible embedding models?” . In short, document chunks are converted into embeddings by preparing and tokenizing each chunk, passing it through an embedding model, receiving a fixed length numerical vector, and storing that vector alongside the original chunk and its metadata for later similarity search.
How is a user query converted into an embedding?
A user query is converted into an embedding by passing the query text through the same or a compatible embedding model used for the indexed document chunks. This happens at retrieval time, after the user submits a question and before vector similarity search begins. 1. Start with the user's query For example: The system first takes the query text exactly as the retrieval pipeline expects to process it. 2. Apply any query preprocessing Depending on the RAG system, the query may be lightly prepared before embedding. This can include: Trimming unnecessary whitespace Normalizing text Removing unsupported characters Adding a task specific prefix required by the embedding model Some embedding models are designed to use different prefixes or instructions for queries and documents. For example, conceptually: The exact formatting depends on the embedding model. 3. Tokenize the query The query text is converted into tokens by the embedding model's tokenizer. Conceptually: Because user queries are usually much shorter than document chunks, they normally fit comfortably within the model's input limit. 4. Pass the tokens through the embedding model The tokenized query is processed by the embedding model. The result is a fixed length numerical vector representing the semantic meaning of the query. For example: The real vector may contain hundreds or thousands of dimensions depending on the model. 5. Compare the query vector with stored document vectors The query embedding is not normally stored permanently just to answer one request. Instead, it is used immediately to search the vector index. A simplified flow is: Similarity may be calculated using methods such as cosine similarity, dot product, or another metric supported by the embedding model and vector index. 6. Retrieve the closest document chunks The vector database returns chunks whose embeddings are closest to the query embedding. For example, the query: may retrieve: even though the wording is different. That semantic matching is the main reason the query is embedded. The query and documents must share a compatible vector space The query embedding must be comparable with the stored document embeddings. This generally means using the same embedding model or a model pair explicitly designed to produce compatible query and document representations. If unrelated models are used, the vectors usually do not share a meaningful coordinate space. This is covered in “Why should document and query embeddings use compatible embedding models?” . Query embeddings are created at runtime Document embeddings are usually generated during ingestion and stored in advance. Query embeddings are generated when the user asks a question. This lets the system compare each new question against the precomputed document vectors. Query preprocessing should be consistent Any preprocessing required by the embedding model should be applied consistently. For example, if the model expects a special query instruction, omitting it can reduce retrieval quality. The retrieval pipeline should follow the embedding model's recommended input format for both query and document representations. In short, a user query is converted into an embedding by preparing and tokenizing the query, passing it through a compatible embedding model, and obtaining a fixed length query vector. That vector is then compared with stored document chunk embeddings to retrieve semantically relevant content.
Why should document and query embeddings use compatible embedding models?
Document and query embeddings should use compatible embedding models because vector similarity is only meaningful when both sides are represented in the same or intentionally aligned vector space. A RAG system compares the query vector with stored document vectors. If those vectors come from unrelated embedding models, the numerical coordinates usually do not have the same meaning. Think of embeddings as coordinates An embedding model maps text into a multidimensional vector space. For example: and: Because both vectors were produced in the same space, their distance or similarity can be interpreted. What happens if unrelated models are mixed? Suppose document chunks were embedded with Model A, but the query is embedded with unrelated Model B. Even if the vectors happen to have the same number of dimensions, dimension 1 in Model A does not necessarily represent the same learned feature as dimension 1 in Model B. Comparing those vectors can therefore produce meaningless similarity scores. Matching dimensions alone is not enough Two embedding models may both output, for example, 1,024 dimensional vectors. That does not make them compatible. Compatibility requires the vectors to be trained or designed so their coordinates belong to the same semantic space. The safest approach For most RAG systems, the simplest approach is: This ensures that both document and query vectors are directly comparable. Some models use different query and document encoders Compatibility does not always mean the exact same input path. Some retrieval models are intentionally trained with: A query encoder A document or passage encoder These encoders may use different instructions, prefixes, or even separate model components while still producing vectors in a compatible space. For example: In that case, using the model's recommended query/document encoding behavior is correct. Why compatibility matters for similarity search Vector databases rank document vectors using functions such as: Cosine similarity Dot product Euclidean distance These calculations assume the vectors are comparable. If query and document embeddings come from incompatible spaces, the ranking can become effectively arbitrary. What if you change the embedding model? If a RAG system switches to a new embedding model, existing document vectors generally need to be regenerated with that new model. Otherwise, new query vectors would be compared against vectors created in the old embedding space. This migration topic is covered separately in “How should embeddings be regenerated when the embedding model changes?” . Model specific input instructions matter too Even when the same embedding model is used, some models expect different input formatting for queries and documents. For example, a model may require conceptually: Ignoring the model's recommended query/document format can reduce retrieval quality even though the underlying model is compatible. Compatibility across an index All vectors searched together in the same index should normally use a consistent embedding configuration. That includes: Model or compatible encoder pair Output dimensionality Recommended normalization Query/document instructions This keeps similarity scores meaningful across the indexed collection. In short, document and query embeddings must be compatible because vector retrieval depends on comparing them inside the same semantic coordinate system. Using unrelated embedding models can make similarity scores meaningless, even when their output dimensions happen to match.
How do you choose an embedding model for RAG?
Choosing an embedding model for RAG means selecting the model that produces the most useful vector representations for your documents, queries, retrieval workload, and production constraints . The best model is not necessarily the one with the largest benchmark score. It is the one that retrieves the right chunks reliably enough for your application at an acceptable cost and latency. 1. Start with retrieval quality The main question is: Evaluate candidate models using representative query document pairs from your own knowledge base. For example: A good embedding model should rank the relevant chunk near the top. Useful retrieval metrics can include: Recall@K Precision@K Hit Rate Mean Reciprocal Rank nDCG These metrics are covered later in the RAG evaluation section. 2. Match the model to your domain General purpose embedding models work well for many applications, but domain specific content may benefit from a model that handles that domain well. Examples include: Legal documents Medical text Scientific papers Source code Financial documents Product documentation If your corpus contains specialized terminology, test whether the model retrieves those concepts correctly. 3. Check language support If your RAG system contains multiple languages, choose a multilingual embedding model that supports the languages you actually use. For example, a system containing English, Hindi, German, and Japanese documents should be evaluated across all of those languages rather than only on English examples. Cross language retrieval may also matter. For example: Not every embedding model handles this equally well. 4. Check the model's input length Every embedding model has an input limit. Your chunks must fit inside that limit. If your RAG pipeline uses large chunks, long context embedding models may be important. For techniques such as late chunking, long context support becomes especially relevant because the model must process a larger document span before producing chunk representations. 5. Consider embedding dimensions Embedding models produce vectors with a fixed number of dimensions. Higher dimensional vectors can require more: Storage Memory Network bandwidth Vector search computation Dimension alone does not determine quality. A smaller embedding model can outperform a larger vector representation on a particular retrieval task. The impact of vector dimensionality is covered separately in “How does embedding dimension affect a RAG system?” . 6. Measure embedding latency Embedding latency matters during both: Document ingestion Query time retrieval Document embeddings can often be generated asynchronously in batches, but query embeddings usually sit directly on the user request path. If query embedding takes too long, overall RAG response latency increases. 7. Consider cost Hosted embedding APIs may charge based on input tokens or requests. For a large corpus, document ingestion can require embedding millions of chunks. Estimate: Also consider the continuing cost of embedding every incoming query. 8. Consider deployment requirements Some organizations need embeddings to run: On premises Inside a private cloud Without sending data to an external API On CPU only infrastructure These requirements may favor an open weight or locally deployable model. Hosted APIs may be easier to operate, while self hosted models can offer more control. 9. Check whether the model uses query/document instructions Some retrieval models expect different formatting for queries and documents. For example, conceptually: Follow the model's recommended input format. Using the same model incorrectly can still reduce retrieval quality. 10. Make sure the similarity metric is appropriate Embedding models may be designed to work best with: Cosine similarity Dot product Euclidean distance Use the similarity function recommended for the model when available. 11. Evaluate on your own RAG dataset Public embedding benchmarks are useful for shortlisting models, but they should not be the final decision. Your evaluation set should include realistic questions such as: Exact fact lookup Paraphrased queries Domain specific terminology Short and long queries Ambiguous queries Multilingual queries if applicable Then compare how often each model retrieves the correct chunks near the top. A practical selection process Do not choose using one factor alone A model with excellent retrieval quality may be too slow or expensive. A very fast model may not understand your domain well enough. A high dimensional model may create unnecessary storage cost without improving retrieval. The decision should balance: Retrieval quality Domain fit Language coverage Context length Vector dimensions Latency Cost Deployment constraints In short, choose an embedding model for RAG by testing how well it retrieves your actual relevant chunks, then balance that quality against language/domain support, input length, dimensions, latency, cost, and deployment requirements. Production selection should be based on your own retrieval evaluation rather than benchmark rankings alone.