Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

What is dot-product similarity in vector retrieval?

Dot product similarity is a way to measure how strongly two vectors point in the same direction. In vector retrieval, it can be used to score how well a query embedding matches a document embedding. How it works Suppose we have two vectors: Query = [1, 2] Document = [3, 4] The dot product is calculated by multiplying matching dimensions and adding the results: 1 × 3 + 2 × 4 = 11 A larger dot product generally means the two vectors are more strongly aligned. Why it is useful in retrieval During retrieval, the query is converted into an embedding. Each document chunk already has an embedding. The system computes a similarity score between the query vector and document vectors, then ranks the documents. With dot product: Higher score → usually stronger match Lower score → weaker match Important difference from cosine similarity Cosine similarity focuses mainly on the angle between two vectors. Dot product depends on both: Direction Vector magnitude This means two vectors pointing in a similar direction can receive different dot product scores if one has a much larger magnitude. That may be desirable or undesirable depending on how the embedding model was trained. Example Consider: A = [1, 1] B = [2, 2] C = [1, 0] A and B point in exactly the same direction. Their cosine similarity is 1 because their direction is identical. But their dot product is: A · B = 4 The magnitude of B contributes to the score. So dot product is not purely measuring angle. What happens with normalized embeddings? If both vectors are normalized to unit length, their magnitude becomes 1. In that case: Dot product = cosine similarity This is why some vector search systems can use dot product and cosine similarity almost interchangeably when embeddings are normalized. Dot product vs cosine similarity Use cosine similarity when you mainly care about directional similarity and want vector magnitude to have little or no effect. Use dot product when the embedding model is designed so that magnitude carries useful information, or when the model provider recommends dot product scoring. The similarity metric should match how the embedding model was trained. Performance consideration Dot product is computationally simple. For large scale vector search, it can be efficient because it only requires element wise multiplication and summation. Vector databases often combine it with approximate nearest neighbor indexes to retrieve similar vectors quickly. Interview takeaway Dot product similarity multiplies corresponding dimensions of two vectors and sums them. A larger value generally indicates stronger alignment, but unlike cosine similarity, dot product is affected by vector magnitude. If embeddings are unit normalized, dot product and cosine similarity produce equivalent rankings. The correct metric should match the embedding model's training and recommendations.

What is Euclidean distance in vector retrieval?

Euclidean distance measures the straight line distance between two vectors in embedding space. In vector retrieval, a smaller Euclidean distance means the query embedding and document embedding are closer together. How it works Suppose: Query = [1, 2] Document = [4, 6] Euclidean distance is: sqrt((4 1)^2 + (6 2)^2) = sqrt(9 + 16) = 5 The smaller this distance, the more similar the vectors are considered. How retrieval uses it During RAG retrieval: User query → convert to embedding → compare with document embeddings → calculate distances → rank documents from smallest distance to largest So unlike cosine similarity or dot product, where a larger score usually means a stronger match, Euclidean distance works in the opposite direction: Smaller distance → more similar Larger distance → less similar Why vector magnitude matters Euclidean distance considers the actual positions of vectors, including their magnitude. For example: A = [1, 1] B = [2, 2] Even though A and B point in the same direction, they are not at the same location. Their Euclidean distance is greater than zero. Cosine similarity, however, would treat them as having the same direction and give a cosine similarity of 1. This is one of the main differences between the two metrics. What happens with normalized embeddings? If embeddings are normalized so every vector has unit length, Euclidean distance and cosine similarity become closely related. For unit normalized vectors: smaller Euclidean distance ↔ higher cosine similarity This means they usually produce the same ranking. So when embeddings are normalized, choosing between cosine similarity and Euclidean distance may make little difference to retrieval order. When should Euclidean distance be used? Use it when: The embedding model is designed for Euclidean distance Absolute position and magnitude in embedding space are meaningful The model provider recommends L2 or Euclidean distance Do not choose the metric only because it sounds intuitive. The distance function should match how the embedding model was trained. Euclidean distance vs cosine similarity Cosine similarity asks: "How similar are the directions of these vectors?" Euclidean distance asks: "How far apart are these vectors in space?" That distinction matters when vectors have different magnitudes. Interview takeaway Euclidean distance measures the straight line distance between two embeddings. Smaller distance means greater similarity. Unlike cosine similarity, Euclidean distance is affected by both direction and magnitude. If embeddings are unit normalized, Euclidean distance and cosine similarity usually produce equivalent rankings. The best metric depends on how the embedding model was trained.

How do you evaluate embedding quality for RAG?

Embedding quality in RAG should be evaluated by asking a practical question: Does this embedding model retrieve the right chunks for real user queries? Looking at embedding dimension or individual similarity scores is not enough. 1. Build a labeled evaluation set Create realistic queries and identify which document chunks should be considered relevant. Example: Query: "How many days can an enterprise customer request a refund?" Relevant chunk: "Enterprise customers may request a refund within 45 days." This gives you a known target for retrieval evaluation. 2. Measure retrieval quality Run the queries through the embedding model and vector search, then measure whether relevant chunks appear near the top. Useful metrics include: Recall@K Precision@K MRR NDCG Hit Rate For RAG, Recall@K is especially important because if the correct evidence is never retrieved, the generator cannot use it. 3. Test difficult semantic cases A good embedding model should handle queries whose wording differs from the document. Example: Query: "Can unused vacation days move to next year?" Document: "Employees may carry forward up to 10 days of unused annual leave." The words are different, but the meaning is similar. This is where embedding quality matters. 4. Include hard negatives Do not evaluate only obvious examples. Include documents that are semantically close but wrong. Example: Query: "Enterprise refund period" Relevant: "Enterprise customers: 45 days" Hard negative: "Standard customers: 30 days" A good embedding model should rank the enterprise policy higher. 5. Compare models under the same conditions When comparing embedding models, keep other variables fixed: Same documents Same chunking Same queries Same Top K Same vector index settings Same similarity metric when appropriate Otherwise, you may incorrectly attribute improvements to the embedding model. 6. Evaluate your actual domain A model that performs well on general benchmarks may still perform poorly on: Medical terminology Legal documents Source code Product identifiers Internal abbreviations Multilingual content So benchmark results are useful for shortlisting, but your own retrieval dataset is more important. 7. Check downstream RAG quality too Better retrieval metrics should ideally improve the final answer. After comparing embedding models, also measure: Final answer correctness Groundedness Citation accuracy Latency Embedding cost Storage requirements A larger or more expensive embedding model is only worthwhile if it produces meaningful retrieval or answer quality gains. Interview takeaway Evaluate embedding quality through retrieval performance on a representative labeled query document dataset. Compare models using metrics such as Recall@K, MRR, and NDCG, include paraphrases and hard negatives, keep the rest of the retrieval pipeline fixed, and validate the results on your own domain. The best embedding model is the one that retrieves the right evidence reliably at an acceptable latency and cost.

What are multimodal embeddings in RAG?

Multimodal embeddings represent different types of content—such as text, images, audio, or video—as vectors that can be compared in a shared or compatible embedding space. In RAG, this allows retrieval to work across more than just text. Why multimodal embeddings are useful Traditional RAG often works like this: Text document → text embedding User query → text embedding Vector search → retrieve relevant text But many real documents contain information that text extraction alone cannot capture well. Examples include: Diagrams Product images Charts Screenshots Scanned forms Slides Video frames Multimodal embeddings let the retrieval system represent some of these non text elements directly. Example Suppose a maintenance manual contains a diagram showing how a valve assembly is connected. The user asks: "Show me the diagram for the pressure control valve." A text only RAG system may retrieve the surrounding paragraph. A multimodal system can also embed the diagram itself and retrieve the image because its representation is semantically related to the query. Shared embedding space Some multimodal models place text and images into the same vector space. That means: Text query: "red sports car" can be embedded and compared directly against image embeddings. Images that visually represent a red sports car should appear close to the query vector. This enables: Text → image retrieval It may also support: Image → text retrieval Image → image retrieval depending on the model. How it fits into a RAG pipeline A multimodal RAG ingestion pipeline may look like: Document → extract text, images, charts, and other elements → generate embeddings for each modality → store vectors with source metadata At query time: User query → generate compatible embedding → search relevant vectors → retrieve text, images, or both → provide the selected evidence to a multimodal LLM Do all modalities need one embedding model? Not necessarily. There are two common designs. One approach uses a shared multimodal embedding model so text and images can be compared directly. Another approach uses separate embedding models and separate indexes, then combines their retrieval results. The right design depends on whether cross modal similarity is required and how well the available models support the domain. Important limitation Multimodal embeddings help with retrieval, but they do not automatically understand every detail inside an image. For example, a chart may require: OCR for labels Structured extraction of values Vision reasoning A caption or textual summary So multimodal embeddings are often one part of a larger document processing pipeline. Interview takeaway Multimodal embeddings convert different content types such as text and images into vector representations that can be searched semantically. In RAG, they enable retrieval across modalities—for example, using a text query to retrieve a relevant image or diagram. They are especially useful when important information cannot be represented reliably through extracted text alone.

When are image embeddings needed in RAG?

Image embeddings are needed in RAG when the visual content itself carries meaning that cannot be recovered reliably from text alone. They are not required for every PDF that happens to contain images. When image embeddings are useful 1. The image contains information that text extraction misses Examples: Product photos Diagrams Medical images UI screenshots Maps Engineering drawings Visual defects If the user's question depends on what the image looks like, an image embedding can help retrieve the right visual element. Example Suppose a repair manual contains several wiring diagrams. The user asks: "Which diagram shows the backup power connection?" OCR may extract labels such as "battery" or "backup power," but it may not capture the actual visual relationship between components. An image embedding can represent the diagram as a visual object and make it retrievable from a compatible text or image query. 2. Users may search with images Image embeddings are especially useful when the query itself can be visual. Example: A user uploads a picture of a machine component and asks: "Find the matching part in the maintenance manual." The query image can be embedded and compared against indexed image embeddings. This supports: Image → image retrieval and, with a multimodal embedding model: Image → text or text → image retrieval 3. Visual similarity matters more than wording In some applications, two images may be similar even when their captions are different or missing. Examples: Similar product designs Similar damage patterns Matching logos Similar UI screens Text embeddings alone may not capture these relationships. When image embeddings may NOT be necessary If an image contains mostly text, OCR may be enough. Example: A scanned invoice If the question is: "What is the invoice total?" the useful information is textual and structured. OCR + text retrieval may work better than relying on an image embedding. Similarly, if a chart has a clear textual description or extracted data table, text or structured retrieval may be more precise. A hybrid approach is often best For many documents, use both: Image → OCR / caption / structured extraction → text embedding and: Image → image embedding This gives you two retrieval signals. The text representation helps with exact labels, numbers, and keywords. The image embedding helps with visual semantics. Keep image and source relationships When indexing an image, preserve metadata such as: Document ID Page number Figure number Caption Nearby section Image type This lets the system retrieve the image together with its surrounding explanation. Interview takeaway Use image embeddings when the visual appearance or structure of an image is important for retrieval, especially for diagrams, screenshots, product images, drawings, or image based queries. If the important information can be reliably extracted as text or structured data, OCR or text embeddings may be enough. In many multimodal RAG systems, the strongest design combines visual embeddings with OCR, captions, and source metadata.

Can text and images be stored in the same RAG retrieval system?

Yes. Text and images can be stored in the same RAG retrieval system, but the design depends on whether their embeddings are directly comparable. Option 1: Shared multimodal embedding space A multimodal embedding model can encode text and images into the same vector space. That means the system can store: Text chunk embeddings Image embeddings inside the same vector collection. A text query can then retrieve either text or images based on similarity. Example: User asks: "Show the diagram for the cooling system." The query is converted into a multimodal embedding. The vector search may return: A text paragraph describing the cooling system A diagram of the cooling system because both live in the same compatible embedding space. Option 2: Separate indexes You can also keep: Text embeddings → text index Image embeddings → image index At query time, the system searches both indexes and combines the results. This is useful when: Different embedding models are used Text and image vectors have different dimensions Their similarity scores are not directly comparable You want different retrieval logic for each modality The final results can then be merged using ranking or fusion logic. Why compatibility matters You should not put arbitrary text embeddings and arbitrary image embeddings into one collection and assume they can be compared. If they were generated by unrelated models, the vectors may represent completely different spaces. Similarity scores would have no useful meaning. The embeddings must either: Come from a model designed for cross modal comparison or Be searched separately and combined later Store modality metadata Each indexed item should identify what it represents. Useful metadata includes: modality = text or image document id page number section figure number caption source URI or file ID This helps the retriever and generator understand what was returned. Keep text and image relationships Suppose page 12 contains: A paragraph describing a circuit Figure 4 showing the circuit Both should be linked through metadata. Then if the image is retrieved, the system can also fetch nearby explanatory text. Similarly, retrieving the paragraph can bring the associated figure. This often produces better answers than treating each modality as completely independent. Do you store the actual image in the vector database? Usually, the vector database stores: Image embedding Metadata A reference to the original image The image itself may live in object storage, a document store, or another file system. The vector record helps find it. Unified vs separate retrieval A shared multimodal index is convenient when one model supports text image comparison well. Separate indexes give more control when different models or retrieval strategies are needed. Neither architecture is automatically better. The right choice depends on the models, scale, and types of queries your application needs to support. Interview takeaway Text and images can be part of the same RAG retrieval system. If they are encoded by a compatible multimodal model, they can be stored and searched in one shared vector space. If different embedding models are used, keep separate indexes and combine results later. In both cases, preserve modality and source metadata so retrieved text and images can be linked back to the same document context.

How should embeddings be regenerated when the embedding model changes?

When the embedding model changes, the safest approach is to regenerate embeddings for the entire indexed corpus using the new model. You should not normally mix old and new embeddings inside the same searchable vector space. Why old and new embeddings should not be mixed Different embedding models create different vector spaces. Even if both models output vectors with the same number of dimensions, those dimensions do not necessarily mean the same thing. For example: Old model: Document A → vector X New model: Same document → vector Y The numbers may look similar in shape, but they belong to different learned representations. Comparing a query embedding from the new model against document embeddings from the old model can produce meaningless similarity scores. What should be regenerated? When changing the embedding model, regenerate embeddings for: All document chunks Titles or summaries that are embedded separately Image embeddings, if the multimodal model changes Any cached query embeddings that are reused The original source content and metadata do not necessarily need to change. You are rebuilding the vector representation, not the documents themselves. Safe migration pattern A practical migration looks like: Existing index → keep serving traffic New embedding model → re embed the corpus → write vectors to a new index or collection → validate retrieval quality → switch queries to the new index → remove the old index later This avoids taking the retrieval system offline while re embedding is still running. Why use a new index? Replacing vectors in place can be risky. If only half the documents have been re embedded, the index temporarily contains vectors from two incompatible spaces. A separate index avoids this problem. It also makes rollback easy if the new model performs worse. Re embed from the original chunks Do not try to convert old embeddings mathematically into new embeddings unless a model provider explicitly supports such a transformation. Normally, you should take the original chunk text and pass it through the new embedding model. So: Original text → new embedding model → new vector not: Old vector → transform somehow → new vector Check dimension and index compatibility A new model may use a different embedding dimension. For example: Old model → 768 dimensions New model → 1536 dimensions Your vector column, index configuration, or collection schema may therefore need to change. This is another reason why a separate index is often safer. Evaluate before switching A newer embedding model is not automatically better for your RAG workload. Before migration, compare: Recall@K MRR or NDCG Domain specific retrieval quality Latency Embedding cost Storage requirements Use the same evaluation dataset for both models. Version your embeddings Store metadata such as: embedding model embedding model version embedding dimension embedded at This makes it easier to identify which vectors need regeneration and prevents accidental mixing. What about newly added documents during migration? If documents continue arriving while the new index is being built, use a controlled strategy. For example: Write new documents to both old and new indexes during migration or Record changes and replay them into the new index before cutover The goal is to ensure the new index is complete when traffic moves to it. Interview takeaway When the embedding model changes, regenerate embeddings from the original content and build a new compatible vector index. Do not mix embeddings from different models in the same similarity space. Validate the new model on real retrieval queries, switch traffic only after the new index is complete, and keep model/version metadata so future migrations are manageable.

What is a vector database in RAG?

A vector database is a database designed to store, index, and search vector embeddings efficiently. In RAG, it is commonly used to find document chunks that are semantically similar to a user's query. Why RAG needs it During ingestion, each document chunk is converted into an embedding. For example: Chunk: "Enterprise customers can request a refund within 45 days." Embedding: [0.14, 0.37, 0.81, ...] The vector database stores that embedding together with information about the chunk. Later, the user asks: "What is the refund period for enterprise customers?" The query is also converted into an embedding. The vector database compares the query vector with stored document vectors and returns the closest matches. What does it usually store? A vector record commonly contains: The embedding vector The original chunk or a reference to it Document ID Chunk ID Page or section Source information Metadata such as tenant, product, language, date, or permissions The vector helps with semantic retrieval. The metadata helps with filtering, traceability, and access control. How is this different from a normal database? A relational database is very good at exact structured queries. Example: customer id = 48291 or: status = 'active' A vector database is designed for similarity queries such as: "Find content whose meaning is closest to this query." That is useful when the query and document do not use exactly the same words. Example User query: "Can I move unused vacation to next year?" Document chunk: "Employees may carry forward up to 10 days of unused annual leave." A keyword only lookup may struggle if the vocabulary differs. A vector search can retrieve the chunk because the embeddings capture semantic similarity. What role does indexing play? If you have only a few hundred vectors, you could compare the query against every vector. But with millions of chunks, that becomes expensive. Vector databases use specialized indexes such as: HNSW IVF IVF PQ These make nearest neighbor search much faster, often by trading a small amount of recall for much lower latency. Is a vector database always required? No. For small datasets, a normal database with vector support may be enough. Some relational databases can store vectors and perform similarity search directly. The important capability is not the product label "vector database." The important capabilities are: Vector storage Similarity search Efficient indexing Metadata filtering Updating and deleting vectors Where it sits in the RAG flow A simplified flow is: Documents → chunking → embeddings → vector database Then at query time: User query → query embedding → vector search → relevant chunks → LLM The vector database belongs mainly to the retrieval layer. It does not generate the final answer. Interview takeaway A vector database stores embeddings and enables efficient nearest neighbor search over them. In RAG, it is used to retrieve document chunks that are semantically similar to a query, often together with metadata for filtering and source tracking. It is part of the retrieval layer, while the LLM uses the retrieved evidence to generate the final answer.

What information is stored with vectors in a RAG database?

A vector in a RAG database is rarely stored by itself. To make retrieval useful, the system usually stores the embedding together with the original content or a reference to it, plus metadata that helps with filtering, traceability, updates, and access control. 1. The embedding vector This is the numerical representation used for similarity search. Example: [0.12, 0.44, 0.78, ...] The vector is what the retrieval engine compares with the user's query embedding. 2. The text chunk or content reference The database often stores either: The actual chunk text or A pointer to where the content is stored Example: "Enterprise customers can request a refund within 45 days." After retrieval, the application needs this content to send useful evidence to the LLM. 3. Document and chunk identifiers Useful identifiers include: document id chunk id parent chunk id source id These help the system trace each retrieved vector back to the original document and support updates or deletion. 4. Document structure For larger documents, it is useful to preserve structure such as: Page number Section Heading Chapter Paragraph position Figure or table number This helps reconstruct meaningful context after retrieval. For example, if a small chunk is retrieved, the system may use its parent section to provide more complete context to the LLM. 5. Source information Useful source metadata may include: File name Document URL Repository path Website page Database record ID This is important for citations and source tracking. 6. Metadata used for filtering Metadata lets the retriever narrow the search before or after vector similarity. Examples: Product Department Region Language Document type Customer Tenant Version Effective date Status Example: User asks about Product A. Instead of searching the entire vector index, the system can search only records where: product = "Product A" This can improve both relevance and efficiency. 7. Permission information Enterprise RAG systems often need access related metadata. Examples: tenant id department id access level allowed group owner id This helps ensure retrieval is scoped to content the user is allowed to access. However, metadata filtering should complement proper authorization controls rather than being the only security boundary. 8. Version and freshness information Useful fields include: document version updated at effective from effective to embedding model embedding model version These help prevent stale documents or incompatible embeddings from being retrieved. Example record A simplified vector record may contain: chunk id: chunk 1042 document id: policy 27 text: "Enterprise customers can request a refund within 45 days." embedding: [0.12, 0.44, ...] section: "Refund Policy" page: 8 product: "Enterprise Plan" version: 3 updated at: 2026 09 15 embedding model: model v2 The vector answers: "What content is semantically similar?" The metadata answers: "Where did it come from, is it allowed, is it current, and how should it be used?" Interview takeaway A RAG vector database usually stores more than embeddings. Each vector should be connected to the original chunk or content reference, document and chunk IDs, source and structural metadata, filtering fields, permissions, version information, and embedding model metadata. The embedding enables similarity search, while the surrounding metadata makes retrieval accurate, traceable, secure, and maintainable.

What is vector indexing?

Vector indexing is the process of organizing embedding vectors so similarity searches can run much faster. Without an index, the system may have to compare the query vector with every stored vector. That is called a brute force or exact scan. For a small collection, this may be acceptable. For millions of vectors, it can become too slow. Why indexing is needed Suppose your RAG system contains 5 million document chunks. A user query becomes one embedding. If the system compares that query with all 5 million vectors every time, retrieval may become expensive and slow. A vector index creates a structure that helps the database search only the most promising parts of the vector space. Simple idea Without an index: Query vector → compare with every vector → sort by similarity → return Top K With an index: Query vector → navigate the index → inspect likely nearby candidates → return Top K The index reduces how much work must be done per query. Exact vs approximate search Some vector search methods return the mathematically exact nearest neighbors. But many production vector indexes use Approximate Nearest Neighbor, or ANN, search. ANN intentionally avoids checking every vector. That makes retrieval much faster, but it introduces a trade off: Higher speed → possibly slightly lower recall The true nearest vector may occasionally be missed. Common vector indexes Popular approaches include: HNSW IVF IVF PQ They organize vectors in different ways. HNSW builds a graph that lets search move efficiently toward nearby vectors. IVF groups vectors into regions or clusters so the system searches only selected regions. PQ compresses vectors to reduce memory and speed up large scale search. The best index depends on the workload. Does an index change the embeddings? No. The embedding model still produces the same vectors. The index is an additional data structure built around those vectors to make searching them efficient. Think of it like an index in a book: The information does not change. The index helps you reach the relevant information faster. What affects index performance? Important factors include: Number of vectors Embedding dimension Similarity metric Required recall Query latency target Memory available Update frequency A configuration optimized for maximum recall may use more memory or take longer to search. A configuration optimized for very low latency may accept slightly lower recall. When is an index unnecessary? For a small dataset, exact search can be perfectly reasonable. An approximate vector index becomes more valuable as the number of vectors grows and full scans become too expensive. So indexing is mainly a scalability optimization. RAG impact If the vector index is poorly configured, the correct document chunk may exist in the database but never appear in the retrieved Top K. That means index configuration directly affects RAG retrieval quality. You should therefore evaluate both: Search latency Retrieval recall not latency alone. Interview takeaway Vector indexing organizes embeddings so nearest neighbor retrieval can be performed efficiently without scanning every stored vector. At large scale, indexes such as HNSW or IVF often use approximate search to reduce latency, trading a small amount of recall for much faster retrieval. The embeddings themselves do not change—the index is the structure that makes searching them scalable.