Retrieval-Augmented Generation

Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.

How does embedding dimension affect a RAG system?

Embedding dimension affects a RAG system because it determines how many numerical values are stored and compared for every embedded document chunk and query. A higher dimensional embedding contains more coordinates, while a lower dimensional embedding contains fewer. For example: The dimension is fixed by the embedding model or by the model's supported output configuration. 1. Higher dimensions increase storage requirements Every chunk in the vector index stores one embedding. If each vector uses more dimensions, each record consumes more storage. Conceptually: So a 1,536 dimensional float vector requires roughly four times as much raw vector storage as a 384 dimensional vector using the same numeric precision. For millions of chunks, that difference becomes significant. 2. Higher dimensions increase memory usage Vector indexes often keep substantial portions of their data structures in memory for fast search. Larger vectors therefore increase memory pressure. This can affect: RAM requirements Cache efficiency Index size Hardware cost The impact becomes more visible as the number of indexed chunks grows. 3. Higher dimensions can increase similarity search cost Similarity functions such as cosine similarity and dot product operate across vector dimensions. More dimensions mean more numerical operations per vector comparison. Approximate nearest neighbor indexes reduce the number of candidates that must be compared, but vector dimensionality still affects search cost. Depending on the database and index type, higher dimensions can therefore increase retrieval latency. 4. Higher dimension does not automatically mean better retrieval A common misconception is: That is not necessarily true. Embedding quality depends primarily on how well the model was trained for semantic representation and retrieval. A well trained lower dimensional model can outperform a higher dimensional model on a particular RAG workload. Dimension should therefore not be used as a standalone measure of model quality. 5. Dimension can affect retrieval quality indirectly The vector needs enough capacity to represent the distinctions that matter for the model's training objective. If a model supports reducing its output dimensions, aggressive reduction may eventually remove useful information and reduce retrieval accuracy. However, moderate dimensionality reduction may preserve most retrieval quality while significantly reducing storage and search cost. This trade off must be evaluated on the actual workload. 6. Query and document dimensions must match Vectors compared in the same index must have compatible dimensions. For example: can be compared directly. But: cannot normally be compared in the same vector operation. This is another reason document and query embeddings must use compatible embedding configurations. 7. Vector database schemas may depend on the dimension Some vector databases or vector columns are configured for a specific dimensionality. If the embedding model changes from one output size to another, the index or schema may also need to change. Existing document embeddings may need to be regenerated before the new vectors can be searched consistently. 8. Dimension affects network and ingestion cost too When embeddings are produced by an external API, larger vectors also mean more data transferred back to the application. For very large ingestion jobs, this can affect: Response payload size Network bandwidth Serialization cost Ingestion throughput The difference may be small per vector but large across millions of chunks. 9. Large scale RAG makes dimensionality more important Suppose a system stores 10 million chunks. Even a modest increase in bytes per vector is multiplied 10 million times. At that scale, dimensionality can materially affect: Storage footprint RAM requirements Index construction time Search latency Infrastructure cost 10. Choose dimension through evaluation, not intuition If an embedding model supports multiple output dimensions, evaluate several choices on representative queries. Compare: Recall@K MRR or nDCG Index size Search latency Memory usage Ingestion cost The goal is to find the smallest or most efficient representation that still provides the retrieval quality your application needs. Practical rule Do not select an embedding model simply because its vector dimension is larger. Instead: In short, embedding dimension affects RAG storage, memory use, vector search cost, and compatibility between query and document vectors. Higher dimensions may provide richer representations for some models, but they also consume more resources and do not guarantee better retrieval quality. The right dimension should be chosen through retrieval and system performance evaluation.

What is cosine similarity in RAG retrieval?

Cosine similarity in RAG retrieval is a measure of how similar two embedding vectors are based on the angle between them. In a RAG system, it is commonly used to compare a user query embedding with stored document chunk embeddings. Basic idea Suppose the query is converted into a vector: and a document chunk is represented as: Cosine similarity measures whether these two vectors point in a similar direction. Formula The cosine similarity between vectors A and B is: where: A · B is the dot product. ||A|| is the magnitude of vector A. ||B|| is the magnitude of vector B. What does the score mean? For general vectors, cosine similarity can range from: In many embedding systems, values are often concentrated in a narrower practical range, so the exact interpretation depends on the embedding model. The important point is that a higher cosine similarity usually means the query and document chunk are more semantically similar . Simple example Imagine: The query embedding should usually be more similar to Chunk A than to Chunk B. Conceptually: The retrieval system would rank Chunk A higher. Why cosine similarity focuses on direction Cosine similarity largely ignores vector magnitude and focuses on orientation. Two vectors can have different lengths but still have high cosine similarity if they point in nearly the same direction. This is useful when the semantic information learned by the embedding model is mainly represented by vector direction. Cosine similarity during RAG retrieval A simplified retrieval flow is: The highest ranked chunks can then be passed to later retrieval stages, reranking, or directly to the generator. Vector normalization matters If vectors are normalized to unit length: then cosine similarity becomes equivalent to the dot product: This is why some vector search systems can use dot product efficiently when embeddings are already normalized. Cosine similarity does not understand text by itself Cosine similarity only compares vectors. The semantic quality comes from the embedding model that produced those vectors. If the embedding model represents a query and a relevant document poorly, cosine similarity cannot correct that representation. Similarity score is not a universal relevance percentage A cosine similarity score such as 0.82 should not automatically be interpreted as: The score is a geometric similarity value, and useful thresholds vary across models, datasets, and applications. Thresholds should therefore be calibrated using real retrieval examples. Cosine similarity vs other vector metrics Other retrieval systems may use: Dot product similarity Euclidean distance The best metric depends on how the embedding model was trained and how its vectors are normalized. These are covered separately in “What is dot product similarity in vector retrieval?” and “What is Euclidean distance in vector retrieval?” . In short, cosine similarity measures the directional similarity between a query embedding and document embeddings. RAG systems can use it to rank chunks by semantic closeness, with higher scores generally indicating more similar vector representations.

What problem does RAG actually solve?

RAG solves a practical gap: an LLM does not automatically know your latest, private, or organization specific information. A language model answers from patterns learned during training. That creates three common problems: Its knowledge may be outdated. It may not know your internal documents, policies, tickets, or product data. It can still generate a confident answer when the required fact is missing. RAG addresses this by retrieving relevant information first and giving that evidence to the model before it answers. Simple example Suppose an employee asks: “What is our current leave policy?” Without RAG, the model may answer from general knowledge or outdated information. With RAG: Question → retrieve the latest HR policy → pass the relevant section to the LLM → generate the answer from that evidence. The key change is that the model is no longer expected to already know the answer. What problem is RAG really solving? At its core, RAG solves knowledge access at generation time . It lets the model use information that: Was never part of its training data Changed after the model was trained Lives inside private company systems Is too large to put into every prompt Why not just put everything in the prompt? For a small document, you sometimes can. But with thousands or millions of documents, sending everything is too expensive, too slow, may exceed the context window, and can bury useful evidence inside irrelevant text. RAG retrieves only the small amount of information that is likely to matter for the current question. Does RAG eliminate hallucination? No. RAG gives the model better evidence, but the model can still misunderstand, ignore, or misuse that evidence. Think of it as: better access to evidence → better chance of a grounded answer , not a guarantee that every answer will be correct. Interview takeaway RAG solves the gap between what the LLM already knows and the information the application needs right now. It retrieves relevant external knowledge at query time and gives it to the model so answers can be more current, domain specific, and grounded.

How does RAG work end to end?

RAG works in two major phases: prepare the knowledge first, then retrieve the right pieces when a user asks a question. 1. Ingestion: prepare the knowledge Suppose your source is an employee handbook. The ingestion flow is: For example, a 100 page handbook might become hundreds of searchable chunks. 2. User asks a question Example: The system does not immediately send this question to the LLM. First, it tries to find the relevant evidence. 3. Convert the query for retrieval For vector retrieval, the query is converted into an embedding. If the system also uses keyword search such as BM25, the original query can be sent to that retriever as well. 4. Retrieve candidate chunks The retriever searches the knowledge base and returns the most relevant chunks. For example: Retrieval can use: Dense vector search Sparse or keyword search Hybrid retrieval Metadata filters 5. Optional reranking The first retriever is optimized to find good candidates quickly, not necessarily to perfectly order them. A reranker can examine the query and retrieved chunks more carefully and move the strongest evidence to the top. 6. Build the LLM context The selected chunks are placed into the prompt together with the user's question and instructions. Conceptually: A common instruction is to answer from the supplied evidence and say when the answer is not available. 7. Generate the answer The LLM now receives both the question and relevant external knowledge. For example: 8. Return citations or sources A good RAG system often keeps source metadata such as document name, page number, URL, or section so the final answer can point back to the evidence. End to end picture Interview takeaway RAG first converts external knowledge into searchable units. At query time it retrieves the most relevant evidence, optionally reranks it, places that evidence into the prompt, and lets the LLM generate an answer grounded in the retrieved context.

What is an ingestion pipeline?

An ingestion pipeline in RAG is the process that turns raw source data into clean, searchable knowledge that the retrieval system can use later. It runs before users ask questions. Typical flow Raw documents → parse → clean and normalize → chunk → add metadata → create embeddings → store in the retrieval index. Example Suppose you have a 200 page product manual. The ingestion pipeline might: Extract text from the PDF Detect headings and page numbers Extract tables separately Split the manual into meaningful chunks Attach metadata such as product name and section Create an embedding for each chunk Store the chunk, vector, and metadata in a vector database After that, the manual is ready for retrieval. Why ingestion matters Retrieval quality depends heavily on what happened during ingestion. A useful way to think about it is: Good source → bad parsing → broken chunks → bad retrieval. If parsing is poor, chunks are badly split, or metadata is missing, the retriever may fail even if the embedding model is good. What should an ingestion pipeline preserve? Useful information often includes: Document ID Title Section heading Page number Source URL Timestamps Access permissions This metadata can later help with filtering, citations, and source tracing. Is ingestion a one time job? Not always. If documents change frequently, ingestion may run continuously or incrementally so new, updated, and deleted content stays synchronized with the retrieval index. Interview takeaway An ingestion pipeline prepares raw knowledge for retrieval. It parses the source, cleans it, splits it into useful chunks, adds metadata, creates embeddings, and stores everything in a searchable index. Good RAG retrieval starts with good ingestion.

What happens during retrieval?

During retrieval, the RAG system tries to find the smallest set of relevant evidence needed to answer the user's question. It happens after the user submits a query and before the LLM generates the final answer. 1. Start with the user query Example: "What is the refund policy for enterprise customers?" The retrieval pipeline first decides how to search for that information. 2. Apply query processing if needed Depending on the system, the query may be: Used as is Rewritten Expanded with related terms Split into subqueries Combined with metadata filters For example, the system may add: "customer type = enterprise" as a filter. 3. Convert the query into the retriever's format For vector search: Query → embedding model → query vector For keyword or BM25 search: Query → terms/tokens → lexical search For hybrid retrieval, both can run in parallel. 4. Search the knowledge base The retriever searches indexed chunks and returns candidates. Example: Chunk A: Enterprise customers receive a 45 day refund window. Chunk B: Standard customers receive a 30 day refund window. Chunk C: Refunds are processed within 5 business days. 5. Rank the candidates The retriever assigns scores and orders results by relevance. A vector retriever may use cosine similarity or dot product. BM25 ranks based on lexical relevance. Hybrid systems may combine both result lists. 6. Apply metadata filters Filters can restrict retrieval to the right subset of data. Examples: Product = "Enterprise" Region = "India" Document type = "Policy" User has permission to access the document This is especially important in enterprise RAG. 7. Optional reranking The first retrieval stage is usually optimized for speed. A reranker can inspect the query and candidate chunks more carefully. For example: Retrieve Top 20 → rerank → keep best 5 This often improves the quality of the final context. 8. Select the context for the LLM The system keeps the most useful chunks while trying to avoid: Irrelevant text Duplicate chunks Too much context Conflicting evidence The selected chunks are then passed to the LLM with the user's question. Interview takeaway Retrieval is the stage where RAG converts the user's question into a search, finds candidate chunks, filters and ranks them, optionally reranks them, and selects the best evidence to send to the LLM. Retrieval quality largely determines whether the model gets the right facts to answer from.

How do you decide the chunk size?

There is no single perfect chunk size for RAG. You choose chunk size by balancing two things: Small enough that each chunk stays focused on one idea. Large enough that the chunk still contains enough context to understand that idea. What happens if chunks are too small? Suppose the document says: "Refunds are allowed within 30 days, but only for unused products." If this is split badly: Chunk 1: "Refunds are allowed within 30 days." Chunk 2: "But only for unused products." The retriever may return only Chunk 1 and lose the important condition. So very small chunks can improve precision but lose context. What happens if chunks are too large? Imagine one chunk contains five pages covering refunds, shipping, warranty, and billing. A query about refunds may retrieve that chunk, but most of the text is irrelevant. Large chunks can: Reduce retrieval precision Consume more prompt tokens Increase latency and cost Make it harder for the LLM to focus on the right evidence A practical starting point For ordinary text, teams often begin with a few hundred tokens per chunk and then tune from there. But the exact number should depend on the document structure. For example: FAQ answer → keep the whole answer together API documentation → keep one endpoint or operation together Legal contract → preserve clause boundaries Source code → chunk around functions or classes Tables → preserve rows, headers, and related context The semantic unit matters more than hitting an exact token count. What should you evaluate? Test different chunk sizes using real questions and measure whether the correct evidence is retrieved. For example: 200 token chunks → Recall@5 = 82% 500 token chunks → Recall@5 = 91% 1000 token chunks → Recall@5 = 87% In that case, 500 tokens may be the better choice for that dataset. Also watch: Retrieval recall Retrieval precision Number of chunks returned Prompt token usage End to end latency Final answer quality What about chunk overlap? Overlap can help when useful information sits near a chunk boundary. For example: Chunk 1: tokens 1–500 Chunk 2: tokens 451–950 The repeated 50 tokens reduce the chance that an important sentence is split across chunks. But too much overlap creates duplicate content, larger indexes, and repetitive retrieval results. Interview takeaway Choose chunk size by preserving meaningful context while keeping each chunk focused enough for retrieval. Start with a reasonable baseline, then evaluate multiple sizes on real queries. The correct chunk size is the one that gives the best retrieval and answer quality for your data—not a universal fixed number.

What are the different chunking methods?

There are several ways to split documents into chunks for RAG. The right method depends on the structure of the content and what users are likely to ask. 1. Fixed size chunking Split text every N characters or tokens. Example: Chunk 1 → tokens 1–500 Chunk 2 → tokens 501–1000 It is simple and fast, but it can cut through sentences or ideas. Use it when the data is fairly uniform and you need a simple baseline. 2. Recursive chunking Try larger natural separators first, then smaller ones only when needed. For example: Document → section → paragraph → sentence This usually preserves structure better than blindly cutting every fixed number of tokens. It is a strong general purpose default for normal text. 3. Sentence or paragraph chunking Split using natural language boundaries. For example: Paragraph 1 → one chunk Paragraph 2 → another chunk This keeps text readable, but chunk sizes can vary a lot. It works well when paragraphs already represent clear ideas. 4. Structure based chunking Use the document's own structure. Examples: Markdown headings HTML sections PDF headings API endpoints Legal clauses Code functions or classes For API documentation, keeping one endpoint with its parameters and response description is often much better than splitting it in the middle. 5. Semantic chunking Split when the topic or meaning changes. For example, a long paragraph may discuss: Refund policy → shipping policy → warranty policy Semantic chunking tries to detect those topic shifts and create separate chunks. It can produce more meaningful retrieval units, but it is usually more expensive and complex than fixed or recursive chunking. 6. Parent child chunking Create small chunks for retrieval but keep a larger parent section for context. Example: Small child chunk → easy to match precisely Parent section → enough context for the LLM This is useful when small chunks retrieve well but are not sufficient to answer by themselves. 7. Hierarchical chunking Represent content at multiple levels. For example: Document → chapter → section → paragraph Retrieval may first locate the relevant section and then search within it. This can work well for very large or strongly structured documents. 8. Content specific chunking Some content should be chunked according to its type instead of using generic text rules. Examples: Code → functions/classes Tables → header + related rows FAQs → question + answer Contracts → clauses API docs → endpoint + request/response details Which method should you choose? A practical approach is: Simple text → recursive or paragraph based chunking Highly structured documents → structure based chunking Long documents with clear hierarchy → hierarchical or parent child chunking Meaning shifts that do not follow structure → semantic chunking The choice should ultimately be tested using real retrieval queries. Interview takeaway The main chunking methods are fixed size, recursive, sentence/paragraph, structure based, semantic, parent child, hierarchical, and content specific chunking. The goal is not just to create equally sized pieces; it is to create retrieval units that preserve enough meaning for the retriever and the LLM.

Fixed-size chunking vs semantic chunking — when would you use which one?

Fixed size chunking and semantic chunking solve the same problem—splitting documents into retrievable pieces—but they optimize for different things. Fixed size chunking Fixed size chunking splits text after a fixed number of tokens or characters. Example: Chunk 1 → first 500 tokens Chunk 2 → next 500 tokens Chunk 3 → next 500 tokens It is: Simple Fast Predictable Easy to scale But it can cut through sentences, sections, or ideas. When should you use it? Use fixed size chunking when: Your documents are fairly uniform You need a fast baseline You have a very large corpus Ingestion cost matters Document structure is weak or inconsistent It is often a good first version of a RAG pipeline because it is easy to implement and evaluate. Semantic chunking Semantic chunking tries to split text when the meaning or topic changes. Example: A long section discusses: Refunds → Shipping → Warranty Instead of splitting every 500 tokens, semantic chunking tries to create: Chunk 1: Refund policy Chunk 2: Shipping policy Chunk 3: Warranty policy The chunks may have different sizes, but each one is more coherent. When should you use it? Use semantic chunking when: Topic boundaries matter Documents contain several ideas inside long sections Fixed size chunks frequently mix unrelated topics Retrieval quality is more important than ingestion simplicity You can afford more preprocessing The main trade off Fixed size chunking optimizes for operational simplicity. Semantic chunking optimizes for meaning. Think of it like this: Fixed size → "split every N tokens" Semantic → "split when the topic changes" Example Suppose a 900 token section contains: 250 tokens about refunds 300 tokens about shipping 350 tokens about warranty With 500 token fixed chunks, one chunk may contain both refunds and shipping. A query about shipping could retrieve extra unrelated refund text. Semantic chunking may separate those topics cleanly, making retrieval more precise. Is semantic chunking always better? No. It can be: More computationally expensive Slower during ingestion Harder to tune Less predictable in chunk size And if the document is already well structured, semantic chunking may add little value. For example, API documentation with one endpoint per section may be better handled with structure based chunking than semantic chunking. Practical approach A good strategy is: 1. Start with recursive or fixed size chunking. 2. Measure retrieval quality. 3. Inspect failure cases. 4. Move to semantic chunking only where topic mixing is hurting retrieval. Do not add semantic chunking just because it sounds more advanced. Interview takeaway Use fixed size chunking when you want a simple, fast, scalable baseline and the content is fairly uniform. Use semantic chunking when preserving topic boundaries materially improves retrieval quality. Semantic chunking can produce cleaner retrieval units, but it costs more and is not automatically better for every dataset.

How do you handle PDFs containing both text and images?

When a PDF contains both text and images, do not treat the whole file as plain text. A good RAG pipeline processes the text and the visual content separately, then keeps the relationship between them. 1. Extract the normal text Start by parsing the PDF text layer. Keep useful structure such as: Page number Heading Section Paragraph order Figure references This text can be chunked and embedded normally. 2. Detect and extract important images Images may contain information that does not exist in the text layer. Examples: Architecture diagrams Screenshots Flowcharts Product images Infographics Extract these images together with their page number and surrounding text. 3. Turn the image into something searchable There are two common approaches. If the image mainly contains text, use OCR to extract that text. If the image contains visual meaning, use a vision model to create a description or structured summary. For example, an architecture diagram might become: "Diagram showing API Gateway sending requests to three microservices, which write to PostgreSQL and Redis." That description can then be embedded and searched like normal text. 4. Preserve the connection between text and image This part is important. Do not store an image description without knowing where it came from. Keep metadata such as: Document ID Page number Figure number Image path or object ID Nearby heading Related text chunk For example: Text chunk: "The system architecture is shown in Figure 4." Figure 4: architecture image Image description: "API Gateway routes traffic to authentication, payment, and order services." All three should remain linked. 5. Retrieve the right modality If the user asks: "Which service connects to Redis in the architecture?" The answer may depend on the diagram rather than the surrounding paragraph. The retriever can search both: Text chunks Image descriptions or multimodal embeddings Then the RAG system can pass the relevant text and, when supported, the original image to a multimodal LLM. 6. Do not flatten everything into one text blob A common mistake is: PDF → extract all text → ignore images That works only when the images are decorative. If important information appears only in diagrams or screenshots, the RAG system will never retrieve it. Practical pipeline PDF → layout aware parsing Then: Text → chunk → embed → index Images → extract → OCR or vision description → embed → index Both should share metadata so retrieval can reconstruct the original context. Interview takeaway For PDFs containing both text and images, use layout aware parsing, process text and images separately, convert visual information into searchable text or multimodal embeddings, and preserve page and figure relationships. At retrieval time, return the relevant text, image, or both instead of assuming the PDF is only text.