Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
How do you perform offline evaluation of a RAG system?
Offline RAG evaluation tests a retrieval augmented generation system against a fixed, labeled dataset before deploying changes to live users. It lets teams compare versions of retrievers, prompts, rerankers, and language models reproducibly. Step by step workflow 1. Create a representative test set. Include real or expert written questions, trusted answers, relevant source IDs, unanswerable questions, and multi document cases. Keep sensitive user data out of the benchmark. 2. Freeze the evaluation conditions. Record the document corpus version, index snapshot, access permissions, model version, prompts, retrieval settings, and evaluation code. 3. Run the complete RAG pipeline. For each question, log retrieved document IDs and ranks, the assembled context, generated answer, citations, latency, and token usage. 4. Score retrieval separately. Compare retrieved items against relevance labels using Recall@K, Precision@K, MRR, or nDCG, depending on the goal. 5. Score generation separately. Evaluate answer correctness, faithfulness to retrieved context, answer relevance, citation support, and appropriate abstention. 6. Compare with a baseline. Use the same questions and evaluation conditions for both versions; examine per question regressions, not just aggregate averages. 7. Review failures and gate release. Manually inspect important failures and decide whether the candidate meets predefined quality, latency, cost, and safety requirements. Beginner friendly example Imagine a support assistant with 100 labeled customer questions . You are deciding whether to replace its retriever. Current version Finds the required evidence in the top 5 results for 78/100 questions. Produces fully supported answers for 70/100 questions. Candidate version Finds the required evidence in the top 5 for 87/100 questions. Produces fully supported answers for 76/100 questions. The candidate improves both observed measures on this benchmark. However, inspect the questions that became worse and check whether latency or cost increased before deployment. These are illustrative rates; “required evidence found” and “fully supported answer” must be defined consistently. A small sample may not establish that the improvement generalizes. Useful offline metrics Retrieval: Recall@K, nDCG@K, evidence set coverage. Generation: Answer correctness, groundedness/faithfulness, relevance, citation validity. Behavior: Abstention accuracy on unanswerable questions. Operations: p50/p95 latency, token usage, estimated cost, errors. Common pitfalls Data leakage: Tuning on the same test questions can inflate scores. Stale labels: Updated documents may invalidate expected answers. Weak automated judges: LLM based ratings can disagree with domain experts; calibrate on human reviewed samples. Changing multiple variables: It becomes hard to attribute improvements to one component. Ignoring slices: Aggregate scores can hide failures for specific languages, regions, tenants, or question types. Offline versus online evaluation Offline evaluation uses a controlled dataset and does not expose live users to an untested candidate. Online evaluation measures real production traffic and user outcomes, typically after offline checks pass. Interview takeaway Offline RAG evaluation is a reproducible regression test for retrieval, generation, and operational behavior. Freeze the test conditions, compare against a baseline, and inspect failure cases before rollout.
How do you perform online evaluation of a RAG system?
Online evaluation of a RAG system measures how well retrieval and generated answers perform with real user traffic after deployment . Unlike offline evaluation on a fixed test set, it captures changing questions, content, latency, and actual user behavior. Practical workflow 1. Set a baseline: Record current answer quality, retrieval success, latency, cost, and user outcomes before releasing a change. 2. Instrument every request safely: Log a request ID, retriever/model/prompt version, retrieved document IDs and scores, response outcome, latency, and explicit feedback where permitted. Redact sensitive data and apply access controls. 3. Release gradually: Use a small canary cohort or randomized A/B test to compare a new pipeline with the current version. Keep user groups consistent and protect against experiments mixing incompatible knowledge base versions. 4. Measure quality and behavior: Track thumbs up/down or expert review, task completion or escalation rate, groundedness on sampled responses, unanswerable question handling, citation usefulness, and answer relevance. 5. Track operational guardrails: Monitor retrieval failures, timeouts, p95 latency, token cost, error rate, and safety incidents. A high feedback score does not justify a regression in reliability. 6. Analyze by segment: Compare new vs recurring queries, languages, departments, document freshness, and answerable vs unanswerable questions. 7. Roll back and learn: Investigate statistically meaningful regressions and serious failures; roll back when preset guardrails are breached. Add discovered failures to the offline regression dataset. Beginner friendly example An internal support assistant changes from basic vector retrieval to hybrid retrieval plus reranking. Control A: Existing vector only RAG, serving most users. Variant B: New hybrid plus reranker RAG, serving a small randomized cohort. Compare: Human reviewed answer relevance and faithfulness, successful ticket deflection, feedback, p95 response latency, and cost per answer. Decision: Expand B only if it improves useful answers without unacceptable latency, cost, or reliability regressions. A higher click or thumbs up rate alone does not prove factual accuracy. Check a sample against retrieved evidence. Online vs offline evaluation Offline: Repeatable, labeled questions; catches regressions before release. Online: Real world behavior and distribution shifts; reveals failures the test set missed. Best practice: Use both. Online failures should become new offline test cases. Common pitfalls Treating implicit engagement as evidence that an answer is correct. Comparing traffic from different time periods without accounting for changing query mix. Making decisions on tiny samples or looking only at averages. Logging confidential content without authorization. Shipping a model improvement that silently worsens citation accuracy or latency. Interview takeaway: Online RAG evaluation is a controlled, privacy aware production measurement loop: instrument → experiment → monitor quality and guardrails → investigate → improve .
What production metrics should be monitored for RAG quality?
Production RAG quality monitoring means tracking whether a deployed retrieval augmented generation system continues to find appropriate evidence, produce useful grounded answers, and operate reliably as documents and user queries change. What should you monitor? 1. Retrieval quality Retrieval success / empty result rate: How often searches return usable passages versus nothing. Context relevance and precision: Whether retrieved chunks actually help answer the question; estimate using sampled human or calibrated LLM judgments. Coverage / context recall: Whether required evidence is missing, using annotated samples rather than assuming every query has known ground truth. Source freshness and access filter correctness: Whether retrieved documents are current and authorized for the requester. 2. Generated answer quality Faithfulness / groundedness: Fraction of factual claims supported by the retrieved passages. Answer relevance and completeness: Whether the response addresses the user's intent and all requested subquestions. Citation correctness: Whether cited passages exist and genuinely support the associated claims. Abstention quality: Whether the assistant appropriately says “I don't know” when evidence is insufficient, without refusing answerable queries. Task outcomes: Human confirmed resolution, successful self service, or escalation, interpreted carefully. 3. Reliability and efficiency End to end and stage latency: p50/p95/p99 for retrieval, reranking, and generation. Timeouts, errors, and fallback rates: Broken vector searches, unavailable sources, model errors, and retries. Tokens and cost per request: Retrieval/context tokens, generated tokens, and per answer spend. Ingestion/index health: Document update lag, failed indexing jobs, missing or stale embeddings. Safety and privacy incidents: Data leakage, prompt injection, and unauthorized source exposure. Beginner friendly example A company policy RAG assistant handles 1,000 questions in a day. Its dashboard reports: 80 searches returned no candidate passages → 8% empty result rate . 50 sampled answers were reviewed; 44 were grounded → 88% sampled groundedness . The p95 end to end response time was 5 seconds . Newly updated policies took 6 hours to become searchable. These numbers are illustrative. They are not universal acceptance targets : establish baselines, segment results, and set thresholds based on actual user risk and service requirements. How do you build a useful dashboard? 1. Instrument each request with a safe trace ID, pipeline version, retrieval timings, document IDs, and outcome signals. 2. Review a representative sample with human raters or validated automated judges; record sample size and uncertainty. 3. Segment by question type, language, source, user cohort, and answerability. A single aggregate score can hide severe failures. 4. Plot trends against a pre change baseline and alert on sustained regressions, not one off fluctuations. 5. Investigate cause: poor retrieval, missing indexed content, contradictory context, generation failure, or a system outage. 6. Add recurring production failures to an offline evaluation dataset and re test before the next release. Interview takeaway: Monitor retrieval quality + answer faithfulness/relevance + citation integrity + abstention + user outcomes , alongside latency, failures, cost, freshness, and security . No single metric measures RAG quality on its own.
What is Graph RAG?
Graph RAG (graph based retrieval augmented generation) is a RAG approach that uses a knowledge graph —entities as nodes and their relationships as edges—to find and organize evidence before an LLM answers a question. Instead of depending only on similarity between a question and isolated text chunks, it can use the connections among concepts, people, systems, and events. Beginner friendly example Imagine company documents describe: Asha manages Project Orion . Project Orion depends on Service Delta . Service Delta is maintained by Team Platform . Question: “Which team maintains the service that Asha's project depends on?” A graph represents the chain: Asha → manages → Project Orion → depends on → Service Delta → maintained by → Team Platform A graph aware retriever can traverse these relationships and fetch the underlying document passages. The LLM can then answer “Team Platform” , citing the documents that substantiate the links. The graph does not itself guarantee those extracted relationships are correct. How does Graph RAG work? 1. Ingest documents: Parse and split the corpus into traceable text units. 2. Extract structure: Identify entities, relationships, and sometimes claims; store their links with provenance back to source passages. 3. Build the graph: Normalize entity names, connect relationships, and optionally group related entities into communities. 4. Retrieve for the question: Search relevant entities, follow appropriate graph connections, and/or retrieve community summaries together with supporting text. 5. Generate with evidence: Pass the selected source context to the LLM and require a grounded, cited response. The term Graph RAG describes a family of designs; graph traversal, graph enhanced vector search, and community summarization need not all appear in every implementation. Graph RAG vs traditional vector RAG Vector RAG: Finds text chunks with high semantic similarity. Often simpler and effective for direct fact lookup. Graph RAG: Adds explicit relationships and structure. This may help multi hop questions, entity centered exploration, or cross document themes. Hybrid designs: Combine graph context with vector search to preserve semantic recall and relationship aware retrieval. Microsoft's GraphRAG project is a specific implementation that builds an LLM derived knowledge graph, detects communities, and produces community summaries. Its local search focuses on entities and supporting text; global search draws on community reports for corpus wide themes. Those details describe Microsoft's implementation rather than requirements for every graph based RAG architecture. When is it useful—and what are the trade offs? Graph RAG is attractive when answers depend on multiple related facts spread across documents, such as tracing product dependencies or organizational relationships. It can be unnecessary for straightforward lookups. Entity extraction, graph construction, updates, and graph traversal add cost and complexity; incorrect relationships can mislead answers. Evaluate graph based retrieval against a strong vector or hybrid baseline on your own questions before adopting it. Interview takeaway Graph RAG augments generation with relationships among retrieved facts , helping systems answer connected or multi document questions while still requiring reliable evidence, citations, and evaluation. Further reading: Microsoft GraphRAG documentation: https://microsoft.github.io/graphrag/ and query overview: https://microsoft.github.io/graphrag/query/overview/.
When is graph retrieval useful in RAG?
Graph retrieval is useful in RAG when answering a question requires following explicit relationships across entities or synthesizing connected evidence from multiple documents. It is particularly valuable for questions that simple similarity search might answer only partially. Beginner friendly example Three internal documents say: Maya owns Checkout API . Checkout API depends on Payments Service . Payments Service is operated by Platform Team . Question: “Which team operates the service that Maya's API depends on?” A relationship aware search can follow: Maya → owns → Checkout API → depends on → Payments Service → operated by → Platform Team It then retrieves the underlying source passages for each edge. The supported answer is Platform Team , assuming those records are current and consistent. When should you choose graph retrieval? 1. Multi hop questions: The answer spans two or more entity relationships (person → project → dependency → owner). 2. Relationship centric questions: Users ask who owns, depends on, reports to, influences, or connects to something. 3. Cross document synthesis: Relevant facts are scattered across reports rather than colocated in a single chunk. 4. Exploratory discovery: Users need to find neighboring entities, trace dependencies, or discover paths between systems. 5. Corpus wide themes: Some GraphRAG approaches organize graph communities and summarize them to answer broad questions like “What themes recur across these incident reports?” This is different from simply traversing a few edges. When might it not help? Direct fact lookup: “What is the refund deadline?” may be answered more cheaply with conventional or hybrid text retrieval. Weak entity structure: Vague or rapidly changing documents may produce incomplete, noisy graphs. Strict freshness needs: A stale graph can misrepresent current ownership or dependencies. Tight cost/latency constraints: Entity extraction, graph maintenance, and multi step search add work. Practical selection checklist Do realistic user questions require relationships or multiple reasoning hops? Can key entities and edges be extracted accurately and linked back to source evidence? Is entity resolution good enough to avoid merging unrelated people or services? Will graph updates keep pace with source changes? Does graph or graph plus vector retrieval measurably improve grounded answer accuracy over a strong hybrid search baseline? Implementation tip: Start with vector or keyword retrieval for candidate entities, expand a bounded neighborhood or path in the graph, retrieve supporting source passages, then rerank and generate with citations. Validate each edge; a plausible graph path is not proof. Graph search modes in practice Microsoft's GraphRAG documentation distinguishes local search for specific entities and their associated evidence from global search over community reports for dataset wide questions. These are implementation specific patterns, not universal requirements of graph retrieval. Interview takeaway Choose graph retrieval when relationships are essential to answering accurately —especially multi hop dependencies and cross document questions. Benchmark it against simpler retrieval and account for extraction quality, freshness, cost, and evidence provenance. References: https://microsoft.github.io/graphrag/query/overview/ and https://microsoft.github.io/graphrag/query/global search/
What is entity resolution in graph-based RAG?
Entity resolution in graph based RAG is the process of deciding whether different names, mentions, or extracted graph nodes refer to the same real world entity . It helps build a connected, accurate knowledge graph instead of fragmenting one person, product, or organization into duplicates. Beginner friendly example Three documents mention: “International Business Machines” released a product. “IBM” maintains the product. “IBM Corp.” operates its support portal. If all three mentions refer to the same organization, entity resolution maps them to one canonical organization node , with the original names retained as aliases and links to their source documents. But “Apple” the technology company and “apple” the fruit must not be merged. Likewise, two different employees named “Ravi Kumar” may need separate nodes. Entity extraction vs entity resolution Entity extraction identifies potential names and things in text. Entity linking connects a mention to a known entity identifier in a catalog or knowledge base. Entity resolution decides which records or nodes represent the same entity, including cases without an existing catalog. They often work together but solve different parts of graph construction. How does it work? 1. Extract and normalize mentions: Standardize capitalization, punctuation, common abbreviations, and entity types without discarding meaningful distinctions. 2. Generate candidates: Use exact or fuzzy name matches, known aliases, external identifiers, embeddings, or blocking rules to find possible duplicates. 3. Compare context: Check descriptions, organization, location, domain, relationships, dates, and source documents. 4. Resolve cautiously: Merge only high confidence matches; retain ambiguous entities separately or send them for review. 5. Preserve provenance: Keep aliases, original mentions, source passages, stable IDs, and the reason for each merge. 6. Maintain the graph: Recheck merges when updated documents or new evidence reveal previously hidden differences. Numerical example Suppose 100 extracted mentions include “Acme Ltd” (42 mentions) , “ACME Limited” (35) , and “Acme” (23) . If evidence confirms that all refer to the same company, the graph can connect 100 mentions to one canonical entity . The count of graph nodes decreases, but the 100 original mentions and citations remain available. If “Acme” also names a different supplier, blindly merging all 23 ambiguous mentions would contaminate retrieval. Context and identifiers matter more than string similarity alone. Why does this improve RAG? Without resolution, relationship paths may break because the same company appears as separate nodes. With incorrect over merging, unrelated facts can mix and create misleading answers. Accurate resolution supports multi hop retrieval, entity centered search, and trustworthy source tracing. Common pitfalls False merges: Two distinct entities are combined, leaking unrelated relationships into results. Missed merges: Aliases remain separate, hiding valid graph connections. Name changes over time: Mergers, rebranding, or employee role changes need temporal context. Unexplained merges: Without provenance or confidence records, it is hard to audit and correct mistakes. Interview takeaway: Entity resolution makes graph RAG connect mentions that truly refer to the same thing while keeping different things separate . Treat it as an evidence based identity decision, not just lowercase string deduplication. Further reading: https://neo4j.com/blog/developer/entity resolved knowledge graphs/ and https://microsoft.github.io/graphrag/index/default dataflow/.
What is graph expansion in RAG?
Graph expansion in RAG is the retrieval step that starts from one or more relevant entities (called seed nodes ) and follows their graph relationships to collect additional, connected evidence. It broadens retrieval from a directly matched entity to useful neighbors without searching the entire graph. Beginner friendly example A company's knowledge graph stores these relationships: Checkout API → depends on → Payment Service Payment Service → owned by → Platform Team Payment Service → uses → Fraud Detector A user asks: “Who owns the service that Checkout API depends on?” 1. Seed: Retrieve the node Checkout API . 2. One hop expansion: Follow depends on to Payment Service . 3. Two hop expansion: Follow owned by from Payment Service to Platform Team . 4. Ground the answer: Fetch the source documents for both edges, then answer “Platform Team” if the evidence confirms it. The uses → Fraud Detector edge may be real but is irrelevant to this question, so the retriever should exclude it. How is expansion controlled? Starting entities: Identify seeds using entity names, full text search, vector similarity, or an entity linker. Hop limit: Restrict traversal depth—for example, 2 hops—so the search does not expand indefinitely. Relationship filters: Follow only types relevant to the question, such as depends on and owned by. Direction: Respect directed edges where meaning depends on direction. Candidate ranking: Prioritize important or question relevant paths, then deduplicate overlapping nodes and edges. Budgets and permissions: Limit number of nodes, edges, runtime, and LLM context tokens; enforce access restrictions for every traversed item. Evidence links: Recover source text supporting each selected relationship, rather than asking the model to trust graph edges alone. Small numerical example Suppose a seed has 5 neighbors , and each of those has 4 new neighbors . At 1 hop , the search may consider 5 additional nodes. At 2 hops , it may consider 5 + (5 × 4) = 25 additional nodes, assuming no shared nodes or cycles. This rapid growth is called neighborhood explosion . Real graphs have overlaps, and pruning is essential for keeping retrieval focused and affordable. Graph expansion vs query expansion Graph expansion: Follow entity relationships to fetch connected facts. Query expansion: Add alternative words or search queries to discover text that uses different wording. Graph expansion also differs from entity resolution , which determines whether multiple names refer to the same entity. When does it help? Use graph expansion for multi hop dependencies, ownership chains, connected incidents, and questions whose evidence spans documents . Avoid large or indiscriminate expansions for simple factual lookups; they add latency and irrelevant context. Benchmark answer quality and cost against simpler retrieval. Interview takeaway: Graph expansion means start with relevant nodes → traverse selected edges for a bounded number of hops → rank and verify evidence → generate a grounded answer . References: https://neo4j.com/labs/genai ecosystem/graphrag/ and https://microsoft.github.io/graphrag/examples notebooks/local search/
What is multi-source RAG?
Multi source RAG (multi source retrieval augmented generation) answers questions using evidence retrieved from more than one knowledge source , such as documents, databases, internal wikis, ticket systems, and approved external resources. It focuses on coordinating evidence across sources rather than merely listing the types of data RAG can ingest. Beginner friendly example A user asks: “Why is customer order 742 delayed, and what is the current shipping policy?” An orders database reports that order 742 is awaiting fulfillment. A support ticket system records a warehouse stock issue for that order. A policy document store says delayed orders qualify for a status update within two business days. A multi source RAG pipeline retrieves the relevant records from all three authorized systems, reconciles their timestamps, and generates a response citing each claim's source. The sources do not automatically agree or have equal authority. Typical architecture 1. Route or fan out the query: Identify which knowledge sources are relevant; a router can avoid unnecessary calls, or the retriever can query multiple sources in parallel. 2. Use source specific connectors: SQL for structured records, keyword/vector retrieval for documents, API search for tickets, and graph traversal when relationships matter. 3. Normalize retrieved evidence: Attach a stable source ID, document or record ID, timestamp, permissions, excerpt, and retrieval score to each result. 4. Enforce authorization: Apply user specific access checks in the originating system and again when building the answer context. Never let a vector match bypass row level or document permissions. 5. Rank and deduplicate: Merge semantically overlapping findings, account for different score scales, and prioritize relevance, authority, and freshness. 6. Resolve discrepancies: Distinguish live operational data from older documentation. When authoritative sources conflict and no clear precedence exists, explain the disagreement rather than inventing a resolution. 7. Generate with provenance: Include only relevant evidence in the prompt and cite source records so answers can be audited. Numerical example Suppose a query retrieves 5 policy passages , 4 support ticket records , and 3 database rows — 12 candidates . After permission checks, deduplication, and relevance ranking, the system retains 6 distinct authorized pieces of evidence for generation. Fewer, better selected items can reduce context cost and prevent contradictory noise. Why use it? Multi source RAG is useful when questions span operational status + policy , product specs + support history , or multiple departments . It provides broader evidence coverage without copying every system into one central index. Challenges and trade offs Data freshness: A live database and a nightly document index may represent different points in time. Conflicting records: Establish source authority rules and surface unresolved differences. Identity matching: The same customer, incident, or service may have different identifiers across systems. Latency and failures: Slow or unavailable sources require timeouts, fallbacks, and clear partial answer handling. Security: Respect each system's entitlements and avoid exposing sensitive records. Evaluation: Test source coverage, routing accuracy, attribution, groundedness, and partial outage behavior—not just final answer fluency. Multi source vs multimodal RAG: Multi source refers to multiple repositories or systems; multimodal refers to formats such as text, image, and audio. A solution can be both. Interview takeaway: Multi source RAG coordinates retrieval across separate knowledge systems, normalizes and verifies evidence, enforces permissions, and synthesizes a traceable answer. Its key difficulty is reconciling diverse sources reliably , not simply querying several indexes.
How do you design RAG across multiple databases or knowledge sources?
Designing RAG across multiple databases or knowledge sources requires an orchestration layer that sends each user question to the right authorized systems, converts their results into a consistent evidence format, and assembles a traceable answer. This question focuses on system design and implementation choices , building on the concept of multi source RAG. Reference architecture Step by step design 1. Inventory sources and ownership. Document each system's schema, supported search API, update frequency, authorization model, rate limits, and authoritative fields. For example, order status belongs to the transactional database, not an old FAQ. 2. Choose live access or indexing per source. Query frequently changing structured records directly through controlled read only APIs or parameterized SQL; index longer, less volatile documents into searchable chunks. Use incremental sync, deletion handling, and version timestamps for indexed data. 3. Build source adapters. Each connector accepts a constrained request and returns {source, record id, text, retrieved at, updated at, access scope, citation url} plus scores where available. Avoid exposing unrestricted SQL generation or backend credentials to the model. 4. Plan and route queries. A lightweight rule or classifier can choose one source for simple questions; decompose cross system questions into subqueries and run independent retrievals concurrently when needed. 5. Enforce access controls at retrieval. Carry the user's identity or scoped service authorization into each source; apply row level, document level, and tenant filters before ranking or sending content to the LLM. 6. Unify evidence carefully. Normalize timestamps, units, and identifiers; map customer or product IDs across systems; deduplicate repeated passages. Do not assume vector similarity and SQL confidence are directly comparable scores. 7. Apply precedence and conflict policy. Prefer the designated system of record for each field and the most relevant current version. If two authoritative sources disagree, show the discrepancy and its provenance rather than silently choosing one. 8. Construct a bounded context. Rerank for the question, reserve token budgets by subquestion, and include source labels, dates, and citation identifiers. The LLM must distinguish verified facts from unavailable data. 9. Harden operations. Set per source timeouts, retries with limits, circuit breakers, cache policies, and clear partial answer behavior. Trace source latency, routing choices, permission failures, stale indexes, and citation quality. 10. Evaluate end to end. Test single source and cross source questions, missing sources, conflicting documents, inaccessible records, freshness, groundedness, answer relevance, and latency/cost. Beginner friendly worked example Question: “Can customer C42 receive a refund for order O17?” Orders DB (live): O17 was delivered 20 days ago . Returns policy index: Standard refund window is 30 days after delivery for eligible purchases. Support system: O17 has no special exception recorded. The system joins evidence using order/customer IDs and answers: “The order is within the standard 30 day window, but eligibility still depends on the policy's item conditions.” Cite the policy and live order timestamp. Do not assert final approval if item eligibility is unknown. Architecture trade offs Centralized index: Low latency unified ranking, but duplication, synchronization, and security trimming become harder. Federated live retrieval: Fresher information and native permissions, but variable latency and heterogeneous result ranking. Hybrid: Often practical—index static documents while querying transaction systems live. Interview takeaway A production grade multi source RAG design is a secure retrieval federation with routing, source aware adapters, evidence reconciliation, citations, freshness controls, and fallbacks. Selecting a central index versus live federation should follow data volatility, permissions, latency, and governance constraints. Further reading: https://learn.microsoft.com/en us/azure/search/retrieval augmented generation overview and https://docs.aws.amazon.com/prescriptive guidance/latest/patterns/develop advanced generative ai chat based assistants by using rag and react prompting.html
What is multimodal RAG?
Multimodal RAG (retrieval augmented generation) retrieves and uses evidence from more than one type of content , such as text, images, diagrams, audio, or video, to help generate an answer. Traditional text only RAG may miss important facts shown in a chart or screenshot. Beginner friendly example A maintenance manual has: Text: “Check the pressure valve before restarting the pump.” Diagram: A labeled image showing where the pressure valve is located . Question: “Where is the valve, and what should I do before restarting?” A multimodal RAG system can retrieve both the diagram and the written instruction. A vision capable model can inspect the image and answer using the location shown in the figure and the safety instruction from the text, with page or figure citations. How does multimodal RAG work? 1. Ingest content: Extract document text, page images, figures, diagrams, audio transcripts, and other relevant assets. Preserve document, page, and timestamp identifiers. 2. Represent the modalities: Depending on the design, produce searchable image captions, OCR text, transcripts, text embeddings, or multimodal embeddings in a compatible vector space. Different modalities do not automatically share a comparable embedding space. 3. Retrieve evidence: Search relevant text and visual/audio items together, or search separate indexes and merge their results. 4. Align context: Preserve which caption, image, table, or transcript belongs to each source page or time segment. 5. Generate: Supply retrieved text and either original media to a capable multimodal model or trustworthy extracted descriptions to a text model. Request evidence grounded answers and citations. 6. Evaluate: Check retrieval quality separately for each modality and verify whether generated claims are supported by the original source, especially visual details. Common implementation patterns Text first approach: Convert visuals or speech into OCR, captions, or transcripts and search the resulting text. Simpler but fine visual details may be lost. Native multimodal retrieval: Use compatible image–text embeddings to find relevant visual and text evidence; return the actual image for inspection where supported. Hybrid approach: Combine lexical search, text embeddings, multimodal embeddings, and reranking to improve coverage. Multimodal RAG is not the same as multi source RAG. Multimodal means different data types ; multi source means different repositories or systems . A single PDF can be multimodal, while five text only databases are multi source. Small numerical example A 12 page troubleshooting guide contains 9 text heavy pages and 3 diagram heavy pages . A text only pipeline may index just the text, missing diagram details. A multimodal system can index all 12 pages with text and visual representations, then retrieve a diagram plus its explanatory paragraph when both are relevant. Indexing more content does not guarantee correct retrieval; it must be evaluated. When is it useful? Useful for product catalogs, engineering drawings, dashboards, scanned forms, medical or scientific figures, video tutorials, and other content where the answer depends on visual or auditory evidence. Trade offs: Additional extraction and indexing cost, vision model latency, modality alignment errors, poor OCR or captions, and complicated evaluation. Store source references and respect document level permissions. Interview takeaway: Multimodal RAG expands retrieval beyond text while keeping the core principle unchanged: retrieve relevant evidence → ground the response → cite and evaluate the sources . Further reading: Microsoft Azure AI Search multimodal overview: https://learn.microsoft.com/en us/azure/search/multimodal search overview ; NVIDIA multimodal retrieval: https://docs.nvidia.com/rag/latest/multimodal retriever.html .