Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
How do you design a document ingestion pipeline for RAG?
Designing a document ingestion pipeline for RAG means creating a reliable process that turns raw documents into searchable knowledge while preserving the information needed for accurate retrieval later. A good design should optimize for both content quality and traceability . Core stages 1. Collect and identify the source Record where each document came from and assign a stable document identifier. Useful source metadata may include: File name or URL Document type Version Last updated time Access permissions 2. Parse the document Extract the content while preserving useful structure such as: Headings Paragraphs Lists Tables Page numbers Section boundaries Different formats may require different parsers. 3. Clean and normalize Remove noise that does not help retrieval, such as duplicated headers or broken whitespace, while avoiding changes that destroy meaning or document structure. 4. Preserve metadata and hierarchy Keep enough context to trace every chunk back to its source. For example: 5. Split content into retrievable units Chunk the document at boundaries that preserve meaning. The best strategy depends on the document structure and how users are expected to query it. 6. Create retrieval representations Prepare each chunk for the retrieval methods the system will use, such as: Embeddings for semantic search Text fields for keyword search Metadata fields for filtering 7. Index and store Store the chunks, metadata, and retrieval representations in the appropriate search or database system. 8. Validate the ingestion result Before considering ingestion complete, verify that: Important content was not lost Chunks can be traced to their source Metadata is correct Duplicate or outdated content is handled Representative queries retrieve useful passages Design principle The ingestion pipeline should be repeatable and update aware . Reprocessing the same document should not silently create uncontrolled duplicates, and updated versions should be identifiable. The correct design also depends on the source type. PDFs, scanned documents, spreadsheets, web pages, and presentations may require different extraction logic before they reach a common indexing stage. For the definition of the overall process, see “What is an ingestion pipeline in RAG?” . The exact preparation steps before indexing are covered in “What steps should happen before documents are indexed in a RAG system?” . In short, design a RAG ingestion pipeline as a controlled sequence that identifies, parses, cleans, structures, chunks, enriches, indexes, and validates documents while preserving enough metadata to support reliable retrieval and source traceability.
What steps should happen before documents are indexed in a RAG system?
Before documents are indexed in a RAG system, they should be validated, parsed, cleaned, structured, and divided into retrieval ready units . Indexing poor quality input usually produces poor retrieval later, so most of the important preparation happens before the data reaches the search index. 1. Validate the source Confirm that the document can be read and that it is the correct version. Useful checks include: File type File integrity Source location Version or last updated time Access permissions 2. Parse and extract the content Extract the useful information from the document while preserving important structure such as: Headings Paragraphs Lists Tables Page numbers Section boundaries Scanned documents may require OCR before usable text can be extracted. 3. Clean and normalize the extracted content Remove noise such as broken whitespace, duplicated headers, or extraction artifacts without deleting information that affects meaning. 4. Preserve metadata and hierarchy Keep information that helps retrieval and traceability, for example: Document ID File name or source URL Page number Section title Parent section Version Permissions or access attributes 5. Detect duplicates and document versions Identify repeated documents or older versions so the index does not silently accumulate conflicting copies of the same content. 6. Split the content into chunks Large documents should be divided into retrievable units while preserving enough surrounding context to keep each chunk meaningful. 7. Prepare retrieval fields Depending on the retrieval approach, prepare what the index needs, such as: Chunk text Metadata fields Keywords Embeddings for semantic retrieval 8. Run quality checks Before writing the data to the index, verify that: Important text was extracted correctly Chunks are understandable Metadata points to the correct source Required pages or sections are not missing Permissions are preserved For the broader architecture around these steps, see “How do you design a document ingestion pipeline for RAG?” . In short, documents should be cleaned, structured, deduplicated, chunked, enriched with metadata, and validated before they are indexed so the retrieval system receives reliable and traceable content.
Why is document parsing important in RAG?
Document parsing is important in RAG because the retriever can only search the information that was successfully extracted from the source document. If parsing is poor, useful content may be missing, reordered, merged incorrectly, or stripped of important structure before indexing even begins. What parsing does Parsing converts a source document into usable content such as: Paragraph text Headings Lists Tables Page numbers Section boundaries Metadata That extracted content is then cleaned, chunked, indexed, and later retrieved. Why parsing quality matters Poor parsing can damage retrieval in several ways: Missing text — important facts may never reach the index. Broken reading order — multi column PDFs can produce text in the wrong sequence. Lost headings — chunks may lose the section context needed to understand them. Flattened tables — rows and columns may become unreadable or misleading. Missing page numbers or metadata — retrieved answers become harder to trace back to the source. Bad chunk boundaries — incorrect structure can cause related content to be split apart. Simple example Suppose a PDF contains: If the parser extracts the content incorrectly as: the retrieval system may no longer know which warranty period belongs to which product. Parsing comes before retrieval quality A strong embedding model or retriever cannot recover information that was never extracted correctly. This is why document parsing should preserve both the content and the structure needed to interpret that content . Different formats may require different parsing approaches. PDFs, scanned files, presentations, spreadsheets, and web pages can all have different extraction challenges. For the preparation stages surrounding parsing, see “What steps should happen before documents are indexed in a RAG system?” . In short, document parsing is important in RAG because it determines the quality and structure of the information that will later be chunked, indexed, retrieved, and used to generate answers.
How should PDF files be parsed for RAG?
PDF files should be parsed for RAG by extracting their content, reading order, structure, and source metadata rather than treating the PDF as one large block of text. A PDF is primarily a page layout format, so text that looks correct visually may not be stored in the same logical order internally. Good parsing therefore needs to reconstruct the document in a form that preserves meaning. 1. Identify the type of PDF First determine whether the PDF is: Text based — the document contains selectable digital text. Scanned — pages are mainly images and require OCR. Mixed — some pages or regions contain native text while others require OCR. For text based PDFs, prefer native text extraction. OCR should generally be used only for pages or regions where reliable text cannot be extracted. 2. Extract text in the correct reading order Do not assume the raw extraction order is correct. PDFs may contain: Multiple columns Sidebars Headers and footers Footnotes Text boxes Captions The parser should reconstruct the order in which a human would normally read the page. 3. Preserve document structure Keep useful structural information such as: Headings and subheadings Paragraph boundaries Lists Section hierarchy Page boundaries This structure helps later chunking preserve meaningful units instead of splitting unrelated text together. 4. Handle tables and visual content separately Tables should be extracted in a structure that preserves their rows, columns, and relationships instead of flattening them into random text. Images, charts, and diagrams should be detected and handled according to whether they contain information needed for retrieval. Scanned PDFs, tables, and images each have additional parsing considerations and should be handled by dedicated processing steps. 5. Preserve metadata Each extracted piece of content should remain traceable to the original PDF. Useful metadata includes: Document ID or file name Page number Section title Source location Document version This metadata can later support filtering, citations, debugging, and source verification. 6. Remove extraction noise carefully Repeated headers, footers, page decorations, or broken whitespace may be removed when they do not add meaning. However, cleaning should not remove headings, labels, references, or other information required to interpret the content. 7. Validate before chunking Before splitting the parsed PDF into chunks, inspect representative pages and confirm that: Text is complete Reading order is correct Headings are preserved Tables are understandable Page numbers are correct OCR has not introduced major errors For scanned documents, see “How should scanned PDFs be handled in RAG?” . Tables and images are covered separately in “How should tables be extracted from PDFs for RAG?” and “How should images inside documents be handled in RAG?” . In short, PDF parsing for RAG should produce clean, correctly ordered, structure aware, metadata rich content that faithfully represents the original document before chunking and indexing.
How should scanned PDFs be handled in RAG?
Scanned PDFs should be handled in RAG by treating each scanned page as an image that must first be converted into reliable machine readable content before chunking and indexing. Unlike a normal text based PDF, a scanned PDF may contain little or no usable text layer. 1. Detect whether OCR is needed Before processing, check whether the page already contains extractable text. If the page is image only or the text layer is unusable, apply OCR. Mixed PDFs may require OCR only on some pages. 2. Improve the page image when necessary Poor scans can reduce OCR accuracy. Useful preprocessing may include: Correcting page rotation Deskewing tilted pages Improving contrast Removing excessive noise Cropping unnecessary borders The goal is not to visually beautify the page, but to make the text easier to recognize. 3. Run OCR OCR converts the visible text in the scanned page into machine readable text. The OCR process should ideally preserve: Reading order Paragraph boundaries Headings Lists Tables or table regions Page numbers 4. Preserve the link to the original page Every extracted block should remain traceable to its source page. Useful metadata includes: Document ID Page number Source file OCR method OCR confidence when available Section or heading information This makes citations, debugging, and reprocessing easier. 5. Handle complex visual regions separately A scanned page may contain more than normal paragraphs. Tables, charts, diagrams, forms, and handwritten notes may require specialized extraction rather than plain OCR alone. Do not assume that converting every visible region into a flat text stream will preserve its meaning. 6. Validate OCR quality Before chunking, inspect representative pages and check for: Missing words Character substitutions Incorrect reading order Lost headings Broken tables Repeated headers or footers Pages with very poor OCR quality may need reprocessing or manual review. Why this matters If OCR errors are indexed, those errors become part of the searchable knowledge base. The retriever may then miss relevant passages or return distorted information to the language model. For the OCR concept itself, see “What is OCR and when is it needed in RAG?” . In short, scanned PDFs should be OCR processed, structure aware, traceable to their original pages, and quality checked before their content is chunked and indexed for RAG.
How should tables be extracted from PDFs for RAG?
Tables in PDFs should be extracted for RAG in a way that preserves their structure and relationships , not just the visible words. If a table is flattened into an arbitrary text sequence, the retriever may lose which value belongs to which row, column, or heading. 1. Detect the table as a separate structure Identify the table boundaries instead of treating it as ordinary paragraph text. The extraction should preserve: Column headers Row labels Cell values Merged cells when meaningful Table title or caption Footnotes Page number and section context 2. Preserve row and column relationships Suppose a table contains: A poor extraction might become: That loses the relationship between each product and its warranty. A better representation keeps each value connected to its headers, for example: This format is often easier to retrieve and interpret. 3. Handle complex tables carefully PDF tables may include: Multi row or multi column headers Merged cells Nested sections Tables spanning multiple pages Missing borders Repeated headers on later pages The parser should reconstruct these relationships before indexing. For multi page tables, rows from later pages should remain connected to the correct table and header structure. 4. Preserve surrounding context A table may be difficult to understand without its title or nearby section. Useful metadata can include: Document ID Page number Section heading Table title Source file Table number This context can later support filtering, citations, and more accurate interpretation. 5. Choose a retrieval friendly representation Depending on the application, tables may be stored as: Structured rows or records Header aware text JSON like data Markdown like text A combination of structured data and natural language summaries The important point is to preserve the meaning of each cell rather than only its visual position. 6. Use OCR only when necessary If the table is part of a scanned PDF, OCR may be required before its contents can be reconstructed. However, OCR alone may not correctly recover row and column relationships, so table structure detection is still needed. 7. Validate extracted tables Before indexing, check that: Headers match the correct columns Rows are complete Numeric values are accurate Units and symbols are preserved Multi page tables were joined correctly The extracted representation is understandable without looking at the original layout For general PDF processing, see “How should PDF files be parsed for RAG?” . In short, tables should be extracted as structured, header aware data that preserves row and column relationships, surrounding context, and source metadata before being indexed for RAG.
How should images inside documents be handled in RAG?
Images inside documents should be handled in RAG based on whether they contain information that is important for answering future queries . Not every image needs special processing. Decorative logos, backgrounds, and repeated icons can usually be ignored, while charts, diagrams, screenshots, scanned text, and annotated figures may contain valuable knowledge. 1. Classify the image First determine what kind of visual it is, for example: Decorative image Photograph Diagram Flowchart Chart or graph Screenshot Scanned text Figure with labels This helps decide whether OCR, visual understanding, or no further processing is needed. 2. Extract visible text when useful If the image contains readable text, OCR can convert that text into machine readable content. This is especially useful for: Screenshots Scanned pages Labels inside diagrams Text embedded in figures OCR alone may not capture relationships between visual elements, so it should not always be treated as a complete description of the image. 3. Preserve captions and nearby text The image caption, figure title, surrounding paragraph, section heading, and page number often explain what the visual means. Useful metadata may include: Document ID Page number Figure number Caption Section title Image position This context should remain linked to the extracted image information. 4. Describe information that depends on the visual If important meaning cannot be recovered from text alone, a vision capable model or specialized visual parser can create a structured description of the image. For example, a system may describe: The description should capture useful facts rather than generic visual details. 5. Handle specialized visuals appropriately Different image types may require different processing: Charts — preserve axes, labels, legends, trends, and important values. Diagrams — preserve entities, labels, connections, and flow direction. Screenshots — extract important UI text and relevant interface relationships. Scanned text — use OCR. 6. Store image derived information with source traceability Any OCR text or generated description should stay connected to the original image and document location so it can be verified later. Avoid indexing meaningless visuals Indexing every image can add noise and unnecessary processing cost. A useful rule is to ask whether the image contains information that a user might reasonably need to retrieve. If not, it can usually be excluded. Charts and diagrams require additional care and are covered separately in “How do you process charts and diagrams for RAG?” . In short, images in documents should be selectively processed using OCR, captions, metadata, and visual understanding when they contain useful knowledge, while decorative or irrelevant images should generally be ignored.
How should DOCX files be parsed for RAG?
DOCX files should be parsed for RAG by extracting their content together with the document structure that gives that content meaning . Unlike many PDFs, DOCX files usually contain explicit structural information such as heading styles, paragraphs, lists, tables, and hyperlinks. A good parser should preserve that structure instead of converting the entire document into one flat block of text. 1. Extract text by document element Read the document in logical order and preserve elements such as: Headings Paragraphs Bulleted and numbered lists Tables Captions Hyperlinks Footnotes or endnotes when relevant 2. Preserve heading hierarchy Heading styles such as Heading 1, Heading 2, and Heading 3 are valuable because they reveal the document's section hierarchy. For example: Keeping this hierarchy helps later chunks retain the section context needed for retrieval. 3. Parse tables separately Tables should preserve their row and column relationships rather than being flattened into an arbitrary text sequence. If a table contains important business or technical information, keep its headers and surrounding section context with the extracted representation. 4. Handle headers, footers, and repeated content carefully Headers and footers may contain useful information such as document titles or versions, but repeated boilerplate on every page or section can create noisy duplicates. Preserve meaningful metadata while removing repetition that does not help retrieval. 5. Preserve metadata Useful metadata may include: Document ID File name Document title Section heading Parent section Author when relevant Version or modified date Source location 6. Handle embedded objects selectively DOCX files can also contain: Images Charts Diagrams Embedded files If these objects contain knowledge that users may query, process them separately using OCR or visual understanding as appropriate. Decorative elements can usually be ignored. 7. Normalize without destroying structure Clean unnecessary whitespace or formatting artifacts, but do not remove headings, list relationships, table structure, or other signals needed to interpret the content. 8. Validate before chunking Before indexing, check that: The reading order is correct Headings are attached to the right sections Lists remain understandable Tables are complete Important links and metadata are preserved Embedded knowledge was not silently dropped For the general role of parsing quality, see “Why is document parsing important in RAG?” . In short, DOCX files should be parsed as structured documents, preserving headings, paragraphs, lists, tables, metadata, and useful embedded content so later chunking and retrieval retain the original meaning.
How should PPT or PPTX files be parsed for RAG?
PPT and PPTX files should be parsed for RAG by treating the presentation as a sequence of structured slides , not as one continuous block of text. A good parser should preserve the meaning created by slide titles, bullet hierarchy, visual elements, speaker notes, and slide order. 1. Process slides in order Each slide should retain its position in the presentation. Useful metadata includes: Presentation ID or file name Slide number Slide title Section name when available Source location This makes retrieved content easier to trace back to the original presentation. 2. Preserve text structure Extract text from: Slide titles Subtitles Bullet points Text boxes Labels Captions Speaker notes when relevant Bullet nesting should be preserved because indentation often represents parent child relationships. For example: Flattening this into unrelated sentences can remove important context. 3. Remove repeated presentation noise carefully Slide masters may add repeated elements such as: Company logos Footers Confidentiality notices Dates Page decorations Repeated decorative content should usually be excluded from retrieval unless it carries useful information. 4. Parse tables as structured data Tables should preserve their row headers, column headers, and cell relationships. Do not simply concatenate all table cells into one text string. 5. Handle charts and diagrams separately Charts, process diagrams, and architecture diagrams may contain information that ordinary text extraction misses. When useful, preserve: Chart title Axis labels Legends Important values Diagram labels Connections and flow direction A vision capable parser or specialized chart/diagram processor may be needed when the meaning depends on the visual structure. 6. Process images selectively If a slide contains screenshots or images with important text, OCR may be useful. Decorative backgrounds and stock images normally do not need to be indexed. 7. Include speaker notes when they add knowledge Speaker notes can contain explanations that are not visible on the slide itself. If the notes contain useful domain knowledge, keep them linked to the correct slide rather than mixing them with unrelated slides. 8. Chunk around slide meaning A single slide is often a natural retrieval unit, but large or information dense slides may need smaller chunks. When splitting, keep the slide title and important context attached to each chunk. 9. Validate the extracted result Before indexing, check that: Slide order is correct Titles remain linked to slide content Bullet hierarchy is preserved Tables and charts remain understandable Notes are attached to the right slide Important visual information was not silently lost For the general principles behind parsing quality, see “Why is document parsing important in RAG?” . In short, PPT and PPTX files should be parsed slide by slide while preserving titles, hierarchy, notes, tables, visuals, and slide level metadata so retrieval retains the presentation's original context.
How should Excel and CSV files be parsed for RAG?
Excel and CSV files should be parsed for RAG by preserving the meaning of rows, columns, headers, sheets, and data types instead of converting the entire file into an unstructured block of text. Tabular data depends heavily on structure. A value such as 2500 is only useful if the system also knows which column, row, product, date, or unit it belongs to. 1. Identify the table structure For CSV files, determine: Header row Delimiter Character encoding Column names Data types For Excel files, also preserve: Workbook name Sheet name Multiple tables within a sheet Merged cells when meaningful Named ranges when useful 2. Keep headers attached to values Do not flatten rows into disconnected values. For example, instead of: preserve the relationships: This gives retrieval systems and language models enough context to interpret each value. 3. Preserve row identity and source metadata Useful metadata may include: File name Sheet name Table name Row number or record ID Column names Version or modified date This makes retrieved records easier to trace back to the source. 4. Handle data types correctly Numbers, dates, percentages, currencies, and text should remain distinguishable. For example, 0.15 may mean very different things depending on whether the source column represents a percentage, probability, or raw numeric value. 5. Handle formulas carefully Excel cells may contain formulas as well as calculated values. For retrieval, the system may need to preserve: The displayed or calculated value The formula when it is important for understanding how the value was derived Do not replace a useful calculated result with only the formula text unless the application specifically needs formula analysis. 6. Decide how to create retrieval units Possible retrieval units include: One row per record A small group of related rows A complete logical table A summarized representation of a larger table The right choice depends on what users are expected to ask. For example, a product catalog may work well as one record per product, while a financial report may need section level context around related rows. 7. Avoid embedding huge tables blindly Very large spreadsheets are often better handled with structured filtering, database queries, or calculations rather than embedding every value as plain text. RAG can still retrieve the relevant table, sheet, or records, but exact aggregation or calculation is usually better performed by structured tools. 8. Validate before indexing Check that: Headers match the correct columns Rows are not shifted Dates and numbers were parsed correctly Empty cells are handled consistently Sheet names and record identifiers are preserved Formulas and calculated values were interpreted as intended For general table handling principles, see “How should tables be extracted from PDFs for RAG?” . In short, Excel and CSV files should be parsed as structured data, preserving headers, row relationships, data types, sheet context, and metadata so retrieval does not lose the meaning behind individual values.