Retrieval-Augmented Generation
Retrieval-augmented generation connects a generative model to a searchable knowledge source. Instead of relying only on model weights, the system fetches relevant documents and places them into the prompt or context.
How should HTML and web pages be parsed for RAG?
HTML and web pages should be parsed for RAG by extracting the main content and meaningful page structure while removing navigation, advertising, cookie notices, repeated templates, and other boilerplate that does not help retrieval. A web page contains much more than the article or documentation text visible to the user, so simply indexing the raw HTML can introduce a large amount of noise. 1. Identify the main content Extract the part of the page that contains the useful information, such as: Article body Documentation content Product details FAQ sections Help center content Forum or knowledge base posts Repeated site wide elements should usually be excluded. 2. Preserve semantic structure Useful HTML structure includes: Page title Headings and subheadings Paragraphs Lists Tables Code blocks Links Captions Heading hierarchy is especially useful because it helps later chunks retain their section context. 3. Remove boilerplate and navigation noise Common content that may not belong in the retrieval index includes: Navigation menus Cookie banners Advertisements Repeated headers and footers Related content widgets Social sharing controls Site wide legal notices The goal is to keep content that answers questions, not every visible element on the page. 4. Preserve important links Links can provide useful context when they point to: Related documentation Source references Downloads Definitions Other pages in the same knowledge base When relevant, keep the anchor text and destination URL as metadata. 5. Handle dynamic pages carefully Some websites render important content using JavaScript after the initial HTML loads. If the useful content is not present in the initial page source, the ingestion process may need a browser based or rendered page extraction step. 6. Preserve page metadata Useful metadata may include: URL Page title Site or domain Section heading Publication date when available Last modified date when available Author when relevant Crawl or ingestion time This information helps with filtering, traceability, freshness checks, and citations. 7. Handle tables, code, and structured elements separately Tables should preserve their row and column relationships, while code blocks should keep formatting that affects meaning. For technical documentation, losing indentation or code boundaries can make retrieved examples difficult to interpret. 8. Normalize URLs and detect duplicate pages Websites may expose the same content under multiple URLs because of: Tracking parameters Print versions Pagination Alternate routes Duplicate category paths Canonical URLs or other deduplication logic can help prevent the same page from being indexed repeatedly. 9. Validate the extracted page Before indexing, check that: The main content was captured Heading order is correct Navigation text is not dominating the result Tables and code remain understandable Important links and metadata are preserved Duplicate or near duplicate pages are handled For the broader parsing principles, see “Why is document parsing important in RAG?” . In short, HTML and web pages should be parsed by isolating useful content, preserving semantic structure and metadata, removing repeated site noise, and validating the result before chunking and indexing.
What metadata should be preserved while parsing documents for RAG?
Metadata preserved during RAG document parsing should help the system understand where a piece of content came from, where it belongs in the source, how current it is, and who is allowed to access it . The exact metadata depends on the application, but several categories are especially useful. 1. Source identity Preserve information that uniquely identifies the source document, such as: Document ID File name Source URL or storage location Document type Repository or knowledge base name This makes retrieved chunks traceable back to their original source. 2. Position inside the document Keep metadata that identifies where the content appeared, for example: Page number Slide number Sheet name Row or record ID Section title Subsection title This is important for citations, debugging, and reconstructing context. 3. Document hierarchy Preserve structural relationships such as: Parent section Heading path Chapter Subchapter Table or figure number For example: This hierarchy can help a retrieved chunk remain understandable even when it is separated from the rest of the document. 4. Version and freshness information When available, preserve: Document version Publication date Last modified time Effective date Ingestion time This helps the system distinguish current information from older content and supports re indexing when documents change. 5. Access and permission metadata For private or multi user knowledge bases, metadata may need to include: Tenant or organization ID User or group permissions Confidentiality level Access control labels Retrieval should respect these permissions so users do not receive content they are not authorized to access. 6. Content type metadata Useful labels may include: Paragraph Table Code block Image description FAQ Product record This can help the retriever apply filters or choose different retrieval strategies for different content types. 7. Processing metadata In some systems it is useful to keep information such as: Parser version OCR usage OCR confidence Chunk ID Parent document ID This is especially valuable for debugging ingestion problems and reprocessing documents later. Do not preserve metadata blindly More metadata is not always better. Keep metadata that supports: Retrieval Filtering Access control Citations Freshness Debugging Metadata that has no practical use can increase storage and operational complexity without improving retrieval. For a deeper explanation of positional metadata, see “Why should page numbers and section headings be preserved in RAG ingestion?” . In short, RAG parsing should preserve enough metadata to identify the source, locate the content within that source, retain hierarchy and version information, enforce permissions, and support reliable retrieval and citations.
Why should page numbers and section headings be preserved in RAG ingestion?
Page numbers and section headings should be preserved during RAG ingestion because they help each retrieved chunk retain location, context, and traceability . When a document is split into chunks, the chunk may be separated from the surrounding pages. Page and section metadata help restore part of that lost context. 1. Page numbers improve source traceability If a retrieved passage includes its page number, the system can point back to the exact location in the original document. This is useful for: Citations Human verification Auditing Debugging retrieval results Reviewing source evidence Without page numbers, a user may know which document was used but still have difficulty finding the supporting passage. 2. Section headings give chunks semantic context A chunk may contain text that is ambiguous on its own. For example: If the chunk also carries the heading: its meaning becomes much clearer. 3. Headings help distinguish similar content Large documents often repeat similar terms in different sections. For example, timeout might appear under: API configuration Database connections User sessions Preserving the section heading helps retrieval and generation interpret which meaning applies. 4. They can improve retrieval and filtering Section titles can be indexed or stored as metadata so the retriever can use them as additional context. Page and section metadata can also support filters or reranking rules when the application needs information from a particular part of a document. 5. They help reconstruct document hierarchy Preserving headings can maintain relationships such as: This hierarchy helps a small chunk retain the broader topic it belongs to. 6. They make retrieval errors easier to diagnose If a RAG system retrieves an incorrect passage, page numbers and headings make it much easier to inspect: What was retrieved Where it came from Whether chunking split the document badly Whether the wrong section was selected Preserve them as metadata, not repeated noise Page numbers and headings usually do not need to be repeated excessively inside every chunk's visible text. They can be stored as structured metadata and attached to the chunk when needed for retrieval, prompting, or citation. For the broader set of useful fields, see “What metadata should be preserved while parsing documents for RAG?” . In short, page numbers preserve exact source location, while section headings preserve semantic context. Together they make RAG retrieval easier to interpret, verify, filter, cite, and debug.
How do you normalize text before indexing it for RAG?
Text normalization before RAG indexing means cleaning extracted text so it is consistent and searchable without changing its meaning . The goal is not to rewrite the document. It is to remove extraction noise and standardize formatting while preserving information that may matter during retrieval. 1. Normalize character encoding Make sure the text uses a consistent encoding such as UTF 8. This helps avoid corrupted characters caused by: Different source encodings Copy and paste artifacts PDF extraction issues Special punctuation 2. Normalize Unicode carefully Visually similar characters may have different Unicode representations. Unicode normalization can make text more consistent, but it should be applied carefully so meaningful symbols are not lost. 3. Clean unnecessary whitespace Common cleanup includes: Removing repeated spaces Converting inconsistent line endings Removing excessive blank lines Joining lines that were broken only because of page layout However, line breaks that represent paragraphs, lists, code, or table structure should usually be preserved. 4. Remove extraction artifacts Parsing may introduce noise such as: Repeated headers Repeated footers Page break artifacts Broken hyphenation Stray control characters For example, a PDF extraction might produce: If the hyphen exists only because the word was split across lines, it can be reconstructed as: 5. Preserve meaningful structure Do not normalize away information such as: Headings Paragraph boundaries Lists Code blocks Table relationships Section labels These signals can improve chunking and retrieval. 6. Preserve numbers, units, and identifiers Be careful with values such as: Dates Currency Percentages Version numbers Product IDs API names Error codes For example, v2.1, HTTP 404, and 10 mg should not be changed into forms that alter their meaning. 7. Standardize only when it helps retrieval In some applications it may be useful to standardize predictable variations, such as: Repeated formatting conventions Known abbreviations Consistent date representations But aggressive lowercasing, stemming, punctuation removal, or rewriting is usually unnecessary before embedding based retrieval and may remove useful information. 8. Keep the original text when possible A useful design is to retain: The original extracted text A normalized version used for indexing when needed This makes debugging easier and allows the normalization process to be changed later without losing the source representation. 9. Validate the result Before indexing, confirm that normalization has not: Removed useful words Changed numeric values Destroyed code formatting Merged unrelated paragraphs Broken tables or lists Altered identifiers For the broader preparation workflow, see “What steps should happen before documents are indexed in a RAG system?” . In short, normalize RAG text conservatively: remove noise and inconsistencies, but preserve the wording, structure, identifiers, numbers, and context that retrieval may depend on.
How do you handle duplicate documents during RAG ingestion?
Duplicate documents during RAG ingestion should be detected and controlled before they create repeated chunks in the index. If the same content is indexed multiple times, retrieval can become biased toward that content, waste storage, and return several nearly identical passages instead of diverse evidence. 1. Assign a stable document identity Each source should have a stable identifier whenever possible. Examples include: Document ID Canonical URL Repository path Database record ID Storage object key This helps the ingestion pipeline recognize that a document has already been processed. 2. Use content hashes for exact duplicates A hash can be generated from the file or normalized document content. If two documents produce the same hash, they are likely exact duplicates. This works well when the same file is uploaded more than once under different names. 3. Normalize before comparing Small formatting differences can make otherwise identical documents look different. Before comparing content, it can be useful to normalize harmless differences such as: Extra whitespace Line ending differences Repeated headers or footers Encoding artifacts Normalization should not remove meaningful content. 4. Detect near duplicates separately Two documents may contain almost the same information but differ slightly because of: Minor edits Different formatting Added disclaimers Updated dates Export differences Near duplicate detection may use similarity techniques rather than exact hashes. Possible approaches include: Text similarity Fingerprinting MinHash or locality sensitive hashing Embedding similarity Near duplicate detection should be used carefully because two similar documents may still represent different valid versions. 5. Distinguish duplicates from newer versions A changed document should not automatically be discarded as a duplicate. Compare version information such as: Version number Last modified time Effective date Source revision ID Content hash If the new document is an updated version, the system may replace, deactivate, or supersede the older indexed content instead of storing both as active copies. 6. Avoid duplicate chunks Even when document level deduplication is in place, chunk level duplication can still occur. For example, repeated legal notices, headers, footers, or template sections may appear across many files. These repeated chunks can often be removed or marked as boilerplate before indexing. 7. Keep provenance If identical content legitimately appears in several sources, removing every copy may lose useful source information. One option is to store the content once while preserving references to all relevant source documents. 8. Define a clear ingestion policy A production system should decide what happens when a duplicate is detected. Common policies include: Skip exact duplicates Replace the existing document Create a new version Keep both but mark one inactive Merge source references The correct choice depends on how the knowledge base handles history and source authority. For handling document revisions over time, see “How do you handle document updates and versions in a RAG knowledge base?” . In short, handle duplicates with stable source identifiers, content hashes, careful similarity checks, and explicit versioning rules so repeated content does not unnecessarily pollute the RAG index.
How do you handle document updates and versions in a RAG knowledge base?
Document updates and versions in a RAG knowledge base should be handled so the system can recognize what changed, keep the active index current, and avoid retrieving stale and current versions at the same time . Versioning is important because simply adding every updated document as new content can leave old chunks in the index and create conflicting answers. 1. Give each logical document a stable ID A document should have an identity that remains the same across updates. Examples include: Document ID Canonical URL Repository path Database record ID Storage object key The version can change while the logical document identity stays constant. 2. Track version information Useful version metadata may include: Version number Last modified time Effective date Source revision ID Content hash Ingestion time This makes it possible to compare the incoming document with the currently indexed version. 3. Detect whether the document actually changed Before reprocessing, compare the new source with the indexed version. A content hash can quickly identify exact matches. If the content has not changed, unnecessary parsing, embedding, and indexing can be skipped. 4. Reprocess changed content If the document changed, parse and prepare the new version using the normal ingestion pipeline. For large documents, systems may optionally identify changed sections or chunks and re index only those parts instead of rebuilding everything. Incremental updates can reduce processing cost, but they require reliable chunk identities and change tracking. 5. Remove or deactivate stale chunks When the new version becomes active, chunks belonging to the previous version should normally be: Deleted Marked inactive Superseded Moved to version history The important point is that outdated chunks should not continue competing with current chunks during normal retrieval. 6. Keep historical versions when the use case requires them Some systems need to answer questions about past policies, contracts, specifications, or regulations. In that case, older versions can be retained with metadata such as: Valid from date Valid until date Version status Superseded by version Retrieval can then filter by the appropriate version or effective date. 7. Update related metadata A document update may change more than its text. Also review: Permissions Title Source URL Section hierarchy Tags Effective dates Stale metadata can cause incorrect filtering even when the text itself is current. 8. Make updates atomic when possible A production system should avoid a period where some chunks come from the old version and others from the new one. One approach is: This reduces inconsistent retrieval during updates. 9. Validate after re indexing After an update, verify that: New content can be retrieved Removed content is no longer returned Old and new versions are not mixed unintentionally Citations point to the correct version Permissions still work correctly For duplicate detection during ingestion, see “How do you handle duplicate documents during RAG ingestion?” . In short, handle RAG document updates with stable document identities, explicit version metadata, change detection, controlled re indexing, and clear activation or retirement of old chunks so retrieval uses the correct version of the knowledge.
How do you identify the number of images in a PDF before RAG ingestion?
To identify the number of images in a PDF before RAG ingestion, inspect the PDF page by page and count the image objects referenced by each page . This can help decide whether the document needs image extraction, OCR, chart processing, or multimodal handling. 1. Inspect every page A PDF parser can enumerate image objects on each page. For each image, useful information to collect includes: Page number Image object ID Width and height File format when available Position on the page This gives a page level inventory before the main ingestion pipeline runs. 2. Distinguish total image occurrences from unique images The same image can be reused on many pages. For example, a company logo may appear 50 times but still represent only one unique image asset. It is therefore useful to track both: Image occurrences — how many times images appear across pages Unique images — how many distinct image objects or contents exist Unique images can be identified using object references or content hashes. 3. Treat scanned pages carefully In a scanned PDF, each page may be stored as one large image. For example: In this case, an image count of three may actually mean the entire three page document requires OCR. 4. Do not assume every visual is an embedded image Some visuals in PDFs are built from: Vector graphics Drawing commands Shapes Text elements These may represent charts or diagrams without appearing as normal raster image objects. So counting embedded images alone does not necessarily tell you how many meaningful visual elements the document contains. 5. Ignore tiny decorative assets when appropriate A PDF may contain small icons, bullets, logos, or background elements that are technically images but are not useful for RAG. After counting images, filtering rules can consider: Image dimensions Repetition frequency Position Whether the image contains useful text or visual information 6. Build an image inventory A practical preprocessing step can produce metadata such as: This inventory helps the ingestion system decide which images should be ignored, OCR processed, described with a vision model, or stored for later multimodal retrieval. 7. Validate the count visually when needed For complex PDFs, compare the programmatic inventory with a few rendered pages. This helps detect cases where: Vector diagrams were not counted One visual is composed of several image objects Background images were mistaken for meaningful figures Reused assets inflated the count In short, identify PDF images by inspecting image objects page by page, distinguish occurrences from unique images, detect scanned full page images, and remember that vector graphics and composite visuals may not be represented as normal image objects.
What is OCR and when is it needed in RAG?
OCR (Optical Character Recognition) is the process of converting text that appears inside an image into machine readable text. In RAG, OCR is needed when important document content cannot be extracted directly as normal digital text. When OCR is needed OCR is commonly required for: Scanned PDFs Photos of documents Faxed documents Screenshots containing text Image only pages Labels or text embedded inside diagrams For example, a scanned contract may look like a normal PDF to a person, but internally each page may only be an image. Without OCR, the RAG pipeline may have little or no text to index. When OCR is not needed If a PDF or document already contains a reliable text layer, native text extraction is usually preferable. Using OCR unnecessarily can: Add processing cost Introduce recognition errors Lose formatting or structure Misread numbers, symbols, or technical terms A good ingestion pipeline therefore detects whether usable text already exists before applying OCR. What OCR produces OCR can extract visible text such as: Words and sentences Headings Page numbers Labels Table cell text Some OCR systems can also return useful information such as text coordinates, blocks, and confidence scores. OCR does not fully understand layout Recognizing characters is only part of the problem. A page may contain: Multiple columns Tables Forms Charts Diagrams Sidebars Even when OCR reads the words correctly, the RAG pipeline may still need layout analysis to determine the proper reading order and relationships between elements. OCR quality affects retrieval quality OCR errors become part of the indexed knowledge if they are not detected. Common errors include: 0 confused with O 1 confused with I Missing punctuation Broken words Incorrect reading order Misread technical identifiers These errors can reduce retrieval accuracy because the indexed text no longer matches the original information. Validate OCR before indexing Before chunking OCR output, check representative pages for: Missing text Incorrect characters Reading order problems Lost headings Broken tables Important numeric errors When OCR confidence is available, very low confidence regions can be flagged for reprocessing or review. For the full scanned document workflow, see “How should scanned PDFs be handled in RAG?” . In short, OCR is used in RAG when useful text exists only inside images or scanned pages. It converts that visible text into machine readable form, but the result should still be validated and combined with layout and source metadata before indexing.
How do you process charts and diagrams for RAG?
Charts and diagrams in RAG should be processed by extracting both their visible text and the relationships represented by the visual structure . OCR alone is often not enough because a chart or diagram may communicate meaning through position, arrows, axes, colors, grouping, or connections. 1. Identify the visual type First determine what kind of visual is present, for example: Bar chart Line chart Pie chart Flowchart Architecture diagram Network diagram Process diagram Annotated figure Different visual types require different extraction strategies. 2. Extract visible text Use OCR or native extraction to capture: Titles Axis labels Legends Data labels Node labels Annotations Captions This text is useful, but it should stay connected to the visual element it describes. 3. Preserve relationships For charts, preserve information such as: Which value belongs to which category Axis meaning and units Series names Trends Relative comparisons For diagrams, preserve: Nodes or components Connections Arrow direction Parent child relationships Processing order Grouping For example, a diagram may show: Extracting only the labels Client, API Gateway, Service, and Database would lose the actual architecture. 4. Use visual understanding when structure matters When relationships cannot be reliably reconstructed from text and layout rules alone, a vision capable model or specialized visual parser can generate a structured description. The description should focus on facts that could help answer questions rather than generic appearance. For example: 5. Preserve captions and surrounding context Charts and diagrams are often explained by nearby text. Keep useful context such as: Figure title Caption Section heading Page number Nearby explanatory paragraph This can make the extracted visual information easier to interpret during retrieval. 6. Store structured data when possible If a chart can be converted into reliable structured values, preserve them. For example: This can support more precise retrieval than a vague description such as "revenue increased." 7. Keep source traceability Image derived content should remain linked to: Document ID Page number Figure number Image or region ID Original source This allows users and developers to verify the retrieved evidence. 8. Validate important visuals Before indexing, check that: Labels were read correctly Values and units are accurate Connections were interpreted correctly Arrow direction was preserved Legends were matched to the right series The description does not invent information that is not visible 9. Avoid processing decorative visuals Logos, backgrounds, icons, and stock images usually do not improve retrieval and can be ignored unless they carry useful information. For the broader image handling workflow, see “How should images inside documents be handled in RAG?” . In short, charts and diagrams should be converted into retrieval friendly representations that preserve not only text, but also values, labels, connections, flow direction, and surrounding document context.
How do you preserve document hierarchy during RAG ingestion?
Document hierarchy should be preserved during RAG ingestion by keeping the structural relationships between chapters, sections, subsections, and the content that belongs under them . When a long document is split into chunks, those chunks can lose the context provided by the original document structure. Preserving hierarchy helps each chunk retain where it belongs. 1. Detect structural levels Identify the document's logical levels, such as: Document title Chapter Section Subsection Paragraph List Table or figure For example: This structure should not be reduced to three unrelated headings. 2. Build a heading path For each content block or chunk, store the chain of headings above it. For example: This heading path gives a retrieved chunk useful context even when the surrounding document is not retrieved. 3. Store parent child relationships Where useful, assign identifiers such as: document id section id parent section id chunk id parent chunk id This allows the system to understand that one piece of content belongs under another. 4. Attach hierarchy metadata to chunks When content is chunked, preserve metadata such as: Document title Chapter title Section title Subsection title Heading path Page number This helps retrieval, reranking, filtering, citations, and prompt construction. 5. Use structure aware chunk boundaries Chunking should respect natural document boundaries whenever possible. For example, avoid combining the end of one section with the beginning of an unrelated section simply because a fixed token limit was reached. Similarly, if a subsection is too large, split its content while keeping the same heading path attached to every resulting chunk. 6. Preserve relationships for non text elements Tables, figures, code blocks, and lists should remain associated with the section that explains them. A table extracted without its parent heading may be difficult to interpret during retrieval. 7. Support hierarchical retrieval when needed In some RAG systems, hierarchy can also be used during retrieval. For example, the system may: Retrieve a small chunk first Identify its parent section Expand context with neighboring or parent content This can provide more complete context without indexing an entire document as one large chunk. 8. Validate the hierarchy Before indexing, verify that: Headings are detected correctly Content belongs to the correct parent section Heading paths are complete Tables and figures remain attached to the right section Chunking does not accidentally cross unrelated structural boundaries For related metadata concepts, see “What metadata should be preserved while parsing documents for RAG?” and “Why should page numbers and section headings be preserved in RAG ingestion?” . In short, preserve RAG document hierarchy by capturing structural levels, parent child relationships, and heading paths, then carrying that context into every chunk so retrieved content remains connected to its original meaning.