Research

PaperQA: Retrieval-Augmented Generative Agent for Scientific Research

A retrieval-augmented research agent that searches scientific literature, evaluates source relevance, and synthesizes evidence-backed answers.

Jakub Lála, Odhran O'Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, Andrew D. WhiteDec 8, 2023arXiv10 min read
IMPORTANTRAGIntermediate
Paper

Key takeaway

PaperQA moves RAG from simple document lookup toward an agentic research workflow over scientific papers.

Updated on Oct 2, 2026

01

Abstract

  • PaperQA is a retrieval-augmented generative agent designed to answer questions over scientific literature.
  • It retrieves evidence from full-text scientific papers rather than relying only on model parametric knowledge.
  • The system explicitly assesses source and passage relevance before synthesis.
  • Generated answers are accompanied by citations to retrieved scientific sources.
  • The paper frames provenance as a key requirement for scientific question answering.
  • PaperQA is evaluated as both an information-retrieval system and a question-answering agent.
  • The system is designed for multi-document scientific questions requiring evidence synthesis.

02

Introduction

  • Scientific literature contains information distributed across papers, sections, figures, and references.
  • LLMs can hallucinate facts and citations when answering scientific questions from memory.
  • Conventional QA benchmarks often do not test literature-scale retrieval and synthesis.
  • PaperQA addresses the gap by combining retrieval, relevance assessment, and generation.
  • The authors introduce LitQA to test research-like questions requiring full-text literature.
  • The target behavior is closer to a human researcher searching papers and synthesizing evidence.
  • Citation provenance is treated as part of answer quality rather than an optional add-on.

03

Problem

  • The problem is to answer scientific questions accurately using a large collection of full-text papers.
  • The agent must find relevant papers and passages before producing an answer.
  • Retrieved evidence must be filtered so irrelevant passages do not dominate synthesis.
  • The final response should connect claims to supporting citations.
  • Questions may require combining evidence from multiple papers.
  • Evaluation therefore needs to test retrieval and synthesis together.
  • LitQA was introduced specifically to measure this literature-level research capability.

04

Background

  • PaperQA uses retrieval-augmented generation over scientific documents.
  • It searches across full-text scientific articles rather than only abstracts.
  • Candidate passages are retrieved for a user question.
  • Retrieved passages are assessed for relevance.
  • Relevant evidence is supplied to the language model for answer generation.
  • The generated answer contains citations associated with supporting evidence.
  • The system therefore creates an explicit evidence-to-answer path.
  • The architecture separates retrieval of candidate evidence from reasoning/synthesis over that evidence.

05

Methodology

  • A PaperQA interaction begins with a scientific question.
  • The agent retrieves candidate scientific sources and passages.
  • It evaluates whether retrieved material is relevant to the question.
  • Relevant passages are passed into the generation context.
  • The language model synthesizes an answer from the selected evidence.
  • Citations are generated to identify supporting papers.
  • The process can combine evidence from several articles.
  • The resulting answer is intended to be both useful and traceable to literature.
Figure 1

Figure 1: PaperQA Workflow Diagram

Original Figure 1 from the PaperQA paper showing the agent workflow: an Agent LLM orchestrates Search, Gather Evidence, and Answer Question tools, iterating until sufficient evidence is available for a cited answer.

Original Figure 1 from Lala et al., PaperQA: Retrieval-Augmented Generative Agent for Scientific Research, arXiv:2312.07559v2. Image hosted by alphaXiv from the paper figure.

Formula notes

Mathematical details

Retrieval similarity

Equation
score(q,d)=sim(E(q),E(d))score(q,d)=sim(E(q),E(d))

Rank candidate documents by similarity between the query embedding and document embedding.

E
-embedding function
d
-document
q
-query
sim
-similarity function

06

Architecture

  • The architecture consists of a retrieval component, relevance assessment, and generative synthesis.
  • Scientific papers form the external knowledge corpus.
  • Retrieval narrows the corpus to candidate evidence.
  • A relevance step filters or ranks evidence before generation.
  • The LLM performs synthesis rather than relying solely on its pretrained memory.
  • Citations connect generated claims to literature sources.
  • The agentic framing allows the system to perform multiple retrieval/assessment operations rather than a single static context lookup.

07

Dataset

  • PaperQA operates over scientific full-text literature.
  • The evaluation includes science questions whose answers depend on information distributed across papers.
  • LitQA is introduced as a more challenging benchmark for literature-based question answering.
  • LitQA questions require retrieval and synthesis from full-text scientific papers.
  • The benchmark is designed to be closer to realistic research questions than standard science QA.
  • Human expert researchers are used as a reference point in the LitQA evaluation.
  • The benchmark therefore tests both finding evidence and integrating it into an answer.

08

Training

  • PaperQA uses a large language model for evidence assessment and answer synthesis.
  • The model is used at inference time as an agent rather than being trained specifically on LitQA.
  • External scientific documents provide the factual evidence.
  • Retrieval reduces the amount of literature placed directly into the model context.
  • Relevance assessment is used to improve evidence quality.
  • Citation generation provides an explicit provenance channel.
  • The system demonstrates that useful research assistance can be built without fine-tuning a scientific foundation model for every question.

09

Experiments

  • The paper evaluates PaperQA on existing science QA benchmarks.
  • It also introduces LitQA for a harder research-style evaluation.
  • Performance is compared with existing LLMs and LLM-agent approaches.
  • Evaluation considers answer correctness in addition to the ability to retrieve relevant literature.
  • LitQA requires evidence synthesis across full-text papers.
  • Human expert performance is used as an important comparison.
  • The reported results show PaperQA substantially improves over the compared LLM/agent baselines on the paper's science QA tasks.

Evidence highlights

  • 01State management
  • 02Tool contracts
  • 03Observability
  • 04Evaluation
  • 05Safety
  • 06Cost control

10

Baselines

  • Baselines include existing large language models used for science question answering.
  • The paper also compares against LLM-agent approaches that attempt to perform research tasks.
  • Retrieval-augmented approaches provide a relevant comparison because they can ground answers in external documents.
  • Human researchers serve as a stronger practical reference on LitQA.
  • The benchmark is intentionally harder than simple single-passage retrieval because synthesis across literature is required.
  • The comparisons test whether retrieval plus evidence assessment provides a meaningful advantage over pure generation.

11

Results

  • PaperQA exceeds the performance of existing LLMs and LLM agents on the science QA benchmarks reported in the paper.
  • On LitQA, PaperQA is reported to match expert human researchers.
  • The results support the value of retrieval over full-text scientific papers.
  • Evidence assessment helps the system focus generation on relevant passages.
  • Citation-backed answers provide a practical provenance mechanism for scientific use.
  • The main empirical claim is not merely better retrieval, but stronger end-to-end research-style question answering.
  • The paper positions PaperQA as a step toward automated processing of scientific knowledge at scale.

12

Ablation

  • The benchmark design itself tests whether retrieval and synthesis are both necessary.
  • Full-text literature retrieval is important because relevant information may not appear in an abstract.
  • Relevance filtering limits the effect of noisy retrieved passages.
  • Human comparison on LitQA provides a higher bar than comparison only against language models.
  • The paper's system-level design combines multiple components rather than treating RAG as a single retrieval call.
  • The results indicate that literature-grounded generation can approach expert-level performance on the reported benchmark.
  • The evaluation also highlights that benchmark difficulty changes substantially when questions require cross-paper synthesis.

13

Limitations

  • Retrieval failures can prevent the agent from finding the evidence needed to answer correctly.
  • Irrelevant passages can mislead the generator even when retrieval technically succeeds.
  • Missing evidence can lead to incomplete or unsupported answers.
  • Scientific terminology and long documents make retrieval challenging.
  • Questions requiring synthesis across several sources increase reasoning complexity.
  • Generated claims can still be incorrect even when relevant evidence is retrieved.
  • Citation correctness is critical because a citation can appear plausible without actually supporting the associated claim.
  • The system therefore depends jointly on retrieval quality, evidence selection, model reasoning, and citation generation.

14

Conclusion

  • PaperQA demonstrates a research-agent pattern: search scientific literature, assess evidence, then synthesize a cited answer.
  • Its central contribution is applying RAG as an agent for scientific research rather than generic document QA.
  • LitQA provides a benchmark explicitly designed around literature-scale retrieval and synthesis.
  • The reported results show performance competitive with or exceeding strong LLM and agent baselines.
  • Matching expert researchers on LitQA is the paper's strongest practical result.
  • The architecture provides provenance through citations.
  • The work suggests that grounded agents can make scientific literature more accessible while retaining links to source evidence.

15

References

  • Jakub Lála et al., “PaperQA: Retrieval-Augmented Generative Agent for Scientific Research,” arXiv:2312.07559.
  • Authors: Jakub Lála, Odhran O'Donoghue, Aleksandar Shtedritski, Sam Cox, Samuel G. Rodriques, Andrew D. White.
  • The paper was submitted to arXiv in December 2023.
  • Primary source: https://arxiv.org/abs/2312.07559.
  • The work introduces the LitQA benchmark.
  • PaperQA retrieves from full-text scientific articles, assesses relevance, and generates cited answers.
  • The paper's central claim is that this agentic RAG design can reach expert-human performance on the reported LitQA benchmark.

Continue reading

Related research

RAG

RAG Survey

A systematic survey organizing RAG architectures, training strategies, applications, evaluation, and open challenges.

Browse all research
Machinoai Logo
Machinoai

Your edge in the AI era. Deep dives into machine learning concepts, paper explainers, verified industry jobs, and developer tools.

YouTube

© 2026 Machinoai. All rights reserved.