What problem does RAG solve?

RAG addresses a core limitation of language models: they cannot reliably answer from information they do not have access to at generation time.

A model normally generates responses from patterns learned during training and from whatever context is included in the prompt. This creates several problems when the required knowledge is:

• Missing from the model's training data
• Outdated because the information changed after training
• Private or organization-specific
• Too large to include entirely in the prompt
• Precise enough that guessing would be risky

The problem in practice

Suppose a user asks:

What is the refund policy for Product X?

If the model was never trained on the latest policy, it may:

• Give an outdated answer
• Produce a generic answer
• Guess details that are not actually documented

RAG changes this by retrieving the relevant policy text first and giving that information to the model as context before generation.

Question
   ↓
Retrieve relevant knowledge
   ↓
Provide retrieved context to the model
   ↓
Generate the answer

This lets the model answer using information that exists outside its internal parameters.

RAG therefore helps bridge the gap between a model's built-in knowledge and the external knowledge needed for a specific question.

It does not eliminate all incorrect answers. If retrieval returns irrelevant, incomplete, or outdated information, the generated response can still be wrong.

For the broader reasons organizations use this approach, see “Why is RAG used?”.

In short, RAG solves the problem of giving a generative model access to relevant external knowledge at the time it answers a question.