What is BM25 and why is it used in RAG?

BM25 (Best Matching 25) is a lexical ranking algorithm that scores how well a document or chunk matches the words in a search query. In RAG, it is commonly used for sparse retrieval, especially when exact terminology matters.

How does BM25 rank chunks?

BM25 considers three main signals:

• Term frequency: A query term appearing more often can increase relevance, but with diminishing returns. Repeating a word many times does not increase its score indefinitely.
• Inverse document frequency (IDF): Terms that occur in fewer documents generally carry more weight than common terms.
• Document-length normalization: BM25 accounts for chunk length so longer passages do not automatically score higher simply because they contain more words.

Two familiar tuning parameters are k1, which controls term-frequency saturation, and b, which controls the strength of length normalization.

Example in RAG

A user asks: "How do I fix error E_CONN_204?"

A support document contains E_CONN_204 and the exact troubleshooting steps. BM25 can rank that chunk highly because the rare error identifier matches directly. A dense embedding retriever might instead favor a generally related network troubleshooting passage.

The RAG pipeline then passes the retrieved source chunk to the LLM to answer the question.

Where BM25 fits

1. Parse and chunk source documents.
2. Build a searchable lexical index for those chunks.
3. Tokenize the user's query and find matching chunks using an inverted index.
4. Use BM25 scores to rank candidates.
5. Retrieve the best chunks and optionally combine them with dense-search results or rerank them before generation.

BM25 retrieves evidence; the LLM generates the answer.

Why use BM25 in RAG?

• Exact matches: Useful for error codes, IDs, product names, technical terms, and uncommon keywords.
• No embedding required: Lexical scoring works without a neural embedding model.
• Efficient retrieval: Inverted indexes can narrow candidate documents quickly.
• Complementary signals: BM25 and dense retrieval often find different useful passages, making them a strong combination for hybrid search.

Limitations

BM25 primarily relies on query terms and text analysis. A passage may be highly relevant but use different wording.

For example, a question about working from home may not match a policy that uses only remote work as strongly as a good dense retriever would.

Tokenization, stopwords, stemming, chunk length, and indexing configuration also affect results. BM25 scores are not automatically comparable to vector similarity scores, so hybrid systems usually normalize or fuse ranked results rather than adding raw scores blindly.

Interview takeaway

BM25 is a term-based ranking algorithm that combines term frequency, term rarity, and document-length normalization. In RAG, it provides fast sparse retrieval and excels at exact terms and identifiers; combining it with dense retrieval can improve coverage across both lexical and semantic queries.