How do you measure embedding latency in RAG?

Embedding latency in RAG is the time required to transform input text into an embedding vector. For user-facing searches, it usually means query embedding latency; document embedding latency is typically measured separately during ingestion.

What exactly should be timed?

1. Query embedding: Start the timer just before requesting an embedding for the user's finalized search query and stop when the vector is available. Report whether measurements include network transmission, provider queueing, and retries.
2. Document embedding: Measure time to embed document chunks during ingestion, plus batch throughput. Do not mix this background workload into user-facing retrieval timing.
3. Cache behavior: Measure cache hits and misses separately. A near-zero cache hit should not be presented as model inference latency.
4. End-to-end RAG latency: Keep query embedding separate from vector search, reranking, prompt assembly, and LLM generation.

Beginner-friendly numerical example

Suppose five uncached query embedding calls take:

40 ms, 50 ms, 45 ms, 65 ms, 100 ms

• Total time = 300 ms
• Average latency = 300 / 5 = 60 ms
• Sorted latencies = 40, 45, 50, 65, 100 ms
• Maximum = 100 ms

With only five measurements, percentile estimates are unstable. For production p50, p95, and p99, collect a much larger, representative sample and state the percentile calculation method.

Recommended instrumentation

User query
   ↓
Query normalization
   ↓
[Start embedding timer]
   ↓
Embedding API or local model inference
   ↓
Embedding vector returned
[Stop embedding timer]
   ↓
Vector database search
   ↓
Reranking → LLM generation

Record a trace or span with:

• Request/trace ID and model name/version
• Input token or character count
• Cache hit/miss
• Batch size, if relevant
• Duration in milliseconds
• Success, timeout, throttling, retries, and error status
• Region/provider or serving instance, when relevant

Avoid logging raw sensitive queries by default.

How do you measure production performance?

• Monitor p50, p95, p99, mean, error rate, and traffic volume.
• Separate cold-start from warmed-up model behavior.
• Compare short and long queries, cache hits/misses, and concurrency levels.
• For remote embedding APIs, distinguish client-observed latency from provider inference time where possible.
• Measure document ingestion throughput as chunks or tokens embedded per second, and track ingestion queue delay separately.
• Use consistent measurement boundaries when comparing embedding providers or model versions.

Common mistakes

• Timing the entire RAG pipeline and calling it embedding latency.
• Mixing cache hits with misses without reporting the mix.
• Ignoring retries or rate limits that raise user-perceived latency.
• Using only average latency while high-percentile delays hurt user experience.
• Comparing models using different query sizes or different network conditions.

Interview takeaway: Measure embedding latency as a clearly bounded query-to-vector operation, trace it independently, report meaningful latency percentiles, and monitor cache effects and ingestion throughput separately.