You’re A/B testing two 70B models—one Multi-Head Attention (MHA), one Grouped Query Attention (GQA). Your colleague argues they’ll have the same inference speed since FLOPs and parameter counts are identical. Is this assumption correct?

No. Equal parameter counts or FLOPs do not imply equal inference speed—and ordinary MHA and GQA architectures do not necessarily have identical parameters or FLOPs in the first place. For autoregressive generation, the crucial difference is often how many key/value (KV) vectors the GPU must store and read from memory.

1. What changes between MHA and GQA?

• MHA: every query head has its own key and value head.
• GQA: multiple query heads share each key/value head. Query heads can still remain separate.
• For example, 64 query heads and 8 KV heads means each KV head serves 8 query heads. With MHA, all 64 query heads have distinct KV heads.
• The Q projection and output projection may be similar, but GQA generally makes the K and V projection matrices smaller. Therefore the premise of literally equal parameter counts is not generally true.

2. Why memory traffic matters more than just FLOPs

During token-by-token decode, the model repeatedly reads previously cached keys and values. At low-to-moderate batch sizes or long contexts, this attention work can be limited by GPU memory bandwidth, rather than peak arithmetic throughput.

GQA reduces the KV cache and the amount of KV data potentially read per decoding step. With a suitable attention kernel, the KV data is shared across several query heads without repeatedly materializing full copies. Less memory traffic can improve tokens/second and allow more simultaneous requests.

Important: The theoretical KV-byte reduction is not a guaranteed 8× end-to-end speedup. Model-weight reads, feed-forward layers, kernel overhead, GPU utilization, batching and interconnect traffic still matter.

3. Beginner-friendly numerical example

Assume both hypothetical models have:

• 80 Transformer layers
• 64 query heads, each of dimension 128
• BF16 KV entries (2 bytes each)
• A 4,096-token context for one sequence

MHA, 64 KV heads: KV bytes per token across layers = 2 (K and V) × 80 × 64 × 128 × 2 = 2,621,440 bytes, or 2.5 MiB. At 4,096 tokens, the KV cache is 10 GiB.

GQA, 8 KV heads: 2 × 80 × 8 × 128 × 2 = 327,680 bytes, or 0.3125 MiB per token. At 4,096 tokens, the KV cache is 1.25 GiB.

That is an 8× smaller KV cache, not an 8× smaller whole-model memory footprint. The calculation excludes allocator fragmentation, metadata, weights and temporary buffers. The 70B model label does not determine these KV dimensions by itself.

4. Does GQA always win?

No.

• On long-context decode or when KV bandwidth/capacity is the constraint, GQA often helps considerably.
• On short prompts, small caches, compute-bound prefill, or where optimized kernels already hide memory traffic, the benefit may be smaller.
• Different K/V projection sizes, kernels and learned weights complicate any claim of equal accuracy or identical FLOPs.
• Changing an MHA checkpoint to GQA is not merely changing a runtime flag; conversion/uptraining or retraining is generally needed, and model quality must be evaluated.

5. How I would run the A/B test

1. Control for confounders: same GPU, precision, serving engine/kernel version, parallelism, prompt/output distributions, and comparable model quality. Record actual parameters and arithmetic instead of assuming equality.
2. Benchmark prefill and decode separately, across 1k/4k/32k contexts and increasing concurrency.
3. Measure prefill TTFT, decode inter-token latency, output tokens/sec, KV cache usage, HBM bandwidth utilization, GPU compute utilization, and p95/p99 latency.
4. Make a product decision using latency SLOs, throughput, capacity and task quality—not FLOPs alone.

Interview-ready answer

“The assumption is wrong. GQA shares fewer K/V heads across many Q heads, often lowering KV-cache memory and read bandwidth substantially. Decode can be memory-bound, so two models with similar FLOPs can have different real-world throughput and latency. I would benchmark decode at realistic contexts and concurrency, while checking model quality, hardware and kernel efficiency. An 8× KV reduction is not an 8× end-to-end speed guarantee.”

Sources: Ainslie et al., GQA: Training Generalized Multi-Query Transformer Models from Multi-Head Checkpoints; Hugging Face Transformers, LLM optimization—GQA; Hardware-Efficient Attention for Fast Decoding (COLM 2025).