A junior estimates LLM training FLOPs as 2 × parameters × tokens. What backpropagation work is missing, and why is the usual dense-Transformer estimate about 6 × parameters × tokens?

2ND is a common approximation for the forward pass, not the full training step. For a dense Transformer, a useful first-order training compute estimate is 6ND FLOPs, where N is model parameters and D is training tokens. The backward pass typically involves two distinct matrix-gradient computations, adding approximately 4ND beyond the 2ND forward pass.

1. Where the missing FLOPs go

For a linear layer Y=XW:

• Forward: form Y by matrix multiplication, roughly 2 FLOPs per weight contribution per token → about 2ND across parameter-heavy model layers.
• Backward—gradient with respect to weights: compute dW=XᵀdY, roughly another 2ND. It tells the optimizer how to update the trainable parameters.
• Backward—gradient with respect to inputs: compute dX=dY Wᵀ, roughly another 2ND. It sends learning signals backward toward earlier layers.
• Total: 2ND + 2ND + 2ND ≈ 6ND.

The two backward calculations are different: without dW, the weights cannot learn; without dX, preceding layers do not get their gradients. This shorthand applies to the bulk of parameter-dominated dense Transformer matrix multiplications, not literally every single operation.

2. Beginner-friendly numerical example

Assume a 7-billion-parameter dense LLM trained on 1 trillion tokens:

• Forward: 2 × 7×10⁹ × 10¹² = 1.4×10²² FLOPs.
• Weight gradients: another 1.4×10²² FLOPs.
• Input/activation gradients: another 1.4×10²² FLOPs.
• Total ≈ 4.2×10²² FLOPs.

The 2ND estimate is three times too small. Equivalently, correcting the estimate adds 200% of the original estimate. Describing this as “300% off” is ambiguous: the correct total is 300% of the initial estimate, not 300% additional compute; measured against the correct total, the underestimate is 66.7%.

3. The derivation in a tiny linear layer

Let an input row have 4 features and W map 4 inputs to 3 outputs. The forward multiplication uses roughly 2×4×3=24 FLOPs per token. In reverse mode:

• dW multiplies the 4 inputs by 3 output gradients: about 24 FLOPs per token.
• dX multiplies the 3 output gradients through the transposed weights to 4 inputs: about 24 FLOPs per token.
• Approximate combined cost: 72 FLOPs, three times forward alone.

This ignores boundary effects, bias, nonlinearity and small operations to isolate the core arithmetic.

4. Important limitations of 6ND

• Attention sequence-length term: the quadratic attention score/value operations can matter at long contexts and need extra FLOPs beyond a simplistic parameter-count formula.
• Embedding and output layers, normalization and optimizer steps: exact work depends on architecture, activation checkpointing, fused kernels and what is counted.
• Mixture of Experts: total parameter count overstates compute if only a subset of experts is active per token; use a model-specific accounting.
• Activation checkpointing: recomputation adds forward work, so the actual training FLOPs can exceed 6ND.
• GPU execution time isn't identical to mathematical FLOPs: hardware utilization, parallelism, communication and memory traffic affect achievable throughput.

5. Translate compute to a budget

Once approximate model FLOPs C are known, estimate GPU-hours using GPU-hours ≈ C / (number_of_GPUs × effective_FLOPs_per_second_per_GPU × 3600) × number_of_GPUs, which simplifies to C / (effective_FLOPs_per_second_per_GPU × 3600). Effective performance should be based on measured throughput or a realistic model FLOPs utilization factor, not advertised peak performance alone. Divide total GPU-hours by the allocated GPU count for elapsed training hours; add engineering headroom for evaluation, data loading, checkpoints, communication and failures.

6. Interview-ready answer

“2ND covers mostly the forward multiplication. Backprop needs both gradients with respect to weights and layer inputs, costing approximately 2ND each. That gives the dense-Transformer budgeting rule 6ND, three times the junior's estimate. I'd then refine the budget for long-context attention, recomputation, MoE sparsity and measured GPU efficiency.”

Research: Hoffmann et al., Training Compute-Optimal Large Language Models (2022); NVIDIA Megatron Core benchmarks.