What is gradient accumulation?

Gradient accumulation is a training technique where gradients from several smaller mini-batches are added together before the optimizer updates the model weights.

It is commonly used when the desired batch size is too large to fit into available GPU memory.

How does it work?

Normally, training follows this pattern:

Mini-batch → Compute gradients → Update weights

With gradient accumulation, the process becomes:

Mini-batch 1 → Compute gradients

Mini-batch 2 → Add gradients

Mini-batch 3 → Add gradients

Mini-batch 4 → Add gradients → Update weights

The optimizer updates the weights only after the chosen number of accumulation steps.

Effective batch size

If:

mini-batch size = 16

and:

accumulation steps = 4

then the effective batch size is approximately:

16 × 4 = 64

This allows training to behave similarly to using a larger batch while processing only a smaller number of examples at a time.

Why is gradient accumulation useful?

It can help when:

• GPU memory is limited.
• A larger effective batch size is desired.
• The model itself already consumes most of the available memory.

Important detail

The loss is often divided by the number of accumulation steps before backpropagation so the accumulated gradient has the intended scale.

Also, gradients must not be cleared after every mini-batch during accumulation. They are cleared after the optimizer update.

What is the trade-off?

Gradient accumulation reduces memory pressure, but it does not make computation free.

Several forward and backward passes are still required before each optimizer step, so training may take longer in wall-clock time.

The next article covers when this technique is useful in “When should gradient accumulation be used?”.

In short, gradient accumulation simulates a larger batch by summing gradients across multiple smaller mini-batches before performing a weight update.