How does batch size affect GPU memory usage?

Batch size directly affects GPU memory usage because larger batches require the GPU to store more input data, activations, and intermediate results at the same time.

Why does a larger batch use more memory?

During training, the GPU must store information for every example in the current batch, including:

• Input tensors.
• Intermediate activations from each layer.
• Temporary computation results.
• Gradients needed for backpropagation.

When the batch size increases, these per-example memory requirements increase as well.

Simple example

Suppose one training example requires a certain amount of activation memory.

If:

batch size = 32

the model stores activations for 32 examples.

If the batch size increases to:

batch size = 128

it must store activations for four times as many examples during that step.

The exact memory increase depends on the model and framework, but activation memory often grows roughly with batch size.

What happens when GPU memory is exhausted?

If the required memory exceeds the available GPU memory, training may fail with an out-of-memory (OOM) error.

A common solution is to reduce the batch size.

Does batch size affect all GPU memory?

Not all GPU memory scales with batch size.

Memory used by:

• Model parameters.
• Some optimizer states.
• Certain framework overheads.

is largely independent of batch size.

However, activation memory and other per-example tensors usually increase as the batch grows.

What if a larger effective batch is needed?

If memory limits prevent using a large physical batch, gradient accumulation can process several smaller mini-batches before performing one optimizer update.

For that approach, see “What is gradient accumulation?”.

In short, larger batches usually consume more GPU memory because the network must keep activations and intermediate data for more examples in memory at the same time.