What is the difference between batch, stochastic, and mini-batch gradient descent?

The difference is how much training data is used to compute each gradient update.

Batch gradient descent

Uses the entire training dataset before making one parameter update.

All training examples → Gradient → One update

It produces a stable gradient estimate, but each update can be slow and memory-intensive for large datasets.

Stochastic gradient descent

Uses one training example at a time for each update.

1 example → Gradient → Update

Updates are very frequent, but the gradient is noisy because it is based on only one example.

Mini-batch gradient descent

Uses a small group of examples for each update.

Small batch → Gradient → Update

For example, with a batch size of 64, the model computes a gradient using 64 examples before updating its parameters.

Mini-batch training is the most common approach in modern deep learning because it offers a practical balance between:

• Computational efficiency
• Memory usage
• Gradient stability
• Frequent parameter updates

For the meaning of batch size, see “What is batch size in deep learning?”. For the underlying optimization idea, see “What is gradient descent?”.

In short, batch gradient descent uses all examples per update, stochastic gradient descent uses one example, and mini-batch gradient descent uses a small subset of examples.