What are sharp and flat minima in neural network optimization?

Sharp and flat minima describe how quickly a neural network's training loss increases when its learned weights move slightly away from a low-loss point. A sharp minimum resembles a narrow valley with steep sides; a flat minimum resembles a wider valley where nearby weights have similar loss.

What does sharp versus flat mean?

• Minimum: A set of model weights where the loss is locally low.
• Sharp minimum: Small changes to the weights can increase the loss substantially in one or more directions.
• Flat minimum: The loss changes more gradually across a specified neighborhood of the weights.
• Important: Sharpness describes the *surroundings* of a minimum, not necessarily its loss value. Two minima can have the same training loss but different local shapes.

Neural networks have millions of parameters, so the real loss landscape has many directions. A minimum can be steep in one direction and gentle in another; “sharp” and “flat” are useful descriptions, not always simple yes/no labels.

Beginner-friendly numerical example

Imagine two different low-loss regions of a model. Near each region's own minimum, approximate its loss along one comparable weight direction. Let δ be a small change to that weight coordinate.

• Flat region: L_flat(δ) = 0.10 + 0.5δ²
• Sharp region: L_sharp(δ) = 0.10 + 5δ²

Step 1 — At the minimum (`δ = 0`):

• Flat: 0.10 + 0.5 × 0² = 0.10
• Sharp: 0.10 + 5 × 0² = 0.10

Both minima start with exactly the same loss, 0.10.

Step 2 — Apply the same small perturbation (`δ = +0.2`):

• Flat: 0.10 + 0.5 × (0.2)² = 0.12
• Sharp: 0.10 + 5 × (0.2)² = 0.30

Step 3 — Compare the loss increase:

• Flat: 0.12 − 0.10 = 0.02
• Sharp: 0.30 − 0.10 = 0.20

Under the same 0.2 parameter perturbation, the sharp region's loss rises 10 times as much in this toy example. The curves are symmetric here, so a change of −0.2 gives the same increases. Actual neural-network loss surfaces need not be symmetric or quadratic.

How does curvature relate to sharpness?

• Higher local curvature: The loss bends upward quickly as you leave a minimum; the location is sharper along that direction.
• Lower local curvature: The loss rises more slowly; the location is flatter along that direction.
• In the example: The second derivatives are 1 for the flat region and 10 for the sharp region. In models with many weights, different directions can have different curvatures. The Hessian summarizes these second-order effects; its detailed computation is a separate topic.

Why might flatter minima generalize better?

• Intuition: If small changes to the parameters cause only small loss increases, the model may be less sensitive to some training noise or parameter perturbations.
• Research observation: Some studies, including work on large-batch training, found relationships between sharper solutions and worse test performance.
• But not a guarantee: Training-loss flatness does not automatically predict test accuracy. The relationship depends on what “flat” means, how it is measured, and the model and data.

Important limitations of the flat-minima hypothesis

• Parameterization dependence: The same prediction function can have different apparent sharpness when its weights are rescaled. For a simple bias-free two-layer ReLU network, replacing u with c·u and v with v/c for c > 0 leaves v·ReLU(u·x) unchanged, yet changes the geometry measured in raw weight coordinates.
• Choice of scale and direction: Perturbations must be compared with a consistent metric and neighborhood size. A measure that looks flat along one direction can miss steep directions elsewhere.
• Correlation is not causation: A sharp-looking solution can still generalize well, as shown by theoretical counterexamples. Compare actual held-out validation or test performance, not just a flatness score.

For the training algorithm motivated by nearby loss increases, see “What is sharpness-aware minimization (SAM)?” (Question 256); this article focuses on the loss-landscape concept, not SAM's gradient steps.

Key takeaway: Sharp minima have rapidly rising loss nearby; flat minima have a gentler surrounding loss landscape. Flatness can be informative, but by itself is not a reliable guarantee of generalization.

Research: Keskar et al. (ICLR 2017), *On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima*; Dinh et al. (ICML 2017), *Sharp Minima Can Generalize For Deep Nets*.