What does the backpropagation algorithm do in neural networks? What are its drawbacks?

Backpropagation computes the gradient of a neural-network loss with respect to parameters using the chain rule. It walks backward through the differentiable computation graph and supplies gradients to an optimizer. The optimizer—not backpropagation—updates weights and navigates the loss surface. Therefore it is incorrect to assert that "backpropagation itself gets stuck in local minima." Nonconvex optimization, unsuitable objectives and optimizer choices can make training difficult even when gradients are computed correctly.

This supplemental lesson focuses on practical drawbacks and failure diagnosis, rather than reteaching the mechanics already covered in Deep Learning Questions 22, 23 and 137. Individual vanishing/exploding-gradient lessons 37–38, initialization lessons 55–57, gradient clipping lessons 73/144 and activation checkpointing lesson 296 provide prerequisite details.

1. What does the algorithm do?

• Forward: Inputs flow through layers to predictions and a scalar loss.
• Backward: Reverse-mode automatic differentiation follows the graph and multiplies local derivatives to calculate ∂L/∂w and other required gradients.
• Optimize: SGD, Adam or another optimizer updates model parameters according to gradients, learning rates and optimizer state.
• Evaluate: Training and validation metrics reveal whether optimization produces a useful model. Correct gradients alone cannot guarantee generalization.

Where can training struggle? A deep computation graph can have vanishing/exploding sensitivities, numerical overflows, non-differentiable operations, large saved activation memory, and implementation errors. An optimizer can also make poor progress even with perfectly valid gradients.

2. The main drawbacks—without confusing causes

Vanishing gradients: Products of small local derivatives can leave earlier layers with tiny learning signals. Saturating activations and poorly conditioned weight matrices can contribute. See Question 37 for the standalone definition.

Exploding gradients: Products of large derivatives can produce exceptionally large norms or nonfinite values. See Question 38 for the basic mechanism and Questions 73/144 for clipping.

Compute and memory: Backward must revisit the graph and often needs saved forward activations. Deeper networks, long sequences or large input tensors can be GPU-memory-bound despite efficient reverse-mode differentiation. Activation checkpointing trades extra forward recomputation for less memory (Question 296).

Differentiability: Hard argmax, thresholding, external discrete decisions or intentionally detached tensors often have no useful ordinary gradient. A differentiable surrogate or stochastic gradient estimator may be required; these introduce approximations and sometimes high variance.

Numerical precision: Large intermediates may overflow to Inf/NaN; tiny derivatives may underflow in FP16. FP32 accumulation and mixed-precision scaling can help in appropriate cases; Questions 293–295 explain the dedicated procedures.

Optimizer and objective: Plateaus, saddle regions, poor loss conditioning, wrong learning rates and noisy minibatches can delay progress. None is proof that backpropagation's chain rule is incorrect or that it intrinsically becomes trapped in a local minimum.

Data and model quality: Poor labels, leakage, distribution shifts and inadequate architecture can cause bad deployed predictions even when training loss decreases and all derivatives are correct.

3. Numerical example: An eight-layer gradient path

For a deliberately simple network with eight scalar linear layers, let:

h₀=x=1; h₁=w₁h₀; …; h₈=w₈h₇.

At the particular training point, assume the downstream loss supplies ∂L/∂h₈=1. This is a *local derivative assumption* used solely to compare the two chains, not a claim about every loss.

By the chain rule:

∂L/∂h₀ = (∂L/∂h₈) × (w₁w₂…w₈).

Case A: Every layer weight is 0.5

• Output: h₈=0.5⁸=0.00390625.
• Input-side gradient: ∂L/∂h₀=0.5⁸=0.00390625.
• First parameter's derivative: ∂L/∂w₁=0.5⁷=0.0078125.
• This earlier layer receives a comparatively weak signal despite a unit upstream derivative.

Case B: Every layer weight is 1.5

• Output: h₈=1.5⁸=25.62890625.
• Input-side gradient: ∂L/∂h₀=1.5⁸=25.62890625.
• First parameter's derivative: ∂L/∂w₁=1.5⁷=17.0859375.
• Now a large gradient can encourage a destabilizing optimizer step.

Separate the derivative from the optimizer update.

Under *illustrative SGD* with learning rate η=0.01:

• Case A first-weight update: −0.01 × 0.0078125 = −0.000078125.
• Case B first-weight update: −0.01 × 17.0859375 = −0.170859375.

If the large-gradient case involved only one trainable scalar parameter, clipping its gradient norm to 1 would make its update approximately −0.01. In a real network, global norm clipping uses the combined norm of all parameter gradients, not a separate cap on each coordinate.

Why this is an example, not a universal rule: Real networks have nonlinearities, residual paths, normalization, multiple paths and nonuniform weights. The takeaway is to monitor layerwise gradient health and use the right mitigation, not to assume all 8-layer networks fail.

4. Numerical example: Backpropagation memory

Suppose a model works with one FP32 hidden tensor of shape [batch=32, sequence=1024, width=2048].

• Element count: 32 × 1024 × 2048 = 67,108,864.
• Bytes: 67,108,864 × 4 = 268,435,456 bytes = 256 MiB.
• Twelve simultaneously retained tensors of this exact size: 12 × 256 MiB = 3,072 MiB = 3 GiB.

This is an illustrative saved-tensor budget, not actual measured Transformer memory. Operations can save a variable number of tensors; model weights, optimizer moments, gradients, attention workspaces and CUDA allocation reserves add other costs.

Mitigations: Profile peak allocated/reserved memory; try smaller microbatches, shorter sequences, selective activation checkpointing or suitable mixed precision. Checkpointing lowers saved activation usage but costs extra recomputation, while gradient accumulation changes effective batch scheduling. Do not assume any one technique will solve all memory problems.

5. Production debugging incident: Training stops improving

Scenario: A team expands a 4-block sequence anomaly classifier to 12 blocks. Several training runs plateau; a few batches cause NaN loss, and long inputs sometimes trigger out-of-memory errors. The following illustrates diagnostic reasoning, not a real MachinoAI production incident.

Symptoms to observe

1. Early-layer gradients are much smaller than final-layer gradients.
2. Some activation histograms show saturation, especially in bounded nonlinearities.
3. Occasional large gradient norms precede training instability.
4. A custom feature transform sometimes returns a detached tensor, causing missing gradients.
5. Memory demand rises with input sequence length.

Root-cause investigation

1. Validate data and objective: Confirm input normalization, label mapping, loss reduction, outliers and missing values. Correct backprop on the wrong objective is still the wrong training signal.
2. Inspect graph connectivity: Check requires_grad, grad_fn, the presence of .grad, and unexpected .detach(), .item() or no_grad operations.
3. Record gradients before clipping: Compare early versus late layer norms, zero fractions and nonfinite rates over multiple batches. A None gradient does not necessarily equal a genuine zero derivative.
4. Inspect activations: Test whether saturation, initialization, normalization or ineffective residual paths explain the weak signals.
5. Isolate first nonfinite backward operation: Temporarily enable torch.autograd.detect_anomaly(check_nan=True) on a small reproduction.
6. Verify custom derivatives: Use torch.autograd.gradcheck with double-precision inputs and finite differences for a small smooth operation.
7. Separate gradient failure from optimizer failure: Compare the unmodified gradient, the clipped gradient and the actual parameter-update norm; review learning rate and optimizer state.
8. Profile GPU memory: Distinguish saved activations from model weights, gradients, optimizer state and temporary kernels.
9. Validate the intervention: Compare training/validation curves, gradient statistics, memory and throughput on the same held-out and production-like data.

Engineering fixes may include correcting graph detachments, better normalization or initialization, residual pathways, a safer learning rate, norm clipping for *large* gradients, checkpointing activation-heavy blocks or reducing microbatch length. None should be applied blindly to every symptom.

Prevention: Add automated small-batch gradient checks for custom ops, per-layer gradient telemetry, NaN alerts, fixed-seed training regressions and representative memory/latency checks to the model release workflow.

6. Practical PyTorch gradient-health inspection

A minimal debugging step for a normal, unscaled training loop:

PYTHON
import torch
from torch.nn.utils import clip_grad_norm_

model.train()
optimizer.zero_grad(set_to_none=True)

# Debugging only; anomaly checks slow down training.
with torch.autograd.detect_anomaly(check_nan=True):
    outputs = model(inputs)
    loss = criterion(outputs, targets)
    loss.backward()

for name, p in model.named_parameters():
    if not p.requires_grad:
        continue
    if p.grad is None:
        print(name, "NO_GRAD: check unused/disconnected path")
        continue
    g = p.grad.detach()
    print(name, {
        "grad_norm": float(g.float().norm()),
        "finite": bool(torch.isfinite(g).all()),
        "zero_fraction": float((g == 0).float().mean()),
    })

# Modifies gradients; inspect the original norms first.
clip_grad_norm_(model.parameters(), max_norm=1.0,
                error_if_nonfinite=True)
optimizer.step()

Implementation notes

• Requires a defined model, criterion, optimizer, inputs and targets.
• Anomaly detection should be used on short reproductions, not always-on full production training.
• Parameters in legitimately unused conditional branches can have .grad=None without a bug.
• If gradient accumulation is intentional, zero gradients only at the correct optimizer-step boundary.
• If using AMP `GradScaler`: Unscale gradients before monitoring/clipping and step through the scaler; do not call bare optimizer.step() on still-scaled gradients (Question 295).
• Don't use retain_graph=True or store graph-connected tensors without a reason: they can retain saved activations and increase memory.

Check a custom differentiable operation using double precision

PYTHON
x = torch.tensor([0.2, -0.7, 1.1],
                 dtype=torch.double, requires_grad=True)

def smooth_example(z):
    return (z.sin() * z).sum()

ok = torch.autograd.gradcheck(
    smooth_example, (x,), eps=1e-6, atol=1e-5, rtol=1e-3
)
print("gradient verified on this input:", ok)

Passing gradcheck confirms consistency between derivatives and finite differences at the tested smooth points; it does not prove all derivatives or the whole model are always correct. Its default numerical settings are designed for double precision, not FP16.

7. Map the symptom to the right mitigation

• Tiny early gradients: Check long Jacobian products, saturating activations, initialization, residual connections and graph detachments. Clipping cannot amplify vanished signals.
• Exploding gradients / Inf / NaN: Check outliers, loss scale, numerical stability, learning rate and global gradient clipping; trace the earliest failing backward operation.
• Missing `.grad` values: Verify trainability, graph continuity, unused branches, discrete decisions and accidental detach/no_grad.
• Backward out-of-memory: Measure activations first, then decide between checkpointing, smaller microbatches, sequence reduction, offloading or precision options.
• Valid gradients but no loss improvement: Investigate optimizer hyperparameters, poorly conditioned objectives, batch construction, model capacity and learning schedule.
• Good training accuracy, bad validation: Investigate overfitting, leakage, domain shift and spurious features—not just gradient math.

Metrics for operations: Per-layer gradient norms and zero fractions, nonfinite rates, optimizer-step counts, loss/validation metrics, peak GPU memory, and end-to-end batch throughput.

8. Common conceptual mistakes and limitations

• Backpropagation computes gradients; the optimizer updates weights.
• Local minima and saddle points are aspects of optimization, not evidence that the chain rule has failed.
• Gradient clipping reduces large gradients; it does not cure vanishing gradients.
• An absent gradient may be a disconnected graph, an unused branch or intentional freezing—not simply a zero derivative.
• A mathematically correct gradient can optimize an inappropriate loss or dataset.
• Finite-difference tests and anomaly detection are diagnostics, not guarantees of correctness.
• In-place tensor operations, stochastic recomputation and untracked Python-side changes can invalidate assumptions about backward consistency.
• Reverse-mode differentiation is efficient relative to naive parameter-by-parameter finite differences, but saved activation memory and backward compute can be significant.

Semantic deduplication / curriculum positioning: Backpropagation basics appear in Questions 22/23/137. Vanishing/exploding gradients are 37/38, initialization 55–57, clipping 73/144, activation checkpointing 296. This article is distinct because it connects those topics through one production-oriented root-cause workflow, numerical gradient/memory calculations, graph-connectivity debugging and mitigation selection. It is a supplemental article after the completed 1–300 curriculum, not a new entry in the original 201–300 proposal.

Key takeaway: Backpropagation is an efficient derivative engine, not a complete training strategy. Its practical constraints include gradient conditioning, graph differentiability, numerical precision, backward compute and saved activations. Diagnose these separately from optimizer choices and model/data correctness.

Sources: PyTorch Autograd Mechanics; PyTorch Autograd and anomaly detection; Glorot & Bengio, Understanding the difficulty of training deep feedforward neural networks; Pascanu, Mikolov & Bengio, On the difficulty of training Recurrent Neural Networks; He et al., Deep Residual Learning for Image Recognition; Chen et al., Training Deep Nets with Sublinear Memory Cost; PyTorch gradcheck; PyTorch clip_grad_norm_.