How does stochastic depth differ from dropout?

Dropout and stochastic depth (DropPath) both regularize neural networks by randomly removing contributions during training, but they operate at different levels: ordinary dropout masks individual activation values, while DropPath masks an entire learned residual branch. The residual shortcut remains available even if its branch is dropped.

Key differences

• What gets removed? Ordinary dropout sets selected activation elements to zero. DropPath removes all components of one residual-branch output together for a given training sample.
• Mask granularity: Standard dropout independently samples masks for individual elements. Modern DropPath typically uses one keep/drop decision per sample per residual branch, broadcast across the branch's channels and positions. Some implementations support a batch-wide mask.
• Where used? Dropout is broadly applicable to layers in different networks. DropPath requires a suitable bypass, so it is common in deep residual CNNs and Vision Transformers.
• What changes during training? Dropout changes which individual features are available; DropPath changes which residual transformations contribute, producing effectively shorter paths.
• What happens at inference? Both are normally turned off: all activation elements stay available and every residual branch contributes.
• How is scaling handled? Common *inverted* implementations divide surviving elements or branches by the keep probability during training. Then evaluation uses the unmasked values or branches without extra scaling. The original stochastic-depth paper used a different convention, covered in “What is stochastic depth (DropPath) in deep networks?”.

Beginner-friendly numerical example

Use a 50% drop probability for each method (keep probability q = 0.5). These are chosen random outcomes: a 50% rate does not force exactly half the elements to disappear in every small example.

1. Dropout — remove selected values

Suppose an activation vector is a = [2, 4]. The sampled element-wise keep mask is m = [1, 0].

• Training: a_train = (m / q) ⊙ a = [4, 0]. The first element stays, scaled by 1/0.5 = 2; the second becomes zero.
• Inference: a_eval = [2, 4]. Nothing is randomly masked.

2. Stochastic depth — remove the whole residual contribution

Consider a residual block receiving x = [2, 4], whose learned branch produces F(x) = [1, 3]. DropPath makes one binary choice b for the entire branch:

• Branch dropped (`b = 0`): y_train = x + (b / q)F(x) = [2, 4]. The shortcut still passes x.
• Branch kept (`b = 1`): y_train = [2, 4] + 2[1, 3] = [4, 10]. Both branch components participate together.
• Inference: y_eval = x + F(x) = [3, 7], with the full branch active.

The two equally likely DropPath outcomes have expected output 0.5[2, 4] + 0.5[4, 10] = [3, 7]. Inverted scaling preserves the expected contribution, not the exact output of every training pass.

Notice: Dropout removed a *single activation element*; DropPath removed the *entire residual contribution* but kept the shortcut. Both can have the same numerical drop probability while applying very different masks.

When should you choose each?

• Dropout: Useful when feature-level randomness helps prevent overreliance on particular activations; it is common in fully connected and other network layers.
• DropPath: Useful in sufficiently deep residual networks where bypassing whole transformations during training can support regularization and optimization.
• Both together? Possible when an architecture supports them, but not automatically better. Excessive dropping of either kind can cause underfitting; validate rates and performance rather than assuming faster training or higher accuracy.

Important nuance: This compares DropPath with ordinary element-wise dropout. Spatial dropout removes entire channels; it still does not necessarily drop a complete *residual branch*. See the existing “What is spatial dropout?” article for that separate method.

For each mechanism in depth, see “What is dropout in deep learning?”, “What happens to dropout during inference?”, and “What is stochastic depth (DropPath) in deep networks?”.

Key takeaway: Dropout masks activation elements; stochastic depth masks residual paths. Both are usually training-only regularizers, but their masks operate at different structural scales.

References: Huang et al. (2016), *Deep Networks with Stochastic Depth*; PyTorch `nn.Dropout` documentation; Torchvision stochastic depth documentation.