Deep Learning

Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.

What is the difference between weight decay and L2 regularization?

Weight decay and L2 regularization both encourage smaller weights, but they are applied differently during optimization. With plain gradient descent, they can behave equivalently under common settings. With adaptive optimizers such as Adam, they are generally not the same. How does L2 regularization work? L2 regularization adds a penalty based on squared weights to the loss: L total = L data + λ × Σw² During backpropagation, this penalty contributes an extra gradient term proportional to the weight: gradient total = gradient data + λ × w The optimizer then processes this modified gradient normally. How does weight decay work? Weight decay directly shrinks the parameter during the update. A simplified form is: w new = w old learning rate × gradient learning rate × weight decay × w old So the shrinkage is applied separately from the loss gradient. Why are they equivalent with standard SGD? With ordinary SGD, adding an L2 gradient term and directly shrinking the weight can produce the same update after matching the coefficients appropriately. That is why the terms are sometimes used interchangeably. Why are they different with Adam? Adam rescales gradients separately for each parameter using adaptive moment estimates. If an L2 penalty is added to the gradient, that penalty is also transformed by Adam's adaptive scaling. With decoupled weight decay, the shrinkage is applied separately and is not distorted by that gradient scaling. Where does AdamW fit in? AdamW explicitly uses decoupled weight decay . This makes the regularization effect easier to control independently from Adam's adaptive gradient update. For the optimizer itself, see “What is AdamW?” . For the basic regularization concepts, see “What is L1 and L2 regularization?” and “What is weight decay in neural networks?” . In short, L2 regularization modifies the loss and therefore the gradient, while weight decay directly shrinks the parameters; they can be equivalent with SGD but differ with adaptive optimizers such as Adam.

How does dropout reduce overfitting in neural networks?

Dropout reduces overfitting by randomly disabling a fraction of neurons during training, forcing the network to avoid relying too heavily on any single neuron or feature. How does dropout work? During each training step, some neuron outputs are randomly set to zero. For example, with: dropout rate = 0.5 about half of the selected neurons are temporarily ignored during that training pass. The exact neurons dropped change from one mini batch to another. Why does this reduce overfitting? Without dropout, groups of neurons can become overly dependent on one another and memorize patterns specific to the training data. Dropout breaks those dependencies. Because any neuron may disappear during training, the remaining neurons must learn representations that are useful in many different combinations. What effect does this have? Dropout encourages the network to: Learn more robust features. Reduce co adaptation between neurons. Spread useful information across multiple neurons. Generalize better to unseen data. Simple intuition Imagine a team where different members are randomly absent during practice. The team cannot depend on one specific person every time, so everyone must learn to contribute independently. Dropout creates a similar effect inside a neural network. Is dropout used during inference? No. During inference, all neurons are normally active. The activations are scaled consistently so predictions reflect the behavior learned during dropout training. For the basic definition, see “What is dropout in deep learning?” . The next article explains inference behavior in more detail: “What happens to dropout during inference?” . In short, dropout reduces overfitting by randomly removing neurons during training, preventing excessive dependence between neurons and encouraging more robust, generalizable features.

What happens to dropout during inference?

During inference, dropout is normally turned off so all neurons remain active when the model makes predictions. Dropout is mainly a training time regularization technique. What happens during training? During training, dropout randomly sets some neuron outputs to zero. For example, with: dropout rate = 0.5 roughly half of the selected activations are dropped on each training pass. This randomness helps reduce overfitting. What changes during inference? During inference: No neurons are randomly dropped. The full network is used. Predictions become deterministic for a fixed input, apart from other stochastic operations. What about activation scaling? Most modern frameworks use inverted dropout during training. With inverted dropout, the activations that remain active are scaled during training so that no additional scaling is needed during inference. For example, if the keep probability is: 0.5 the surviving activations are typically scaled by: 1 / 0.5 = 2 during training. This keeps the expected activation magnitude similar between training and inference. Why is dropout disabled at inference time? The goal during inference is to use the complete learned model rather than randomly removing parts of it. Keeping dropout active would normally make predictions vary from one forward pass to another. Is dropout ever intentionally used during inference? Yes, in specialized techniques such as Monte Carlo dropout , dropout can remain enabled across multiple inference passes to estimate predictive uncertainty. However, standard inference disables it. For the training time role of dropout, see “How does dropout reduce overfitting in neural networks?” . In short, dropout is normally disabled during inference, all neurons are used, and modern inverted dropout implementations require no extra activation scaling at prediction time.

What is spatial dropout?

Spatial dropout is a dropout technique that removes entire feature maps or channels instead of dropping individual activation values. It is commonly used in convolutional neural networks, where nearby activations within the same feature map are often highly correlated. Why not use ordinary dropout? Standard dropout randomly sets individual activation values to zero. In convolutional layers, neighboring pixels in a feature map may contain very similar information. Dropping only a few individual values may therefore have a limited regularization effect because nearby activations can still carry almost the same feature. How does spatial dropout work? Suppose a convolutional layer produces several feature maps: Feature map 1, Feature map 2, Feature map 3, Feature map 4 With spatial dropout, training might temporarily remove an entire feature map: Feature map 1, Feature map 2, DROPPED, Feature map 4 The dropped channel is set to zero across all of its spatial positions for that training pass. Why does this help? Removing whole feature maps forces the network to avoid depending too heavily on any single learned feature. This can: Reduce overfitting. Encourage different filters to learn useful information. Improve robustness when feature maps are strongly correlated. Where is it commonly used? Spatial dropout is mainly useful in convolutional architectures processing data such as: Images. Video. Spatial or temporal feature maps. Frameworks may provide variants such as SpatialDropout, Dropout2d, or Dropout3d, depending on the tensor structure. What happens during inference? Like standard dropout, spatial dropout is normally disabled during inference, so all feature maps are available for prediction. For the basic regularization idea, see “What is dropout in deep learning?” and “How does dropout reduce overfitting in neural networks?” . In short, spatial dropout regularizes convolutional networks by randomly removing entire feature maps during training rather than individual activation values.

What is label smoothing in neural network training?

Label smoothing is a regularization technique that replaces perfectly hard class labels with slightly softened target probabilities during training. Instead of telling the model that the correct class must have probability 1 and every other class probability 0, label smoothing assigns a small amount of probability to the other classes. Simple example For a 3 class problem, a hard target might be: [1.0, 0.0, 0.0] With label smoothing, it might become something like: [0.9, 0.05, 0.05] The exact values depend on the smoothing factor. Why is label smoothing used? Hard labels encourage the model to become extremely confident about the training targets. That can make the model: Overconfident. More sensitive to mislabeled examples. More likely to overfit. Label smoothing reduces this pressure by preventing the target distribution from being perfectly one hot. How does it help? It can: Improve generalization. Reduce overconfidence. Make predicted probabilities better calibrated in some settings. Make training more robust to small amounts of label noise. How is the smoothing factor used? If the smoothing value is ε, part of the probability mass is moved away from the correct class and distributed across the classes. A common formulation is: smoothed target = (1 ε) × one hot target + ε × uniform distribution Does it change the model architecture? No. Label smoothing changes the training targets used by the loss function, not the neural network architecture itself. It is commonly used with classification losses such as cross entropy. For the related loss function, see “What is cross entropy loss?” . In short, label smoothing softens one hot targets so the model is discouraged from becoming excessively confident, which can improve regularization and generalization.

How does early stopping work in neural network training?

Early stopping is a regularization technique that stops neural network training when performance on a validation set stops improving. It helps prevent the model from continuing to fit the training data after it has already reached its best generalization point. How does it work? During training, the model is evaluated on a validation dataset after each epoch or at regular intervals. A metric such as validation loss is monitored. For example: Epoch 1: validation loss = 0.62 Epoch 2: validation loss = 0.51 Epoch 3: validation loss = 0.46 Epoch 4: validation loss = 0.48 Epoch 5: validation loss = 0.50 If the validation loss keeps getting worse, training may be stopped. What is patience? Early stopping usually does not stop training after a single bad epoch. A patience value specifies how many evaluations the model can go without improvement before training stops. For example: patience = 3 means training can continue for three checks without a new best validation score. Why is patience useful? Validation metrics naturally fluctuate. Patience prevents training from stopping because of one temporary increase in validation loss. What happens to the model weights? A common implementation saves the weights from the best validation result. When training stops, those best performing weights can be restored instead of keeping the weights from the final epoch. Why does early stopping reduce overfitting? Training loss can continue decreasing even after validation performance begins to deteriorate. Stopping near the best validation point prevents the model from spending additional epochs fitting training specific noise. For the basic concept, see “What is early stopping in deep learning?” . The next article explains a common signal used with early stopping: “What is a validation loss curve?” . In short, early stopping monitors validation performance and ends training when improvement has stalled, usually restoring the model from its best validation point.

What is a validation loss curve?

A validation loss curve is a plot of the model's loss on the validation dataset as training progresses. It helps show whether the model is improving on unseen data or beginning to overfit. What does the curve show? The horizontal axis usually represents: Epochs or training steps The vertical axis represents: Validation loss A healthy training run often shows validation loss decreasing as the model learns useful patterns. Why compare it with training loss? Training loss tells you how well the model fits the training data. Validation loss tells you how well that learning transfers to data the model did not train on. Looking at both curves together is especially useful. What patterns should you look for? If both training and validation loss decrease, the model is generally learning well. If training loss keeps decreasing but validation loss starts increasing, the model is likely beginning to overfit. If both losses remain high, the model may be underfitting or optimization may be poor. Why is the minimum validation loss important? The epoch with the lowest validation loss often represents the point where the model generalizes best. This is why early stopping commonly monitors validation loss and saves the best model checkpoint. Simple example Suppose validation loss changes like this: 0.62 → 0.50 → 0.43 → 0.41 → 0.45 → 0.49 The model improved until the loss reached 0.41, then validation performance began to deteriorate. That turning point may indicate the beginning of overfitting. For the stopping strategy based on this signal, see “How does early stopping work in neural network training?” . The next article explains a common pattern in more detail: “Why can validation loss increase while training loss decreases?” . In short, a validation loss curve shows how model performance on unseen validation data changes during training and is one of the clearest tools for spotting overfitting and choosing when to stop training.

Why can validation loss increase while training loss decreases?

Validation loss can increase while training loss keeps decreasing when the model starts fitting the training data too closely and stops generalizing well to unseen data. This is a classic sign of overfitting . What is happening? During training, the model is optimized directly on the training dataset. So it can continue reducing training loss by learning increasingly specific patterns from that data. Some of those patterns may include noise or details that do not generalize. What happens to validation loss? The validation dataset is not used to update the model. If the model becomes too specialized to the training data, its performance on validation examples can start getting worse. A typical pattern is: Training loss: 0.60 → 0.45 → 0.30 → 0.20 Validation loss: 0.62 → 0.48 → 0.44 → 0.52 Here, training performance keeps improving, but validation performance begins to deteriorate. Why does this indicate overfitting? The growing gap between training and validation loss means the model is becoming better at the data it has seen without becoming better at unseen data. What can you do? Common approaches include: Early stopping. Dropout. Weight decay or other regularization. Data augmentation. More training data. Reducing model complexity when appropriate. Is every small increase a problem? No. Validation loss can fluctuate naturally from one epoch to another. What matters is a sustained upward trend while training loss continues to fall. For the broader concept, see “What is overfitting in deep learning?” and “What is a validation loss curve?” . In short, validation loss can rise while training loss falls because the model is continuing to fit the training set but is starting to generalize worse to unseen data.

How can you tell whether a deep neural network is overfitting?

A deep neural network is likely overfitting when it performs very well on the training data but noticeably worse on validation or test data. The clearest sign is a growing gap between training performance and validation performance. What should you monitor? Track both training and validation metrics during training. For example: Training loss: 0.50 → 0.30 → 0.15 → 0.08 Validation loss: 0.52 → 0.38 → 0.40 → 0.47 Here, training loss keeps improving while validation loss begins to rise. That pattern strongly suggests overfitting. Common signs of overfitting Training loss keeps decreasing while validation loss increases. Training accuracy becomes very high while validation accuracy stops improving. The gap between training and validation performance keeps growing. The model performs poorly on unseen test data despite excellent training results. Why does this happen? The network begins learning details that are specific to the training set, including noise and accidental patterns. These details help it perform better on training examples but do not generalize to new data. How can you confirm it? Use a separate validation set during training and a test set for final evaluation. If the model consistently performs much better on training data than on unseen data, overfitting is likely. What can you do about it? Common approaches include: Early stopping. Dropout. Weight decay or other regularization. Data augmentation. More training data. Reducing model complexity when appropriate. For the basic concept, see “What is overfitting in deep learning?” . For a common visual signal, see “What is a validation loss curve?” . In short, you can tell a deep neural network is overfitting when training performance keeps improving while validation or test performance stops improving or gets worse.

How can you tell whether a deep neural network is underfitting?

A deep neural network is likely underfitting when it performs poorly not only on validation or test data, but also on the training data itself. This means the model has not learned the underlying patterns well enough. What should you monitor? Compare training and validation performance. A typical underfitting pattern looks like: Training loss: high Validation loss: high and both may improve only slowly or stop improving too early. Common signs of underfitting Training loss remains high. Training accuracy remains low. Validation performance is also poor. The gap between training and validation performance is small because both are weak. The model cannot capture obvious patterns in the training data. Why does underfitting happen? Common causes include: The model is too simple for the problem. Training stops too early. The learning rate or optimization setup is poor. Regularization is too strong. The input features or data representation are not informative enough. How is this different from overfitting? With overfitting, training performance is strong but validation performance is much worse. With underfitting, even the training performance is poor. How can it be reduced? Possible approaches include: Train for more epochs. Increase model capacity when appropriate. Reduce excessive regularization. Improve the optimization settings. Improve the quality or representation of the input data. For the basic concept, see “What is underfitting in deep learning?” . For the opposite behavior, see “How can you tell whether a deep neural network is overfitting?” . In short, a deep neural network is underfitting when it cannot perform well even on its training data, showing that the model has not learned the problem sufficiently.