Deep Learning
Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.
Why is learning rate warm-up used in deep neural networks?
Learning rate warm up is used in deep neural networks to prevent unstable parameter updates during the early stages of training. At the beginning of training, the model's weights, gradients, and optimizer statistics may not yet be well calibrated. Starting immediately with a large learning rate can therefore make the first few updates too aggressive. What does warm up change? Instead of using the full learning rate from the first step, training starts with a much smaller value and gradually increases it. For example: 0.00001 → 0.0001 → 0.0005 → 0.001 Once the target learning rate is reached, the normal learning rate schedule continues. Why does this help? Warm up gives the model time to enter a more stable training regime before large updates are allowed. It can help reduce: Sudden spikes in loss. Unstable gradients. Overshooting early in training. Divergence caused by aggressive initial updates. Why is it useful for very deep networks? In deep networks, small changes in early layers can propagate through many later layers. Because of this, unstable updates near the start of training can have a large effect on the entire model. Warm up reduces the size of those early updates. Why is it common with large batch sizes? Large batch training often uses larger peak learning rates. Jumping directly to that high value can be unstable, so warm up gradually moves the optimizer toward the intended learning rate. Why is it common in Transformer training? Transformer models are often large and sensitive to optimization settings, especially early in training. Warm up is therefore frequently combined with schedules such as linear decay or cosine decay. For the basic mechanism, see “What is a warm up learning rate?” . In short, learning rate warm up improves early training stability by starting with small updates and gradually increasing the learning rate before normal training begins.
How does momentum accelerate neural network training?
Momentum accelerates neural network training by combining the current gradient with a running average of previous update directions. This gives the optimizer a kind of velocity, so it can move faster in directions where gradients consistently point the same way. Without momentum Standard gradient descent updates parameters using only the current gradient: w new = w old learning rate × gradient If the loss surface is narrow or curved, updates may zigzag from side to side and make slow progress toward the minimum. With momentum Momentum keeps part of the previous update: v t = β × v (t 1) + gradient t w new = w old learning rate × v t where β controls how much past direction is retained. A common value is around: β = 0.9 Why does this accelerate training? If gradients repeatedly point in a similar direction, momentum accumulates that direction over time. This can: Increase movement along useful directions. Reduce side to side oscillation. Help the optimizer move through shallow regions faster. Improve convergence on elongated or uneven loss surfaces. Simple intuition Imagine a ball rolling downhill. Instead of reacting only to the slope directly under it, the ball also carries speed from its previous motion. That accumulated velocity helps it move faster in a consistent downhill direction. Does momentum always make updates larger? No. If gradients change direction frequently, previous momentum can partially cancel the current direction. This damping effect is one reason momentum can reduce oscillation. For the basic definition, see “What is momentum in gradient descent?” . In short, momentum speeds up training by accumulating consistent gradient directions while smoothing out oscillations, allowing the optimizer to move more efficiently across the loss surface.
What is Nesterov accelerated gradient?
Nesterov accelerated gradient (NAG) is a momentum based optimization method that calculates the gradient after first looking ahead in the direction of the current momentum. It is designed to make momentum more responsive by estimating where the parameters are about to move before computing the next correction. How is it different from standard momentum? Standard momentum calculates the gradient at the current parameter position. Nesterov momentum first makes a temporary look ahead step: look ahead = w β × v and then calculates the gradient at that look ahead position. A simplified update is: v t = β × v (t 1) + gradient(look ahead) w new = w old learning rate × v t Why does the look ahead help? Momentum may build up speed in one direction and overshoot a useful region. Nesterov momentum checks the slope slightly ahead of the current position. This gives the optimizer an earlier opportunity to correct its direction. What are the benefits? Nesterov accelerated gradient can: Reduce overshooting. Improve responsiveness to changes in the loss surface. Preserve the acceleration benefits of momentum. Sometimes converge faster or more smoothly than standard momentum. Simple intuition Imagine a ball rolling downhill with momentum. Standard momentum reacts to the slope where the ball currently is. Nesterov momentum looks slightly ahead in the direction the ball is already moving and adjusts before it gets there. For the underlying idea, see “What is momentum in gradient descent?” and “How does momentum accelerate neural network training?” . In short, Nesterov accelerated gradient improves momentum by computing the gradient at a look ahead position, allowing the optimizer to correct its direction earlier and often move more efficiently.
What is the Adam optimizer?
Adam (Adaptive Moment Estimation) is an optimizer that adjusts the learning rate separately for each parameter using moving averages of the gradients and squared gradients. It combines ideas from momentum and adaptive learning rate methods. What does Adam track? For each parameter, Adam maintains two moving averages: First moment: an average of recent gradients. Second moment: an average of recent squared gradients. The first moment helps capture the direction of movement, while the second moment helps scale the update according to gradient magnitude. Simplified idea At each step, Adam updates estimates similar to: m t = β₁ × m (t 1) + (1 β₁) × gradient v t = β₂ × v (t 1) + (1 β₂) × gradient² These estimates are then bias corrected and used to update the parameter. A simplified update looks like: w new = w old learning rate × m̂ / (sqrt(v̂) + ε) where m̂ and v̂ are the bias corrected moment estimates. Why is Adam popular? Adam is widely used because it: Adapts the update size for each parameter. Uses momentum like smoothing. Often works well with little initial tuning. Handles noisy or sparse gradients effectively. Common default settings Typical defaults are: learning rate = 0.001 β₁ = 0.9 β₂ = 0.999 These are common starting points, but they can still require tuning. Is Adam always better than SGD? No. Adam often converges faster early in training, while SGD with momentum can sometimes provide better final generalization for certain tasks. For a direct comparison, see “What is the difference between SGD and Adam?” . The next article explains the update process in more detail: “How does the Adam optimizer work?” . In short, Adam combines momentum with adaptive per parameter learning rates by tracking moving averages of gradients and squared gradients.
How does the Adam optimizer work?
The Adam optimizer works by combining momentum with adaptive learning rates for each parameter. It tracks both the average gradient direction and the average squared gradient magnitude, then uses them to scale each update. Step 1: Calculate the gradient After backpropagation, Adam receives the current gradient: g t This gradient shows how the loss changes with respect to each parameter. Step 2: Update the first moment Adam keeps an exponential moving average of recent gradients: m t = β₁ × m (t 1) + (1 β₁) × g t This behaves similarly to momentum and helps smooth the update direction. Step 3: Update the second moment Adam also tracks an exponential moving average of squared gradients: v t = β₂ × v (t 1) + (1 β₂) × g t² This measures the recent scale of the gradients. Step 4: Correct the initial bias Because m t and v t start at zero, their early values are biased toward zero. Adam corrects this using: m̂ t = m t / (1 β₁^t) v̂ t = v t / (1 β₂^t) Step 5: Update the parameter The parameter is then updated using: w new = w old learning rate × m̂ t / (sqrt(v̂ t) + ε) The small value ε prevents division by zero. What does this achieve? Adam effectively gives each parameter its own adaptive step size. Parameters with consistently large gradients receive relatively smaller updates, while parameters with smaller gradients can receive relatively larger updates. At the same time, the first moment estimate provides momentum like smoothing. Typical defaults are: β₁ = 0.9 β₂ = 0.999 ε ≈ 1e 8 For the overview, see “What is the Adam optimizer?” . In short, Adam combines a moving average of gradients with a moving average of squared gradients, applies bias correction, and uses both to produce adaptive, momentum like parameter updates.
What is the RMSprop optimizer?
RMSprop (Root Mean Square Propagation) is an optimizer that adapts the learning rate for each parameter based on a moving average of recent squared gradients. It was designed to make gradient descent more stable when gradient magnitudes vary significantly across parameters. What does RMSprop track? For each parameter, RMSprop maintains an exponential moving average of squared gradients: v t = β × v (t 1) + (1 β) × g t² where: g t is the current gradient. β controls how much past squared gradients are retained. How is the parameter updated? The update is scaled using the square root of that moving average: w new = w old learning rate × g t / (sqrt(v t) + ε) The small value ε prevents division by zero. Why does this help? If a parameter repeatedly receives large gradients, RMSprop reduces its effective step size. If another parameter receives smaller gradients, its updates can remain relatively larger. This allows different parameters to adapt to different gradient scales. What problem does RMSprop address? On loss surfaces where gradients are much larger in one direction than another, ordinary gradient descent may oscillate heavily. RMSprop reduces those large direction updates and can make optimization smoother. Common settings A common starting value is: β ≈ 0.9 The learning rate still needs to be chosen, although RMSprop automatically rescales individual parameter updates. How is RMSprop related to Adam? RMSprop tracks a moving average of squared gradients. Adam builds on a similar idea but also tracks a moving average of the gradients themselves, giving it a momentum like component. For that optimizer, see “What is the Adam optimizer?” . The next article explains RMSprop step by step: “How does RMSprop work?” . In short, RMSprop adapts each parameter's update by dividing its gradient by a moving estimate of recent gradient magnitude, helping stabilize and balance training.
How does RMSprop work?
RMSprop works by scaling each parameter's gradient using a moving average of recent squared gradients. This gives each parameter an adaptive effective learning rate. Step 1: Calculate the current gradient After backpropagation, RMSprop receives the gradient: g t This tells the optimizer how the loss changes with respect to the parameter. Step 2: Track squared gradients RMSprop maintains an exponential moving average of squared gradients: v t = β × v (t 1) + (1 β) × g t² This value estimates the recent magnitude of the gradients. Step 3: Scale the update The parameter update is divided by the square root of that moving average: w new = w old learning rate × g t / (sqrt(v t) + ε) where ε is a small value added for numerical stability. Why square the gradients? Squaring makes gradient magnitude positive and gives larger gradients more influence on the moving average. If a parameter has repeatedly large gradients, v t becomes larger. That increases the denominator and reduces the effective update size. If gradients are smaller, the denominator remains smaller and the parameter can receive relatively larger updates. Why is this useful? This adaptive scaling can: Reduce oscillation in directions with large gradients. Allow faster movement in directions with smaller gradients. Handle parameters with very different gradient scales. Make optimization more stable on uneven loss surfaces. How is RMSprop different from Adam? RMSprop mainly tracks squared gradients. Adam also tracks a moving average of the gradients themselves, adding a momentum like component. For the overview, see “What is the RMSprop optimizer?” . In short, RMSprop tracks recent squared gradients and uses them to scale each parameter update, automatically reducing steps where gradients are large and allowing relatively larger steps where they are small.
What is Adagrad?
Adagrad (Adaptive Gradient Algorithm) is an optimizer that automatically gives each parameter its own learning rate based on the history of its gradients. Parameters that receive large or frequent gradients get smaller future updates, while parameters with small or infrequent gradients can keep relatively larger updates. How does Adagrad work? For each parameter, Adagrad accumulates the squared gradients: G t = G (t 1) + g t² where g t is the current gradient. The parameter update is then scaled by this accumulated value: w new = w old learning rate × g t / (sqrt(G t) + ε) The small value ε prevents division by zero. Why is this called an adaptive learning rate? The effective learning rate becomes different for each parameter. If a parameter has received many large gradients, G t becomes large and its future updates shrink. If another parameter has received only small or rare gradients, its effective learning rate stays relatively larger. Where is Adagrad useful? Adagrad can work well for problems with sparse features or sparse gradients , where some parameters are updated much less frequently than others. Examples include some natural language processing and recommendation system settings. What is the main limitation? G t keeps accumulating throughout training and never decreases. Because of this, the effective learning rate continually becomes smaller. Eventually it can become so small that useful learning nearly stops. This limitation motivated optimizers such as RMSprop, which use a moving average of squared gradients instead of accumulating them forever. For that approach, see “What is the RMSprop optimizer?” . In short, Adagrad adapts the learning rate separately for each parameter using accumulated squared gradients, making it useful for sparse problems but sometimes causing the learning rate to decay too aggressively.
What is AdamW?
AdamW is a version of the Adam optimizer that applies weight decay separately from the gradient based parameter update. The W stands for weight decay . Why was AdamW introduced? Adam already adapts the learning rate for each parameter. If L2 regularization is added directly to the gradient, Adam's adaptive scaling also changes the effect of that regularization. AdamW avoids this by decoupling weight decay from the gradient update . Simplified idea Standard Adam updates parameters using the gradient and its moment estimates. AdamW adds a separate shrinkage step: w new = Adam update(w) learning rate × weight decay × w This means the parameter is reduced directly according to the weight decay coefficient. Why is decoupling useful? Decoupling makes the strength of weight decay easier to control independently from Adam's adaptive gradient scaling. This can improve regularization behavior and often works well in modern deep learning models. Where is AdamW commonly used? AdamW is widely used for: Transformer models. Large language models. Vision Transformers. Many modern deep neural networks. Is AdamW the same as Adam with L2 regularization? Not exactly. With plain SGD, L2 regularization and weight decay can behave equivalently under common settings. With adaptive optimizers such as Adam, directly adding an L2 penalty to the gradient is not generally equivalent to decoupled weight decay. That distinction is the main reason AdamW exists. For the base optimizer, see “What is the Adam optimizer?” . The related regularization concept is covered in “What is weight decay in neural networks?” and “What is the difference between weight decay and L2 regularization?” . In short, AdamW is Adam with decoupled weight decay, allowing regularization to be applied directly to the parameters instead of being mixed into Adam's adaptive gradient update.
What is weight decay in neural networks?
Weight decay is a regularization technique that gradually shrinks neural network weights during training. It discourages the model from relying on excessively large parameter values, which can help reduce overfitting. How does weight decay work? After the optimizer computes the normal parameter update, weight decay applies an additional shrinkage to the weights. A simplified update is: w new = w old learning rate × gradient learning rate × weight decay × w old The last term pulls the weight slightly toward zero. What does the weight decay value control? The weight decay coefficient determines how strongly the weights are shrunk. If it is: Too small, the regularization effect may be negligible. Too large, the model may be overly constrained and underfit. Why can smaller weights help? Very large weights can make a model highly sensitive to small changes in the input. Encouraging smaller weights can produce smoother, simpler solutions that often generalize better. Is weight decay the same as setting weights to zero? No. Weight decay gradually reduces parameter magnitudes during training. It does not force all weights to become exactly zero. How is this related to L2 regularization? For standard gradient descent, L2 regularization and weight decay can behave equivalently under common settings. With adaptive optimizers such as Adam, they are not generally identical because an L2 penalty is mixed into the gradient and then adaptively scaled. That is why AdamW uses decoupled weight decay . For that optimizer, see “What is AdamW?” . The next article covers the distinction directly: “What is the difference between weight decay and L2 regularization?” . In short, weight decay regularizes a neural network by continuously shrinking parameter values during training, helping limit overly complex solutions and reduce overfitting.