Deep Learning

Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.

What is DropConnect and how does it differ from dropout?

DropConnect is a regularization method that randomly disables individual weights (connections) during training. Unlike ordinary dropout, which randomly sets activation values to zero, DropConnect changes which connections can contribute to each neuron's calculation on that training pass. Where is the mask applied? Dropout: Apply a random binary mask to an activation vector. In the example below, masking an input activation removes its contribution to every neuron that receives it . DropConnect: Apply a random binary mask element by element to the weight matrix . A connection from one input to one output can be removed while that same input's connection to another output remains. Training: A freshly sampled mask changes which weights participate in the forward pass; a masked weight receives no gradient from that masked connection on that pass. The learned weight is not permanently deleted . Inference: Random masking is normally disabled, but inference scaling or averaging must follow the implementation: the original DropConnect work developed a distributional approximation rather than assuming a masked nonlinear network can always be replaced exactly by an unmasked one. A compact way to write a bias free, linear DropConnect layer is: y train = ((M ⊙ W) / q) x Here x is the input vector, W the learned weight matrix, M a binary weight mask whose entries are 1 with keep probability q, and ⊙ means element wise multiplication. Dividing by q is the inverted scaling convention used in this example; the original paper used a different convention and more careful inference. Beginner friendly numerical comparison Suppose a layer has two inputs and two output neurons , with no bias or activation function for simplicity: x = [2, 3] W = [[1, 2], [3, 4]] (each row contains one output neuron's weights) With no masking, the output is: Output 1: 1×2 + 2×3 = 8 Output 2: 3×2 + 4×3 = 18 Full output: [8, 18] Use keep probability q = 0.5 in both examples. 1. DropConnect: mask individual connections Imagine the independently sampled weight mask is M = [[1, 0], [0, 1]]. Kept and scaled weight matrix: (M ⊙ W)/0.5 = [[2, 0], [0, 8]]. Training output: [[2, 0], [0, 8]] × [2, 3] = [4, 24]. Notice that input 3 does not connect to output 1 , but does connect to output 2 . DropConnect can keep or remove those two connections independently. 2. Dropout: mask input activations instead For comparison, apply ordinary dropout to this layer's input vector , using sampled activation mask m = [1, 0]: Kept and scaled input: x' = (m ⊙ x)/0.5 = [4, 0]. Training output: W x' = [[1, 2], [3, 4]] × [4, 0] = [4, 12]. Here the second input is unavailable to both output neurons . That is a shared missing input activation , unlike independently masked weights. At inference (simple inverted scaling, linear example): No mask is applied; both give the full output [8, 18] . The masks are random examples, not guaranteed to drop exactly half the entries every pass. Scaling and inference: an important detail Inverted scaling: Scaling retained weights by 1/q during training gives E[((M ⊙ W)/q)x] = Wx for the linear pre activation . Ordinary inverted dropout similarly preserves its linear expectation. Nonlinear layers: In general, E[activation(masked input)] is not equal to activation(E[masked input]). Therefore, simply using full weights at inference does not reproduce the exact average prediction over every DropConnect mask. Original method: Wan and colleagues modeled the distribution of pre activations and approximated the average nonlinear output using a Gaussian based approach. Actual implementations may instead use simpler deterministic scaling or Monte Carlo averaging; state which convention is used. When can DropConnect help, and what are its limitations? Benefit: It perturbs individual connections rather than removing whole activation values, encouraging a network to avoid depending too heavily on particular weights; early experiments evaluated it on fully connected layers in image recognition networks. Use case: Consider it when regularizing dense / fully connected layers where per weight connectivity noise is useful; benchmark against standard dropout on held out data. Limitations: Sampling a mask for every weight (sometimes per sample) can increase memory or implementation cost. Heavy masking may cause underfitting, and nonlinear inference can require an approximation. It is not guaranteed to outperform dropout. Key takeaway: Dropout masks activations; DropConnect masks individual weights. Both randomly regularize training, but a single input can remain connected to one output and disconnected from another under DropConnect. For dropout fundamentals, see “What is dropout in deep learning?” . For a different comparison involving entire residual branches, see “How does stochastic depth differ from dropout?” (Question 250). The next curriculum article covers mixup augmentation . Research: Wan et al. (2013), Regularization of Neural Networks using DropConnect . Reference: PyTorch Dropout documentation.

What is mixup augmentation for neural network training?

Mixup is a data augmentation technique that creates a new training example by taking a weighted average of two existing examples—and the same weighted average of their labels. It encourages a neural network to make smoother, more consistent predictions between training samples instead of memorizing each example in isolation. How does mixup work? Choose two training examples (xₐ, yₐ) and (xᵦ, yᵦ), and a mixing coefficient λ between 0 and 1 : Mixed input: x mix = λxₐ + (1 − λ)xᵦ Mixed target: y mix = λyₐ + (1 − λ)yᵦ Here x can be an image or another numeric input, while y is typically a one hot class label vector . The same λ must be used for both the input and its target. This produces a soft target instead of a single hard class label. Beginner friendly numerical example Suppose we are training a two class cat vs dog classifier. To make the arithmetic simple, represent each tiny input by two numeric features: Sample A (cat): xₐ = [2, 4], label yₐ = [1, 0] Sample B (dog): xᵦ = [8, 10], label yᵦ = [0, 1] Chosen mixing coefficient: λ = 0.25 1. Mix the inputs: x mix = 0.25[2, 4] + 0.75[8, 10] = [0.5, 1] + [6, 7.5] = [6.5, 8.5] 2. Mix the labels using the same coefficient: y mix = 0.25[1, 0] + 0.75[0, 1] = [0.25, 0.75] The new example has a 25% cat / 75% dog training target . This is a supervision weight , not a claim that the input is literally 25% of a cat. 3. Calculate training loss with the soft target: Suppose the model predicts class probabilities p = [0.40, 0.60]. Soft label cross entropy gives: L = −(0.25 × ln(0.40) + 0.75 × ln(0.60)) ≈ 0.6122 Equivalently, for the same mixed input , the loss can be written as 0.25 × CE(p, cat) + 0.75 × CE(p, dog). We train against both labels with different weights , not just the dog class. For real images, mixup blends corresponding pixel values from two images; the toy vectors simply make the calculation easier to follow. Where does λ come from? The Beta distribution In standard mixup, λ is sampled randomly for training pairs: λ ~ Beta(α, α), where α 0 controls the strength of mixing. α < 1 (for example, 0.2): Samples concentrate near 0 or 1 , often favoring one original example. α = 1: All mixing coefficients from 0 to 1 are equally likely (uniform distribution). α 1 (for example, 2): Samples concentrate more strongly around 0.5 , producing more balanced mixtures. The λ = 0.25 above is an illustrative sampled value , not a fixed hyperparameter. Larger α does not automatically mean higher accuracy; tune it using validation performance. Benefits, limitations, and practical use Benefits: Can reduce overfitting, discourage overly confident predictions, and improve generalization or robustness on suitable tasks. The original paper demonstrated gains on several image and other classification benchmarks. Where useful: Image and audio classification and some other supervised learning problems with meaningful input interpolation. Mix training samples after arranging them into compatible numeric representations. Limitations: Blended samples can be unrealistic; mixing unrelated data or targets may hurt learning. Very strong mixing can cause underfitting, and mixup is not guaranteed to outperform ordinary training. Training versus inference: Apply mixup to training data . Evaluate and predict on normal, unmixed examples so validation measures real world performance. Mixup is different from basic label smoothing: both the input and its target are blended according to the chosen pair , rather than only softening a label. See “What is data augmentation in deep learning?” and “What is label smoothing in neural network training?” for those prerequisites. The next question explains the separate CutMix augmentation approach. Key takeaway: Mixup blends two inputs and their labels with the same random coefficient, then trains the model using the resulting soft target. Research: Zhang et al. (2018), mixup: Beyond Empirical Risk Minimization . Implementation: Torchvision MixUp.

What is CutMix augmentation in image classification?

CutMix is an image classification augmentation technique that replaces a rectangular region of one training image with pixels from another image, then weights the class labels according to how much image area each source contributes. It helps a classifier learn from partially replaced images without throwing away the replaced region's pixel budget. How does CutMix work? 1. Select two examples: A base image A and donor image B, with their class labels yₐ and yᵦ. 2. Choose a rectangular patch: A mixing coefficient λ₀, often drawn from Beta(α, α), controls the intended area; the patch location is randomized. 3. Paste donor pixels: Replace the selected rectangle in image A with the corresponding rectangle from image B. All other pixels stay from A. 4. Correct the label weights: Measure the actual pasted rectangle after rounding and boundary clipping , not just the initially sampled coefficient. 5. Train with a mixed target: Use the two original class labels, weighted by the respective image area fractions. If the image has width W, height H, and the clipped patch spans corners (x₁, y₁) and (x₂, y₂), then: A patch = (x₂ − x₁) × (y₂ − y₁) λ eff = 1 − A patch / (W × H) y mix = λ eff × yₐ + (1 − λ eff) × yᵦ Important: λ eff measures the retained base image fraction . It may differ from the sampled λ₀ because the final patch can be rounded or clipped. Beginner friendly numerical example Suppose we have two 8 × 8 pixel images , each containing 64 pixels : Base image A: Cat, label yₐ = [1, 0]. Donor image B: Dog, label yᵦ = [0, 1]. Initially sampled coefficient: λ₀ = 0.70. Actual selected patch: After the rectangle is converted to pixel coordinates, it covers 4 rows × 4 columns , for example rows 3–6 and columns 2–5. Those 16 pixels of image A are replaced by the corresponding pixels of image B. Step 1 — Correct the mixing coefficient using the actual area. A patch = 4 × 4 = 16 λ eff = 1 − 16/64 = 0.75 Thus 75% of the pixels are from the cat image and 25% from the dog image. We use 0.75 , not the initial 0.70 , to construct the training target. Step 2 — Derive the mixed class label. y mix = 0.75[1, 0] + 0.25[0, 1] = [0.75, 0.25] This label means 75% cat supervision and 25% dog supervision . It does not necessarily mean 25% of the visible animal is a dog: CutMix weights labels by pixel area , not by object area. Step 3 — Calculate training loss. Suppose the classifier predicts probabilities p = [0.80, 0.20]. With soft target cross entropy: L = −(0.75 × ln(0.80) + 0.25 × ln(0.20)) ≈ 0.5697 Equivalently, L = 0.75 × CE(p, cat) + 0.25 × CE(p, dog). The network learns from the single patched image , supervised by a weighted combination of both labels . Why is CutMix useful? Keeps information: The patch is filled with another image's pixels instead of blanking out part of the training image. Promotes varied visual evidence: Training with partially replaced objects can discourage reliance on a single distinctive region and can improve robustness to occlusion. Potential generalization gains: The original CutMix research reported improvements in image classification and localization benchmarks; success still depends on the task and model. Limitations and practical considerations Area is only a label proxy: A patch might contain background, obscure a small critical object, or introduce unnatural boundaries. The label weights may therefore imperfectly reflect the visible objects. Choose mixing strength carefully: Overly large or frequent pasted regions can hurt fine grained or small object recognition. Tune the augmentation probability and the Beta distribution parameter α using validation results. Training versus inference: Apply CutMix when constructing training examples , usually by pairing images in a shuffled batch. Validate and infer on normal images unless a separate test time augmentation method is explicitly used. Keep images and targets synchronized: Always derive the mixed label from the actual pasted area , including any clipping. For prerequisites, see “What is data augmentation in deep learning?” and “What is mixup augmentation for neural network training?” . Mixup interpolates whole inputs; CutMix retains real pixels and swaps a spatial rectangle . The full mixup explanation remains in Question 252. Key takeaway: CutMix pastes a rectangle from another image and trains on class labels weighted by the actual retained and pasted pixel areas. Research: Yun et al. (2019), CutMix: Regularization Strategy to Train Strong Classifiers With Localizable Features . Implementation: Torchvision CutMix.

What is an exponential moving average (EMA) of neural network weights?

An exponential moving average (EMA) of neural network weights is a smoothed, separate copy of the model's parameters. After an optimizer update, each EMA weight moves partway toward the corresponding new model weight. It remembers earlier weight values while giving more influence to recent ones. How are EMA weights calculated? For each parameter: θ EMA,t = β × θ EMA,t−1 + (1 − β) × θ model,t θ model,t: Current training model weight after optimizer update t. θ EMA,t−1: Previous weight in the EMA copy. β: Decay coefficient, usually between 0 and 1. Larger values make the EMA respond more slowly. 1 − β: Weight placed on the most recent training model value. A common initialization is θ EMA,0 = θ model,0. The optimizer updates the training model ; EMA weights are subsequently updated by the averaging equation, not directly by gradient descent. Beginner friendly numerical example Follow a single model weight over three optimizer updates. Set the initial model and EMA weights to 10 , and choose β = 0.8. Thus each EMA update retains 80% of the previous EMA and takes 20% of the latest optimized weight . 1. Initialize: θ model,0 = 10 and θ EMA,0 = 10. 2. Update 1: Optimizer changes the model weight to 14 . New EMA: 0.8 × 10 + 0.2 × 14 = 8 + 2.8 = 10.8. 3. Update 2: Model weight becomes 12 . New EMA: 0.8 × 10.8 + 0.2 × 12 = 8.64 + 2.4 = 11.04. 4. Update 3: Model weight becomes 16 . New EMA: 0.8 × 11.04 + 0.2 × 16 = 8.832 + 3.2 = 12.032. The latest training model weight is 16 , but its EMA weight is 12.032 . EMA follows the overall movement while reducing the effect of individual jumps. Real networks apply the same equation element wise across all tracked parameter tensors . What does the decay factor control? Lower decay (e.g., 0.8): The EMA catches up to recent weights more quickly but smooths less. Higher decay (e.g., 0.99 or 0.999): Changes more slowly, incorporating a longer history; often used when there are many training updates. Tradeoff: Very large decay early in training can leave EMA weights lagging behind rapid model improvements. Some implementations delay EMA or gradually increase the decay. The value 0.8 in the example is chosen for transparent arithmetic; it is not a recommended default for every training job . How is EMA used during training and evaluation? Train as usual: Compute gradients and update the normal model with optimizer.step(). Then update EMA: Track a second set of weights from the latest normal model parameters. The EMA copy itself is not gradient trained. Evaluate both models: Run validation using the latest model weights and the EMA copy; use whichever performs better for the target task. You can checkpoint and deploy EMA weights if appropriate. Consider normalization buffers: BatchNorm running mean and variance are non parameter buffers. If the EMA model uses such layers, their statistics must be copied, updated or recomputed appropriately to ensure meaningful evaluation. PyTorch example: torch.optim.swa utils.AveragedModel(model, multi avg fn=get ema multi avg fn(decay)) maintains averaged weights. After optimizer.step(), call ema model.update parameters(model). The framework handles initial weight copying; check its documentation for normalization buffers and saved checkpoints. Benefits, limitations, and applications Benefits: Smoothing can reduce sensitivity to individual optimizer updates and sometimes improve validation accuracy, stability or generalization. Applications: Image classification and other deep learning training; Mean Teacher semi supervised learning maintains EMA teacher weights from the student model. Limitations: Maintaining another copy consumes memory and adds updates. High decay can make EMA slow to adapt; the best EMA configuration is task dependent, and EMA is not guaranteed to outperform current weights. Important distinction: EMA tracks model parameters , not the optimizer's moving average of gradients. The next question covers stochastic weight averaging (SWA) separately. Key takeaway: EMA blends each previous averaged weight with the newest optimized weight to maintain a smoother alternative set of model parameters. Research and documentation: PyTorch weight averaging (EMA and SWA), PyTorch AveragedModel, and Tarvainen & Valpola (2017), Mean Teacher .

What is stochastic weight averaging (SWA)?

Stochastic Weight Averaging (SWA) creates a new set of neural network weights by taking the simple arithmetic mean of several weight snapshots from the same training run . Instead of deploying only the latest optimized weights, SWA combines selected points along the optimizer's later trajectory into one model . It is not an ensemble that averages multiple models' predictions. How does SWA work? 1. Train the main model normally: An optimizer continues updating the network using training batches. 2. Start averaging later in training: Once the model has learned useful features, take weight snapshots at selected updates or epochs—not necessarily at every update. 3. Average each parameter separately: The snapshots must belong to the same architecture and compatible parameter positions within the same training trajectory. 4. Maintain an SWA copy: The optimizer still trains the current model; the SWA copy is updated by averaging, without its own gradient steps. 5. Evaluate the averaged model: After handling normalization statistics, compare validation performance against the latest training model checkpoint. For n selected snapshots θ₁, θ₂, …, θₙ: θ SWA = (θ₁ + θ₂ + … + θₙ) / n This calculation applies element wise to every trainable weight tensor . You can compute it without storing every checkpoint using the running average: θ SWA,new = (n × θ SWA,old + θ next) / (n + 1) Here n counts snapshots already averaged , not all optimizer steps since training began. Beginner friendly numerical example Consider a toy model with two weights. Suppose we save three checkpoints late in the same training run : Checkpoint 1: θ₁ = [2, 4] Checkpoint 2: θ₂ = [4, 8] Checkpoint 3: θ₃ = [6, 6] Step 1 — Average the first two snapshots: θ SWA,2 = ([2, 4] + [4, 8]) / 2 = [3, 6] Step 2 — Include the third snapshot: θ SWA,3 = (2 × [3, 6] + [6, 6]) / 3 = ([6, 12] + [6, 6]) / 3 = [4, 6] Check using the direct arithmetic mean: θ SWA = ([2, 4] + [4, 8] + [6, 6]) / 3 = [4, 6] The latest model weights are [6, 6], while the SWA weights are [4, 6]. Every snapshot contributes one third to the final average. These numbers illustrate the arithmetic, not a guarantee that [4, 6] gives better predictions. When are weights collected, and what about learning rate? Typically start late: Begin collecting snapshots after ordinary training has made substantial progress. Use a suitable learning rate schedule: The original SWA work used a constant or cyclical learning rate during the averaging phase so the optimizer explores multiple nearby solutions instead of settling at one point. Collect repeatedly: For example, sample the model after each epoch during an SWA phase. A different sampling cadence is possible; choose it deliberately. Keep training the original model: Averaging is an additional process alongside optimizer updates, not a replacement for the optimizer. In PyTorch, torch.optim.swa utils.AveragedModel(model) maintains an equal weight running average by default. After a selected optimizer update, call swa model.update parameters(model). torch.optim.swa utils.SWALR can move the learning rate toward a chosen fixed SWA rate. Why does BatchNorm need special attention? BatchNorm stores running means and variances of activations in buffers. These statistics depend on the model's weights and data distribution; simply averaging parameters does not automatically make the existing BatchNorm statistics correct for the new SWA weights. Common approach: After the final SWA update, run torch.optim.swa utils.update bn(train loader, swa model) to recompute BatchNorm statistics by passing training inputs through the SWA model . Alternative: Some implementations average buffers using AveragedModel(..., use buffers=True); check whether that approach gives appropriate validation results. No BatchNorm: Skip this step when the model has no BatchNorm style running statistics. Benefits, limitations, and applications Benefits: The original work found that averaging points visited by SGD can land in a broader, well performing region of weight space and sometimes improve generalization, robustness of the selected checkpoint, or model calibration. Applications: Image classifiers and other deep networks trained along a compatible optimization path; it is easy to test as a late training extension. Limitations: Requires additional training time for collecting snapshots and a separate parameter copy. Averages of very different or incompatible solutions may perform poorly; SWA does not guarantee better accuracy. Recomputing BatchNorm statistics adds a data pass. Training versus inference: Average parameters during training ; evaluate and deploy the single averaged model , normally with no further snapshot averaging at inference. Brief comparison with EMA: SWA normally gives equal weight to chosen snapshots; an exponential moving average gives more weight to recent parameters. The detailed EMA update and decay factor explanation is covered in “What is an exponential moving average (EMA) of neural network weights?” (Question 254). Key takeaway: SWA averages several compatible neural network weight snapshots gathered during training to obtain one alternative model, which must be validated before use. Research: Izmailov et al. (2018), Averaging Weights Leads to Wider Optima and Better Generalization . Implementation: PyTorch — Weight Averaging (SWA and EMA) and PyTorch AveragedModel.

What is sharpness-aware minimization (SAM)?

Sharpness Aware Minimization (SAM) is a training method that tries to keep neural network loss low not only at the current weights, but also at nearby, slightly perturbed weights. It uses an extra gradient calculation to guide the optimizer toward weights that are less sensitive to small changes. What does SAM optimize? For model parameters w, training loss L(w), and perturbation radius ρ 0, the idealized min–max objective is: min w max {||ε||₂ ≤ ρ} L(w + ε) Inner maximum: Within the radius ρ, find nearby weights where the loss becomes as large as possible. Outer minimum: Update the model to reduce that nearby high loss. ε: A temporary change to the weights. ||ε||₂ is its Euclidean length, and ρ limits that length. Solving the exact inner maximization is costly for a neural network. Standard SAM approximates its direction using the first gradient , then calculates the gradient again at the perturbed weights: g₁ = ∇L(w) ε̂ = ρ × g₁ / ||g₁||₂ (for a nonzero gradient norm) g₂ = ∇L(w + ε̂) For a simple SGD base optimizer with learning rate η, restore the original weights w , then update: w new = w − η × g₂ The first gradient decides where to probe; the second gradient guides the actual update. Real implementations usually compute both gradients with the same mini batch . SAM can also be combined with other base optimizers, but the final update then follows that optimizer's rule. Beginner friendly numerical example Imagine a toy model with two weights : w = [3, 4] For easy calculation, choose the artificial loss L(w) = ½(w₁² + w₂²), whose gradient is ∇L(w) = [w₁, w₂]. Use radius ρ = 0.5 and learning rate η = 0.1 . 1. First gradient: At w = [3, 4], g₁ = [3, 4]. 2. Normalize its length: ||g₁||₂ = √(3² + 4²) = 5. 3. Find the perturbation: ε̂ = (0.5 / 5) × [3, 4] = [0.3, 0.4]. Its length is 0.5, so it stays within the allowed radius. 4. Take the second gradient: Temporarily evaluate at w + ε̂ = [3.3, 4.4]. Since this toy gradient equals the weight vector, g₂ = [3.3, 4.4]. 5. Restore, then update: Return to the original w = [3, 4] and apply w new = [3, 4] − 0.1 × [3.3, 4.4] = [2.67, 3.56]. Result: The updated model weights are [2.67, 3.56] , not the temporary probe [3.3, 4.4]. The toy quadratic makes the mathematics simple. In a large nonlinear network, the direction ε̂ is usually only an approximation to the worst case perturbation. What does radius ρ control? Small ρ: SAM tests a narrow neighborhood and behaves more similarly to the underlying optimizer. Larger ρ: SAM probes farther away, placing more emphasis on reducing losses around the current parameters. Too large: The probe can become unhelpful or destabilize learning. The best radius depends on parameter scale, optimizer configuration, dataset and model; tune it on validation data. When is SAM useful, and what are its limitations? Potential benefit: The original research found improved generalization on image classification and fine tuning benchmarks, including some noisy label experiments. The aim is to make optimization less sensitive to small parameter changes. Extra training cost: A typical update uses two forward and backward evaluations rather than one, often adding substantial computation and training time. Implementation detail: Restore the original weights before the base optimizer step. With BatchNorm, avoid accidentally updating running statistics twice because of the extra forward pass. Limitations: SAM is not guaranteed to improve accuracy. A poor ρ, limited compute, or a mismatch between algorithm and task can outweigh its benefits. Inference: SAM affects training updates . At inference, use the model's ordinary trained weights; there is no perturbation search or second gradient. The next question, “What are sharp and flat minima in neural network optimization?” (Question 257), explains the loss landscape geometry that motivates SAM. This article focuses on the algorithm and its numerical update , rather than repeating that separate topic. Key takeaway: SAM probes the loss at nearby higher loss weights, computes a second gradient, then uses it to update the original weights. Research: Foret et al. (ICLR 2021), Sharpness Aware Minimization for Efficiently Improving Generalization ; Google Research overview.

What are sharp and flat minima in neural network optimization?

Sharp and flat minima describe how quickly a neural network's training loss increases when its learned weights move slightly away from a low loss point. A sharp minimum resembles a narrow valley with steep sides; a flat minimum resembles a wider valley where nearby weights have similar loss. What does sharp versus flat mean? Minimum: A set of model weights where the loss is locally low. Sharp minimum: Small changes to the weights can increase the loss substantially in one or more directions. Flat minimum: The loss changes more gradually across a specified neighborhood of the weights. Important: Sharpness describes the surroundings of a minimum, not necessarily its loss value. Two minima can have the same training loss but different local shapes. Neural networks have millions of parameters, so the real loss landscape has many directions. A minimum can be steep in one direction and gentle in another; “sharp” and “flat” are useful descriptions, not always simple yes/no labels. Beginner friendly numerical example Imagine two different low loss regions of a model. Near each region's own minimum, approximate its loss along one comparable weight direction . Let δ be a small change to that weight coordinate. Flat region: L flat(δ) = 0.10 + 0.5δ² Sharp region: L sharp(δ) = 0.10 + 5δ² Step 1 — At the minimum (δ = 0): Flat: 0.10 + 0.5 × 0² = 0.10 Sharp: 0.10 + 5 × 0² = 0.10 Both minima start with exactly the same loss, 0.10 . Step 2 — Apply the same small perturbation (δ = +0.2): Flat: 0.10 + 0.5 × (0.2)² = 0.12 Sharp: 0.10 + 5 × (0.2)² = 0.30 Step 3 — Compare the loss increase: Flat: 0.12 − 0.10 = 0.02 Sharp: 0.30 − 0.10 = 0.20 Under the same 0.2 parameter perturbation , the sharp region's loss rises 10 times as much in this toy example. The curves are symmetric here, so a change of −0.2 gives the same increases. Actual neural network loss surfaces need not be symmetric or quadratic. How does curvature relate to sharpness? Higher local curvature: The loss bends upward quickly as you leave a minimum; the location is sharper along that direction. Lower local curvature: The loss rises more slowly; the location is flatter along that direction. In the example: The second derivatives are 1 for the flat region and 10 for the sharp region. In models with many weights, different directions can have different curvatures. The Hessian summarizes these second order effects; its detailed computation is a separate topic. Why might flatter minima generalize better? Intuition: If small changes to the parameters cause only small loss increases, the model may be less sensitive to some training noise or parameter perturbations. Research observation: Some studies, including work on large batch training, found relationships between sharper solutions and worse test performance. But not a guarantee: Training loss flatness does not automatically predict test accuracy. The relationship depends on what “flat” means, how it is measured, and the model and data. Important limitations of the flat minima hypothesis Parameterization dependence: The same prediction function can have different apparent sharpness when its weights are rescaled. For a simple bias free two layer ReLU network, replacing u with c·u and v with v/c for c 0 leaves v·ReLU(u·x) unchanged, yet changes the geometry measured in raw weight coordinates. Choice of scale and direction: Perturbations must be compared with a consistent metric and neighborhood size. A measure that looks flat along one direction can miss steep directions elsewhere. Correlation is not causation: A sharp looking solution can still generalize well, as shown by theoretical counterexamples. Compare actual held out validation or test performance , not just a flatness score. For the training algorithm motivated by nearby loss increases, see “What is sharpness aware minimization (SAM)?” (Question 256); this article focuses on the loss landscape concept , not SAM's gradient steps. Key takeaway: Sharp minima have rapidly rising loss nearby; flat minima have a gentler surrounding loss landscape. Flatness can be informative, but by itself is not a reliable guarantee of generalization. Research: Keskar et al. (ICLR 2017), On Large Batch Training for Deep Learning: Generalization Gap and Sharp Minima ; Dinh et al. (ICML 2017), Sharp Minima Can Generalize For Deep Nets .

What is gradient noise scale in deep learning?

Gradient noise scale (GNS) measures how much individual training example gradients vary relative to the strength of their average direction. It provides a useful rough guide to the batch size beyond which using more examples in parallel may yield diminishing optimization speed benefits. What does gradient noise scale measure? Suppose each training example produces a gradient vector gᵢ at the same fixed model weights . Gradient signal: G = E[gᵢ], the average (full data or population) gradient. Gradient noise: S = E[||gᵢ − G||²], the average squared distance of individual gradients from that mean. For a vector gradient, this equals the trace of its covariance matrix . Simple gradient noise scale: B noise = S / ||G||² when ||G|| 0. In plain terms, large noise compared with the mean gradient implies a larger noise scale . This is the commonly used simple gradient noise definition; other task dependent and curvature weighted variants also exist. Beginner friendly numerical example Imagine the gradient of one model weight for four equally likely training example types: g = [−4, 0, 4, 8] Treat these four values as a tiny illustrative population , so the variance uses division by 4 , not the sample variance estimator's division by 3. 1. Find the mean gradient: G = (−4 + 0 + 4 + 8) / 4 = 2. 2. Find differences from the mean: [−4 − 2, 0 − 2, 4 − 2, 8 − 2] = [−6, −2, 2, 6]. 3. Square the differences: [36, 4, 4, 36]. 4. Compute gradient variance: S = (36 + 4 + 4 + 36) / 4 = 20. 5. Compute squared mean (signal strength): G² = 2² = 4. 6. Calculate the noise scale: B noise = S / G² = 20 / 4 = 5. Interpretation: Under this simplified model, the gradient noise scale is about 5 examples . It does not mean every training run should use batch size 5. What happens when the batch size changes? If each mini batch contains B independent draws with replacement from this illustrative distribution, and the mini batch gradient is their average: Variance(mini batch gradient) = S / B = 20 / B Batch size 1: Gradient estimate variance is 20 / 1 = 20. Batch size 2: Variance is 20 / 2 = 10. Batch size 5: Variance is 20 / 5 = 4, matching the squared mean gradient signal G² = 4. Batch size 10: Variance is 20 / 10 = 2 — smaller still, but using twice as many examples per update. This is why B noise ≈ 5 identifies a signal to noise crossover . The actual most efficient training batch size is not automatically 5: it also depends on GPU utilization, learning rate, model architecture, optimizer, and the performance target. If examples are sampled without replacement from a finite dataset, the variance law gains a finite population correction. Why does this matter in practice? Choose useful batch sizes: When B is far below B noise, gradients are relatively noisy, so larger batches can substantially improve gradient estimates and permit efficient data parallelism. Recognize diminishing returns: When B is far above B noise, more parallel examples reduce gradient noise but may not proportionally reduce the optimizer steps needed to reach a target loss. Adjust during training: The noise scale can change as training progresses, sometimes increasing as the model improves. Measuring it periodically can inform experiments with batch size schedules. Estimate it in larger models: Researchers can compare gradient statistics at different batch sizes rather than computing a separate full per example gradient for every item, although estimators introduce their own uncertainty. Limitations and practical cautions Not an exact critical batch size: It is a heuristic informed by assumptions about optimization dynamics. Recent work notes that noise scale estimates need not perfectly predict the true batch size efficiency crossover. Small average gradients: When ||G||² is close to zero, the ratio can become very large or unstable. Sampling assumptions: The 1/B variance law assumes independent, similarly distributed examples; correlations, data sampling, or distributed training details can change it. Compute and convergence: Batch size affects memory, hardware throughput, learning rate tuning and the total number of updates. GNS alone does not determine best wall clock time or generalization. For batch size basics, see “What is batch size in deep learning?” and “How does batch size affect neural network training?” ; this article focuses only on measuring gradient signal relative to noise . The next question introduces the Hessian as a separate concept. Key takeaway: Gradient noise scale compares gradient variance with squared mean gradient strength. It helps estimate where larger batches may start offering diminishing optimization returns, but is not a universal optimal batch size. Research: McCandlish et al. (2018), An Empirical Model of Large Batch Training ; OpenAI, How AI Training Scales ; Merrill et al. (NeurIPS 2025), Critical Batch Size Revisited .

What is the Hessian of a neural network loss function?

The Hessian is a matrix of second derivatives of a neural network's loss with respect to its trainable weights. A gradient tells us how the loss changes; the Hessian tells us how those gradient values change as the weights move. It is a local description of curvature in the loss landscape. What is inside the Hessian? For two weights w₁ and w₂, the Hessian is: H = [[∂²L/∂w₁², ∂²L/(∂w₁∂w₂)], [∂²L/(∂w₂∂w₁), ∂²L/∂w₂²]] Diagonal entries: ∂²L/∂w₁² and ∂²L/∂w₂² describe how the loss gradient changes when each weight changes by itself. Off diagonal entries: ∂²L/(∂w₁∂w₂) and ∂²L/(∂w₂∂w₁) describe how one weight affects the gradient with respect to the other weight. They capture interactions between parameters. Symmetry: If the loss has continuous second derivatives, the mixed derivatives are equal, so H is symmetric. Size: A model with P scalar trainable parameters has a P × P Hessian. Beginner friendly numerical example Consider a toy loss for a model with two weights: L(w₁, w₂) = 1.5w₁² + w₁w₂ + 1.5w₂² Step 1 — Compute the first derivatives (the gradient). With respect to w₁: ∂L/∂w₁ = 3w₁ + w₂. With respect to w₂: ∂L/∂w₂ = w₁ + 3w₂. Step 2 — Differentiate each gradient component again. First diagonal: ∂²L/∂w₁² = 3. First off diagonal: ∂²L/(∂w₁∂w₂) = 1. Second off diagonal: ∂²L/(∂w₂∂w₁) = 1. Second diagonal: ∂²L/∂w₂² = 3. Step 3 — Arrange the four values in the Hessian matrix. H = [[3, 1], [1, 3]] Notice that the off diagonal 1s are not zero : the cross term w₁w₂ couples the two weights. For this quadratic example, the Hessian has the same values at every point; in real networks it generally changes as weights change. Step 4 — Interpret the eigenvalues. The characteristic equation is: det(H − λI) = (3 − λ)² − 1 = 0 Therefore, the Hessian has eigenvalues λ₁ = 4 and λ₂ = 2 . Eigenvalue 4: Stronger upward curvature in the direction [1, 1]. Eigenvalue 2: Gentler upward curvature in the direction [1, −1]. Both positive: This quadratic loss curves upward in every direction. Its stationary point [0, 0] is a strict local (indeed global) minimum. The eigenvectors above give directions ; their normalized versions are divided by √2. An eigenvalue measures curvature per unit distance along its corresponding normalized direction. What do Hessian eigenvalues mean in general? Positive eigenvalue: The loss bends upward along that eigenvector direction. Negative eigenvalue: The loss bends downward along that direction. At a stationary point, a Hessian with both positive and negative eigenvalues indicates a saddle point , not a local minimum. Zero or near zero eigenvalue: Indicates a locally weak or inconclusive second order direction; higher order behavior may matter. Eigenvalue signs are especially informative at stationary points where the gradient is zero. A large positive eigenvalue reflects a strongly curved direction, not by itself a guarantee of poor test performance. See “What are sharp and flat minima in neural network optimization?” (Question 257) for the separate generalization discussion. Why don't we calculate the full Hessian of every network? A network with P = 1,000,000 parameters has a Hessian with P² = 10¹² entries . Storing that dense matrix in 32 bit floating point would require about 4 TB before accounting for other training memory. Computing and storing it explicitly is usually impractical. Instead, algorithms can compute a Hessian vector product (HVP) : Hv = H × v Here v is a chosen vector with the same number of entries as the model has parameters. We can obtain the product using automatic differentiation without constructing the full Hessian matrix . Small HVP example: With H = [[3, 1], [1, 3]] and v = [1, −1]: Hv = [3×1 + 1×(−1), 1×1 + 3×(−1)] = [2, −2] This also confirms that v is an eigenvector with eigenvalue 2 , because Hv = 2v. In PyTorch, torch.func.hessian can form a Hessian for small problems; composing torch.func.jvp with torch.func.grad can compute HVPs more economically in memory. Where is the Hessian useful, and what are its limitations? Optimization: Local second order information helps explain how aggressively weights can move in different directions and underpins curvature aware optimization methods. Diagnostics: Hessian eigenvalues and HVP based estimates can reveal locally unstable directions or saddle point structure. Limitations: Deep network losses are often nonconvex and nondifferentiable at some points; full Hessians are expensive, estimates may be noisy, and curvature describes a local region, not necessarily generalization. Practical choice: Many training pipelines use first order optimizers for speed and reserve Hessian or HVP calculations for analysis and specialized methods. The next question explains second order optimizers in detail. For gradient fundamentals, refer to the existing MachinoAI gradient articles rather than treating the Hessian as a new gradient algorithm. Key takeaway: The Hessian collects all second derivatives of loss into a matrix, exposing how parameter directions curve and interact; Hessian vector products make some of that information usable without storing the entire matrix. Research: Pearlmutter (1994), Fast Exact Multiplication by the Hessian . Documentation: PyTorch — Jacobians, Hessians and Hessian vector products and torch.func.hessian.

What are second-order optimizers in deep learning?

Second order optimizers are training methods that use information about how the loss curves —not just the current gradient—to choose an update direction and scale. This can make their steps more effective when some parameter directions are much more sensitive than others, but it also makes each step more expensive. How are they different from first order methods? First order update: Uses the gradient g, which tells the optimizer the local direction of increasing loss. A simple gradient descent step is w new = w − ηg. Second order update: Also uses a Hessian H, or an approximation to its action or inverse , to adjust the gradient for local curvature. Newton's method: A basic full step rule is w new = w − H⁻¹g. In implementations, solve Hd = g for d instead of explicitly calculating an inverse, then take a step w new = w − ηd, possibly with damping or a line search. Core intuition: An ordinary gradient step applies the same learning rate scale to every coordinate; Newton's step can reduce movement in highly curved directions and increase movement in gently curved directions . The Hessian's individual entries are covered in “What is the Hessian of a neural network loss function?” (Question 259); here we focus on how optimizers use that information. Beginner friendly numerical example Suppose a tiny neural network has only two weights, w = [w₁, w₂], and a simple toy loss: L(w₁, w₂) = w₁² + 4w₂² Start at w = [2, 1]. The gradient is g = [2w₁, 8w₂] = [4, 8] , and the curvature matrix is H = [[2, 0], [0, 8]] . The second coordinate has four times the curvature of the first. 1. First order gradient descent, learning rate η = 0.1: w GD = [2, 1] − 0.1[4, 8] = [1.6, 0.2] Initial loss: L([2, 1]) = 2² + 4×1² = 8. Loss after the step: L([1.6, 0.2]) = 1.6² + 4×0.2² = 2.56 + 0.16 = 2.72. 2. Newton style second order update, using a full η = 1 step: For this diagonal Hessian, divide each gradient component by its corresponding curvature: H⁻¹g = [4/2, 8/8] = [2, 1] w Newton = [2, 1] − [2, 1] = [0, 0] Loss after the Newton step: L([0, 0]) = 0. Why the difference? The Newton style method adjusts movement to the different curvatures, while the first order example uses one shared learning rate. A single exact Newton step reaches the optimum here only because this toy loss is a positive definite quadratic. In real deep networks, second order steps generally need repeated iterations, careful step sizes, and reliable curvature estimates. This comparison also uses different step size choices , so it is an illustration—not evidence that second order methods are universally faster. Which second order methods are used? Full Newton: Uses an exact Hessian to form a curvature adjusted step. Useful as a reference method and in small, smooth problems, but often impractical for large neural networks. Quasi Newton (BFGS, L BFGS): Approximates curvature from changes in weights and gradients across previous steps, without explicitly calculating the full Hessian. L BFGS stores a limited history, reducing memory compared with full BFGS. Hessian free / truncated Newton: Uses iterative solvers and Hessian vector products to approximately solve for a step without materializing a dense Hessian. Other approximations: Methods may use a diagonal or block approximation or a positive semidefinite surrogate such as a Gauss–Newton matrix to make curvature information more practical. Important distinction: Adam uses squared gradient statistics to adapt learning rates; it is not a classical Hessian based second order optimizer . Benefits, limitations, and practical applications Potential benefit: Curvature aware directions can handle poorly scaled loss functions and sometimes reach useful training losses in fewer optimizer iterations . Compute cost: Curvature estimates, iterative linear solves, and repeated loss or gradient evaluations can make an individual update substantially more expensive. Fewer iterations do not necessarily mean less wall clock time. Memory cost: A dense Hessian for P parameters has P² entries. L BFGS avoids that matrix but still stores a history of vectors, and its performance depends on stable gradients. Numerical challenges: The Hessian can be singular or indefinite in nonconvex networks. Unmodified Newton directions may not be descent directions; damping, line searches, trust regions, or positive definite approximations help. When useful: Relatively small or smooth problems, deterministic or large batch objectives, some fine tuning settings, and specialized curvature aware deep learning research. SGD or Adam are often simpler and more cost effective at massive scale. PyTorch practical note: torch.optim.LBFGS accepts a closure that recomputes loss and gradients, because one optimizer step may involve multiple evaluations. It has additional memory cost proportional to its history size and is not always a good fit for noisy mini batches. Start with a smaller history and test real time to target loss, not just iteration counts. For the separate intuition behind parameter space geometry, see “What are sharp and flat minima in neural network optimization?” (Question 257). This article is about the optimizer families and their updates , not a repeated discussion of landscape shape. Key takeaway: Second order optimizers use curvature—or an approximation—to adjust gradient steps. They can improve convergence in some settings, but the extra computation and memory often make first order optimizers preferable for large deep networks. Research and documentation: Goodfellow, Bengio & Courville, Deep Learning , Chapter 8: Optimization for Training Deep Models; Vinyals & Povey (2012), Krylov Subspace Descent for Deep Learning ; PyTorch torch.optim.LBFGS reference.