Deep Learning

Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.

What is cross-entropy loss?

Cross entropy loss is a loss function commonly used for classification problems. It measures how well the probabilities predicted by a model match the correct class labels. A prediction receives a small loss when the model assigns high probability to the correct class, and a large loss when it assigns low probability to the correct class. Simple intuition Suppose the correct class is cat . If the model predicts P(cat) = 0.95, the loss is small. If the model predicts P(cat) = 0.10, the loss is much larger. This means cross entropy does not only care whether the final class prediction is correct. It also considers how confident the model is . Binary cross entropy For binary classification, cross entropy is commonly written as: Loss = [y log(p) + (1 y) log(1 p)] Here: y is the true label, usually 0 or 1. p is the predicted probability of the positive class. Binary cross entropy is commonly paired with a sigmoid output. Multiclass cross entropy For multiclass classification, the model usually produces one probability for each class, often using softmax. If the correct class is c, the loss for one example can be written as: Loss = log(p c) where p c is the predicted probability of the correct class. Why is cross entropy widely used? Cross entropy strongly penalizes confident incorrect predictions and provides a smooth training signal that can be optimized with gradient based methods. For the broader idea of loss functions, see “What is a loss function in deep learning?” . For the common output functions, see “What is the sigmoid activation function?” and “What is the softmax function?” . In short, cross entropy loss measures how closely predicted class probabilities match the true labels, with larger penalties when the model gives low probability to the correct class.

What is mean squared error in neural networks?

Mean Squared Error (MSE) is a loss function commonly used for regression problems. It measures the average squared difference between a model’s predictions and the actual target values. The formula is: MSE = (1 / n) × Σ(y ŷ)² where: y is the actual value. ŷ is the predicted value. n is the number of examples. Simple example Suppose the actual values are: [3, 5] and the model predicts: [2, 7] The errors are: 3 2 = 1 5 7 = 2 After squaring: 1² = 1 ( 2)² = 4 So: MSE = (1 + 4) / 2 = 2.5 Why are the errors squared? Squaring has two important effects: Positive and negative errors cannot cancel each other out. Larger errors receive a much larger penalty than smaller errors. This makes MSE sensitive to large prediction mistakes. When is MSE used? MSE is commonly used when a neural network predicts continuous numerical values, such as: House prices Temperature Demand Numerical measurements Because MSE gives larger penalties to large errors, it can be strongly affected by outliers. For the broader task type, see “What is regression ?” . For the general concept of training objectives, see “What is a loss function in deep learning?” . In short, mean squared error measures the average squared distance between predicted and actual numerical values, making it a common loss function for regression.

What is gradient clipping?

Gradient clipping is a technique used to prevent gradients from becoming excessively large during neural network training. It is mainly used to control the exploding gradient problem , where very large gradients can cause unstable parameter updates, rapidly changing weights, or numerical errors. How gradient clipping works Before the optimizer updates the model parameters, the gradients are checked and limited if they are too large. Two common approaches are: Clip by value: restrict each gradient element to a fixed range. Clip by norm: scale the entire gradient vector when its norm exceeds a chosen threshold. For example, with norm clipping: if ||g|| threshold, scale g down This preserves the overall gradient direction while reducing its magnitude. Why is gradient clipping useful? It can help: Prevent extremely large parameter updates. Stabilize training. Reduce the chance of NaN or infinite values. Make recurrent and very deep networks easier to train in some cases. Gradient clipping does not solve the underlying cause of unstable gradients by itself. It acts as a safeguard during optimization. For the related problem, see “What is the exploding gradient problem?” . The detailed effect of clipping on training stability is covered separately in “How does gradient clipping stabilize neural network training?” . In short, gradient clipping limits excessively large gradients before the optimizer applies them, helping keep neural network training stable.

What is L1 and L2 regularization?

L1 and L2 regularization are techniques used to reduce overfitting by adding a penalty for large model weights to the training objective. They both encourage simpler models, but they penalize weights differently. L1 regularization L1 regularization adds the absolute values of the weights to the loss: Loss = Original Loss + λ × Σ|w| where: w represents the model weights. λ controls the strength of regularization. L1 can push some weights exactly to zero, which can create a sparse model where fewer features have strong influence. L2 regularization L2 regularization adds the squared values of the weights to the loss: Loss = Original Loss + λ × Σw² L2 usually reduces weights smoothly toward smaller values instead of forcing many of them exactly to zero. Main difference L1: encourages sparsity and can set some weights to zero. L2: discourages very large weights and usually keeps many weights small but nonzero. Why do they help? By penalizing overly large weights, L1 and L2 can reduce how strongly a model fits noise or small details in the training data. This can improve generalization to unseen data. What does λ do? The regularization strength λ controls how much the penalty influences training. If λ is too small, regularization may have little effect. If λ is too large, the model may become too simple and underfit. For the broader concept, see “What is regularization in deep learning?” . A related concept, weight decay , is covered separately in “What is weight decay in neural networks?” . In short, L1 regularization penalizes the absolute size of weights and promotes sparsity, while L2 regularization penalizes squared weights and encourages smaller, more distributed parameter values.

What is layer normalization?

Layer normalization is a normalization technique that normalizes the activations within each individual example rather than across a mini batch. It is commonly used in sequence models and Transformer architectures because its behavior does not depend on the batch size. How layer normalization works For a given example, layer normalization computes statistics across the features of that layer: Calculate the mean of the activations. Calculate their variance. Normalize the activations using those values. Apply learnable scale and shift parameters. A simplified form is: normalized = (x mean) / sqrt(variance + ε) Then the normalized values are adjusted using learned parameters: output = γ × normalized + β where γ and β are learned during training. Why is layer normalization useful? It can help: Stabilize the scale of activations. Make optimization more consistent. Work well even with very small batch sizes. Support sequence models where examples may have different lengths. How is it different from batch normalization? Batch normalization uses statistics collected across examples in a mini batch, while layer normalization computes statistics within each example. That difference makes layer normalization especially useful in architectures such as Transformers. The detailed comparison is covered separately in “What is the difference between batch normalization and layer normalization?” . In short, layer normalization normalizes the features of each individual example, helping stabilize training without depending on statistics from other examples in the batch.

What is the difference between batch normalization and layer normalization?

Batch normalization and layer normalization both normalize activations to make neural network training more stable, but they normalize across different dimensions. Batch normalization Batch normalization computes statistics using multiple examples in the same mini batch. For each feature or channel, it typically: Computes the mean across the batch. Computes the variance across the batch. Normalizes the activations. Applies learned scale and shift parameters. Because it depends on batch statistics, its behavior can be affected by very small batch sizes. During inference, batch normalization usually uses running estimates of the mean and variance collected during training. See “What is batch normalization?” for the dedicated explanation. Layer normalization Layer normalization computes statistics within each individual example rather than across the batch. For a given example, it normalizes across the features of the layer and then applies learned scale and shift parameters. This means layer normalization does not depend on other examples in the mini batch. It therefore behaves consistently even when batch sizes are small or vary between training runs. See “What is layer normalization?” for the dedicated explanation. Main differences Batch normalization: uses statistics across examples in a mini batch. Layer normalization: uses statistics across features within each individual example. Batch normalization: can be sensitive to batch size. Layer normalization: is largely independent of batch size. Batch normalization: commonly uses running statistics during inference. Layer normalization: normally uses the current example’s own statistics during both training and inference. Where are they commonly used? Batch normalization is widely used in many convolutional neural networks. Layer normalization is widely used in sequence models and Transformer architectures. Neither method is universally better. The appropriate choice depends on the model architecture, batch size, and training setup. In short, batch normalization normalizes using statistics across a batch, while layer normalization normalizes features within each individual example.

What are residual connections in neural networks?

Residual connections are connections that allow a neural network to pass an earlier layer’s output directly to a later layer, where it is combined with the transformed output. A common form is: output = F(x) + x where: x is the original input to a block. F(x) is the transformation learned by the layers inside the block. F(x) + x is the final output. Why are residual connections useful? Very deep neural networks can become difficult to train because information and gradients must pass through many layers. Residual connections create a shorter path through the network, which can help: Improve gradient flow. Make very deep networks easier to optimize. Preserve useful information from earlier layers. Reduce the degradation problem where adding more layers makes training performance worse. Simple intuition Instead of forcing a block to learn a completely new representation, the block can learn only the change that should be added to the existing representation. Input x → Transform F(x) → Add x → Output If the transformation is not useful, the network can learn a value close to zero for F(x), allowing the original input to pass through more directly. Where are residual connections used? Residual connections became especially well known through ResNet , where many residual blocks are stacked to build very deep convolutional networks. They are also widely used in other modern architectures, including Transformer based networks. For the architecture built around this idea, see “What is a ResNet?” . A closely related term, skip connection , is covered separately in “What is a skip connection in a neural network?” . In short, residual connections let a network add an earlier representation directly to a later transformed representation, helping deep models train more effectively.

What is a ResNet?

ResNet (Residual Network) is a deep neural network architecture that uses residual connections to make very deep networks easier to train. Instead of requiring every block to learn a completely new transformation, a ResNet block adds the original input back to the transformed output. A common form is: output = F(x) + x where x is the block input and F(x) is the transformation learned by the layers inside the block. Why was ResNet introduced? As neural networks become deeper, simply adding more layers can make optimization harder and can even reduce training accuracy. ResNet addresses this by creating shortcut paths that allow information and gradients to move more directly through the network. This makes it practical to train much deeper models. What is a residual block? A typical residual block contains: One or more convolutional layers. Normalization and activation operations. A shortcut connection that carries the input around those layers. An addition operation that combines the shortcut with the transformed output. A simplified flow is: Input x → Convolution layers → F(x) Input x ────────────────────┐ F(x) + x ←─────────────────┘ Why are ResNets useful? Residual networks can: Improve gradient flow. Make deeper networks easier to optimize. Preserve useful information across many layers. Support very deep computer vision models. Popular versions include ResNet 18, ResNet 34, ResNet 50, ResNet 101, and ResNet 152 , where the number roughly refers to the depth of the network. For the underlying idea, see “What are residual connections in neural networks?” . The internal building block is covered separately in “What is a residual block?” . In short, ResNet is a deep neural network architecture built around residual connections, allowing very deep models to train more effectively.

What is a fully connected layer?

A fully connected layer is a neural network layer in which every neuron receives input from every neuron in the previous layer. It is also called a dense layer or linear layer in many deep learning libraries. How it works Each output neuron computes a weighted combination of all input values, adds a bias, and may then pass the result through an activation function. A simplified form is: z = Wx + b where: x is the input vector. W contains the learned weights. b contains the learned biases. z is the layer output before an optional activation function. Because every input contributes to every output neuron, a fully connected layer can combine information from all features at once. Where are fully connected layers used? They are commonly used: In multilayer perceptrons. Near the end of CNNs to convert extracted features into predictions. As projection layers inside many neural network architectures. In classification and regression output heads. Why are they called fully connected? If a previous layer has 100 outputs and the fully connected layer has 20 neurons, each of those 20 neurons receives all 100 input values. That means the layer contains: 100 × 20 = 2,000 weights plus 20 bias values. Main limitation Fully connected layers can require many parameters when the input is large. This is one reason CNNs use convolutional layers for images instead of connecting every pixel directly to every neuron. For the basic components being learned, see “What are weights and biases in a neural network?” . In short, a fully connected layer connects every input feature to every output neuron, allowing the network to combine information across the entire input representation.

What is a convolutional layer?

A convolutional layer is a neural network layer that applies multiple learnable filters to an input to detect local patterns and produce feature maps. It is one of the core building blocks of a Convolutional Neural Network (CNN). How it works Each filter moves across the input and performs a convolution operation on small local regions. For each position, the filter computes a weighted combination of nearby input values. The result from one filter forms a feature map . If a layer contains many filters, it produces many feature maps: Input → Filter 1 → Feature map 1 Input → Filter 2 → Feature map 2 Input → Filter 3 → Feature map 3 Each filter can learn to respond to a different type of pattern. What does a convolutional layer learn? In early CNN layers, filters may learn simple patterns such as: Edges Corners Color transitions Simple textures Deeper convolutional layers can combine these simpler features into more complex patterns. Important settings The behavior and output size of a convolutional layer depend on settings such as: Kernel or filter size: the size of the local region examined at once. Stride: how far the filter moves between positions. Padding: whether extra values are added around the input boundary. Number of filters: how many output feature maps the layer produces. These concepts are covered separately in “What is a kernel or filter in a CNN?” , “What is stride in a convolutional neural network?” , and “What is padding in a CNN?” . For the underlying mathematical operation, see “What is convolution in deep learning?” . In short, a convolutional layer uses multiple learnable filters to scan local regions of an input and generate feature maps that represent useful patterns.