Deep Learning

Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.

What is ReLU and why is it widely used?

ReLU (Rectified Linear Unit) is an activation function commonly used in neural networks, especially in hidden layers. It keeps positive inputs unchanged and converts negative inputs to zero. Its definition is: So: How ReLU behaves ReLU is piecewise linear: For negative inputs, the output is zero. For positive inputs, the output increases linearly with the input. Mathematically: Why is ReLU widely used? ReLU became popular because it combines a useful nonlinearity with a very simple computation. 1. It is computationally inexpensive ReLU only needs a comparison with zero: There is no exponential or other expensive nonlinear operation in the basic function. 2. It helps deep networks model nonlinear relationships Although ReLU is linear on each side of zero, it is nonlinear as a whole. Placing ReLU between learned linear transformations prevents a deep network from collapsing into a single linear transformation. A simplified hidden layer can be written as: The combination of learned weights and ReLU allows the network to build piecewise linear decision functions and increasingly complex representations across layers. 3. Its gradient is simple for positive inputs For positive inputs: For negative inputs: At exactly zero, the derivative is not mathematically unique, so deep learning frameworks use a practical convention. Because the derivative is 1 in the positive region, gradients can pass through active ReLU units without being repeatedly shrunk by the activation itself. 4. It produces sparse activations Negative pre activation values become zero. Therefore, for a given input, some neurons may output exactly zero. For example: This creates sparse intermediate activations, where only a subset of units are active for that input. Example inside a neural network Suppose a neuron computes: After ReLU: If another neuron computes: then: A layer may therefore transform values like this: What is the dying ReLU problem? A ReLU neuron can become effectively inactive if its input remains negative for all or most training examples. In that region: If the gradient remains zero, the parameters feeding that unit may stop receiving useful updates through that activation path. This is commonly called the dying ReLU problem . It can be reduced by appropriate initialization, learning rate choices, normalization, or by using variants such as Leaky ReLU when suitable. ReLU vs Leaky ReLU Standard ReLU: Leaky ReLU: where α is a small positive constant. Leaky ReLU preserves a small gradient in the negative region, which can help prevent units from becoming permanently inactive. PyTorch example Output: In short, ReLU is widely used because it is simple, fast, introduces nonlinearity, and provides favorable gradient behavior for positive activations. Its main limitation is that units can become inactive when they remain in the negative region.

What is the sigmoid activation function?

The sigmoid activation function maps any real valued input to a value between 0 and 1. It is especially useful when a model output needs to represent a probability like score for a binary outcome. Its formula is: The function has an S shaped curve: How sigmoid transforms values Some example values are: Large negative inputs produce values close to 0, while large positive inputs produce values close to 1. A value of 0 always maps to: Why is sigmoid useful for binary classification? For a binary classifier, a neural network can produce a single raw score called a logit . For example: The transformed value can be interpreted as a probability like prediction for the positive class when the model and loss are configured appropriately. For example: A classification threshold such as 0.5 may then be applied when a hard class decision is required. Sigmoid derivative The derivative of sigmoid has a convenient form: where σ(x) is the sigmoid output. For example, if: then: The derivative reaches its maximum value at the center of the curve and becomes much smaller toward either extreme. What is saturation? When the input is very large or very negative, sigmoid approaches 1 or 0. For example: In these regions, the derivative becomes very small. This behavior is called saturation . During backpropagation, very small derivatives can reduce gradient magnitude as gradients move through many layers. This is one reason sigmoid is generally not the default choice for hidden layers in modern deep neural networks. Sigmoid in the output layer Sigmoid remains very useful in output layers where each output represents an independent binary decision. Examples include: Spam vs. not spam Fraud vs. legitimate transaction Disease present vs. absent Multiple independent labels in multilabel classification For multilabel classification, several output neurons can each use sigmoid independently: Unlike softmax, these values do not need to sum to 1 because multiple labels can be true at the same time. Sigmoid and binary cross entropy Sigmoid is commonly paired conceptually with binary cross entropy loss. However, in practical deep learning frameworks, it is often better to use a loss function that accepts raw logits and performs the sigmoid transformation internally for improved numerical stability. For example, in PyTorch: BCEWithLogitsLoss combines the sigmoid operation with binary cross entropy in a numerically stable implementation. If probabilities are needed for inference, sigmoid can be applied explicitly: Main strengths and limitations Output range: 0 to 1. Shape: Smooth S shaped curve. Common use: Binary or multilabel probability like outputs. Main strength: Naturally maps unrestricted scores into a probability like range. Main limitation: Saturation causes very small gradients at extreme input values. In short, sigmoid converts an unrestricted real valued score into a value between 0 and 1, making it especially useful for binary and multilabel outputs. Its main drawback is saturation, which makes it less suitable for many hidden layer applications.

What is the softmax function?

Softmax is a function that converts a vector of raw scores into a probability distribution across multiple classes. For a vector of scores z, softmax for class i is: This guarantees that every output is between 0 and 1 and that all outputs sum to 1. Example Suppose a model produces three class scores: After softmax, they become approximately: These can be interpreted as the model's relative probabilities for the three mutually exclusive classes. Where is softmax used? Softmax is commonly used at the output of a multiclass classification model when exactly one class should be selected from several possible classes. For example: The class with the highest probability is typically chosen as the prediction. Important distinction Softmax compares classes against each other, so increasing the probability of one class affects the others. For problems where multiple labels can be true at the same time, sigmoid is usually more appropriate. See the existing answer on “What is the sigmoid activation function?” for that case. Numerical stability In practice, implementations subtract the largest input score before exponentiation: This produces the same probabilities while reducing the risk of numerical overflow. In short, softmax turns multiple class scores into a normalized probability distribution, making it especially useful for single label multiclass classification.

What are weights and biases in a neural network?

Weights and biases are the main trainable parameters in a neural network. They control how strongly inputs influence a neuron and how easily that neuron becomes active. A neuron typically computes: Here: x values are inputs. w values are weights . b is the bias . What are weights? A weight determines the importance and direction of an input's influence. For example: A large positive weight strengthens an input's positive contribution, while a negative weight can reduce or reverse its effect. What is a bias? The bias is an additional trainable value added after the weighted inputs are combined. It allows a neuron to shift its response instead of being forced to depend only on the input values. For example: Why are they important? During training, the network adjusts its weights and biases so its predictions better match the training objective. Conceptually: In short, weights control the influence of inputs, while biases shift the neuron's response. Together, they are the parameters a neural network learns from data.

How does a neural network learn?

A neural network learns by adjusting its trainable parameters so that its predictions produce a lower error on the training data . At a high level, training repeats four steps: 1. Forward pass The network receives an input and produces a prediction using its current parameters. For example: The parameters involved are covered in the existing answer “What are weights and biases in a neural network?” . 2. Calculate the loss A loss function measures how far the prediction is from the desired target. For example: The loss converts that mismatch into a numerical value the training process can minimize. Different tasks use different loss functions, such as cross entropy for many classification problems or mean squared error for some regression problems. 3. Backpropagation computes gradients Backpropagation works backward through the network and calculates how much each trainable parameter contributed to the loss. It computes gradients such as: A gradient tells the optimizer which direction would increase the loss, allowing it to move parameters in the opposite direction. 4. The optimizer updates the parameters An optimizer such as SGD or Adam changes the parameters using the gradients. A simplified gradient descent update is: The learning rate controls how large each update is. Repeating the process The network performs these steps repeatedly across many training examples: As training progresses, the parameters can move toward values that reduce the loss and improve performance on the training objective. The goal is not simply to memorize the training data, but to learn patterns that also work on unseen data. Techniques such as validation, regularization, and appropriate training duration are used to help evaluate and improve generalization. In short, a neural network learns by making predictions, measuring their error, computing gradients with backpropagation, and repeatedly updating its parameters with an optimizer.

What is a forward pass in a neural network?

A forward pass is the process of sending input data through a neural network, layer by layer, to produce an output or prediction. For a simple network: At each layer, the network applies its current weights and biases, followed by the layer's activation or other transformation. For a fully connected layer, a simplified step is: where x is the incoming data, W and b are learned parameters, and f is the activation function. Example Suppose a neuron receives: Its pre activation value is: That value is then transformed and passed to the next layer. This continues until the network produces its final output. Why is the forward pass important? The forward pass is how the model makes predictions during both training and inference . During training, the prediction from the forward pass is compared with the target using a loss function. The resulting loss is then used in backpropagation to compute parameter updates. For those concepts, see the existing answers “What are weights and biases in a neural network?” , “What is an activation function in deep learning?” , and “How does a neural network learn?” . In short, a forward pass is the one way flow of data through the network that produces its current prediction.

What is a loss function in deep learning?

A loss function measures how far a model's prediction is from the desired target. It converts that error into a number that training tries to minimize. For example, if a model predicts: a loss function assigns a value representing how incorrect that prediction is. Why is a loss function needed? A neural network cannot improve unless it has a measurable objective. The loss provides that objective. During training: A lower loss generally means the model is performing better on the training objective, although low training loss alone does not guarantee good performance on unseen data. Common loss functions The correct loss depends on the task. Binary classification: Binary cross entropy is commonly used. Multiclass classification: Cross entropy is commonly used. Regression: Mean squared error (MSE) or another regression loss is commonly used. Each of these deserves its own explanation rather than being expanded here. Loss vs. accuracy Loss and accuracy are not the same thing. Accuracy tells you how many predictions were correct, while loss also reflects how wrong or how confident the predictions were, depending on the chosen loss function. Because loss is differentiable in typical training setups, it can be used to compute gradients and update the model. For the broader training flow, see “How does a neural network learn?” and “What is a forward pass in a neural network?” . In short, a loss function gives the neural network a numerical objective to minimize so it can learn from its prediction errors.

What is a gradient in deep learning?

A gradient tells us how the loss changes when a model parameter changes slightly. It is the key signal used to decide how weights and biases should be updated during training. For a parameter w, the gradient is written as: It measures the rate of change of the loss with respect to w. How to interpret a gradient Suppose: Increasing the parameter would increase the loss, so an optimizer will usually move the parameter in the opposite direction. If: increasing the parameter would reduce the loss locally, so the update direction changes accordingly. The sign indicates direction, while the magnitude indicates how sensitive the loss is to that parameter. Why gradients matter A neural network may contain millions or billions of trainable parameters. Gradients provide a systematic way to determine how each parameter should change to reduce the loss. A simplified update is: The gradients themselves are typically computed using backpropagation , while an optimizer uses them to update the parameters. For related concepts, see “What is a loss function in deep learning?” , “What are weights and biases in a neural network?” , and “How does a neural network learn?” . In short, a gradient measures how sensitive the loss is to a model parameter and provides the direction used to improve that parameter during training.

What is Binary Classification ?

Binary classification is a classification task where a model chooses between two possible classes . Examples include: Spam or not spam Fraud or legitimate Disease present or absent Customer churn or no churn The two classes are often represented as: How it works A neural network typically produces a score for the positive class. That score can be converted into a value between 0 and 1 and compared with a threshold. For example: The threshold does not always have to be 0.5; it can be adjusted depending on the cost of false positives and false negatives. Common setup in deep learning Binary classification commonly uses: A single output score The sigmoid activation function to produce a probability like value Binary cross entropy as the training loss For the activation itself, see “What is the sigmoid activation function?” . Loss functions are introduced in “What is a loss function in deep learning?” . Binary vs multiclass classification Binary classification has exactly two classes. If a model must choose one class from three or more mutually exclusive classes, that is a multiclass classification problem. In short, binary classification is the task of assigning an input to one of two possible classes, often using a probability like score and a decision threshold.

what is Multi-class classification ?

Multi class classification is a classification task where a model must choose one class from three or more mutually exclusive classes . Examples include: Classifying an image as cat, dog, or bird Recognizing a handwritten digit from 0 to 9 Predicting one product category from several possible categories How it works A model usually produces one score for each class. For example: These raw scores can be converted into probabilities using the softmax function : The class with the highest probability is typically selected as the prediction. For how softmax performs this conversion, see “What is the softmax function?” . Multi class vs binary classification In binary classification , the model chooses between exactly two classes. In multi class classification , the model chooses one class from three or more possibilities. For the two class case, see “What is Binary Classification ?” . Important distinction Multi class classification assumes the classes are mutually exclusive for a given prediction. If multiple labels can be correct at the same time, that is a different problem called multi label classification . In short, multi class classification assigns an input to one class from several possible classes, commonly using one output score per class and softmax to interpret those scores as probabilities.