Deep Learning
Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.
What is regression ?
Regression is a prediction task where a model estimates a continuous numerical value rather than choosing a class. Examples include: Predicting a house price Estimating temperature Forecasting sales Predicting delivery time How regression works A regression model receives input features and produces a numerical output. For example: Unlike classification, the output is not a class such as "cat" or "dog". It is a number that can vary across a continuous range. Regression in a neural network For many regression problems, the final layer produces one numerical value directly: The model is trained by comparing its prediction with the actual target using an appropriate loss function. For example: The loss measures this prediction error and guides training. See “What is a loss function in deep learning?” for the broader idea. Regression vs classification Regression: predicts a numerical value, such as price or temperature. Classification: predicts a category, such as spam/not spam or cat/dog/bird. For classification examples, see “What is Binary Classification ?” and “what is Multi class classification ?” . In short, regression is used when the output you want to predict is a continuous numerical value rather than a class label.
What is backpropagation?
Backpropagation is the method used to compute how much each trainable parameter in a neural network contributed to the model's loss. It works by moving backward through the network and calculating gradients using the chain rule of calculus. At a high level: Those gradients are then used by an optimizer to update the model's weights and biases. Why is backpropagation needed? A neural network may contain millions of trainable parameters. Backpropagation efficiently determines how the loss changes with respect to each one. For example, it computes quantities such as: These values tell the training process which parameters need to change and in which direction. Backpropagation itself computes gradients ; it does not perform the parameter update. The optimizer handles that next step. For the related concepts, see “What is a gradient in deep learning?” , “What is a loss function in deep learning?” , and “How does a neural network learn?” . A separate answer, “How does backpropagation work?” , can cover the step by step mechanics without repeating them here. In short, backpropagation is the backward gradient computation process that makes neural network training possible.
How does backpropagation work?
Backpropagation works by calculating gradients from the output layer back toward the earlier layers of a neural network. The process starts after the forward pass has produced a prediction and a loss value. Step 1: Measure the loss The network first compares its prediction with the target using a loss function. That loss becomes the starting point for gradient calculation. See “What is a loss function in deep learning?” for the role of the loss itself. Step 2: Start from the output Backpropagation begins at the output side of the network and determines how a small change in the final layer would affect the loss. Conceptually: Step 3: Apply the chain rule backward Each earlier layer affected the layers after it, so backpropagation uses the chain rule to propagate gradients backward through these dependencies. A simplified example is: To determine how a parameter in Layer 1 affects the loss, the gradients along the path are combined: This is repeated until gradients have been calculated for all trainable weights and biases. Step 4: Store the gradients After the backward pass, each trainable parameter has a gradient such as: The gradient tells us how sensitive the loss is to that parameter. See “What is a gradient in deep learning?” for a focused explanation. Step 5: The optimizer updates the parameters Backpropagation stops after computing the gradients. An optimizer then uses those gradients to update the parameters. So the distinction is: Backpropagation: computes gradients. Optimizer: uses those gradients to update parameters. In short, backpropagation works by starting from the loss and applying the chain rule backward through the network to calculate gradients for each trainable parameter.
What is gradient descent?
Gradient descent is an optimization method used to reduce a model's loss by repeatedly adjusting its trainable parameters in the direction that lowers the loss. A simplified update is: The gradient tells us the direction in which the loss increases, so gradient descent moves in the opposite direction. Simple intuition Imagine the loss as a landscape and the current model parameters as a point on that landscape. Gradient descent uses the local slope to decide which way to move toward a lower loss region. How it fits into training For the gradient itself, see “What is a gradient in deep learning?” . For how gradients are computed, see “How does backpropagation work?” . Role of the learning rate The learning rate controls the size of each update. Too small: training can be very slow. Too large: updates may overshoot useful regions or become unstable. A dedicated answer, “What is a learning rate?” , covers this in more detail. In short, gradient descent improves a neural network by repeatedly moving its parameters in the direction that reduces the loss.
What is an optimizer in deep learning?
An optimizer is the algorithm that updates a neural network's trainable parameters using the gradients computed during training. Backpropagation calculates the gradients. The optimizer uses those gradients to decide how the weights and biases should change. Basic idea A simple parameter update looks like: The optimizer controls how this update is applied. More advanced optimizers can also use information from previous updates or adapt the step size for individual parameters. Common optimizers SGD (Stochastic Gradient Descent): updates parameters directly from gradients, often with optional momentum. Adam: keeps running estimates of gradient behavior and adapts the update size for each parameter. The dedicated answer “What is the difference between SGD and Adam?” covers that comparison in detail. Why does the optimizer matter? The optimizer affects how quickly and how reliably the model moves toward a lower loss solution. A poor optimizer configuration can make training slow or unstable, even when the network architecture is otherwise suitable. The optimizer also depends heavily on the learning rate . See “What is a learning rate?” for that concept. For the optimization principle behind these updates, see “What is gradient descent?” . In short, an optimizer is the mechanism that turns gradients into parameter updates during neural network training.
What is the difference between SGD and Adam?
SGD and Adam are optimizers used to update neural network parameters during training. The main difference is how they use gradients to decide the size and direction of each update. SGD Stochastic Gradient Descent (SGD) updates parameters using the current gradient and a learning rate. With momentum, SGD can also use information from previous updates to smooth and accelerate training. SGD is simple, memory efficient, and can work very well when the learning rate and schedule are tuned carefully. Adam Adam (Adaptive Moment Estimation) keeps running estimates of: The average gradient The average squared gradient It uses these estimates to adapt the effective update size for each parameter. This often makes Adam easier to use as a strong default because it usually requires less manual tuning at the beginning of training. Key practical difference SGD: uses a more direct gradient based update and usually relies more heavily on learning rate tuning and scheduling. Adam: automatically adapts updates per parameter using gradient history. SGD: uses less optimizer state and therefore less extra memory. Adam: stores additional running statistics for each parameter. SGD: is often preferred in some settings when carefully tuned generalization is important. Adam: is commonly preferred for fast experimentation and many Transformer style or complex deep learning workloads. Neither optimizer is universally better. The best choice depends on the model, data, objective, and training setup. For the broader role of optimizers, see “What is an optimizer in deep learning?” . For the update size concept, see “What is a learning rate?” . In short, SGD applies relatively direct gradient updates, while Adam adapts updates using running statistics of past gradients.
What is a learning rate?
A learning rate controls how large each parameter update is during neural network training. A simplified gradient based update is: The gradient provides the direction of change, while the learning rate determines how far the optimizer moves in that direction . Why does it matter? Too small: training may improve very slowly and require many updates. Too large: the optimizer may overshoot useful regions, causing unstable training or preventing convergence. Well chosen: training can make steady progress toward a lower loss solution. Simple intuition Think of training as walking downhill toward a lower loss: The best value depends on the model, optimizer, batch size, and training setup. In practice, the learning rate may also change during training using a learning rate schedule rather than staying fixed. For related concepts, see “What is gradient descent?” and “What is an optimizer in deep learning?” . In short, the learning rate is the step size that controls how aggressively a neural network updates its parameters during training.
What are epochs in deep learning?
An epoch is one complete pass through the entire training dataset during neural network training. If a dataset contains 10,000 training examples, then one epoch means the model has processed all 10,000 examples once. Why are multiple epochs used? A model usually does not learn enough from seeing the training data only once. Training therefore repeats over the dataset for multiple epochs. With each epoch, the optimizer continues updating the model parameters based on the gradients computed during training. Epochs and batches Training data is usually split into smaller groups called batches rather than processed all at once. For example: So one epoch contains 10 parameter update steps in this example. A dedicated answer, “What is batch size in deep learning?” , covers batches in more detail. Can too many epochs be a problem? Yes. Training for too few epochs can leave the model undertrained, while training for too many can contribute to overfitting. Techniques such as early stopping can help decide when to stop training. In short, an epoch is one full pass through the training dataset, and deep learning models are usually trained for multiple epochs so their parameters can improve over repeated exposure to the data.
What is batch size in deep learning?
Batch size is the number of training examples processed together before the model performs one parameter update. For example, if a dataset has 1,000 examples and the batch size is 100: That means the optimizer performs 10 update steps during one epoch. Why use batches? Processing the entire dataset at once can require too much memory. Batches let training work on smaller chunks of data while still making frequent parameter updates. What happens with different batch sizes? Small batch size: uses less memory and produces noisier gradient estimates, but updates happen more frequently. Large batch size: uses more memory and produces smoother gradient estimates, but updates happen less frequently. The best batch size depends on the dataset, model, hardware, and optimizer settings. For the full pass concept, see “What are epochs in deep learning?” . The related optimization variants are covered in “What is the difference between batch, stochastic, and mini batch gradient descent?” . In short, batch size controls how many training examples are used to compute each parameter update.
What is the difference between batch, stochastic, and mini-batch gradient descent?
The difference is how much training data is used to compute each gradient update . Batch gradient descent Uses the entire training dataset before making one parameter update. It produces a stable gradient estimate, but each update can be slow and memory intensive for large datasets. Stochastic gradient descent Uses one training example at a time for each update. Updates are very frequent, but the gradient is noisy because it is based on only one example. Mini batch gradient descent Uses a small group of examples for each update. For example, with a batch size of 64, the model computes a gradient using 64 examples before updating its parameters. Mini batch training is the most common approach in modern deep learning because it offers a practical balance between: Computational efficiency Memory usage Gradient stability Frequent parameter updates For the meaning of batch size, see “What is batch size in deep learning?” . For the underlying optimization idea, see “What is gradient descent?” . In short, batch gradient descent uses all examples per update, stochastic gradient descent uses one example, and mini batch gradient descent uses a small subset of examples.