Deep Learning

Deep learning is a branch of machine learning that uses neural networks with multiple layers to learn useful representations and patterns from data. Instead of requiring a developer to manually define every feature, a deep learning model can learn increasingly useful representations as information passes through its layers. Basic idea A simplified deep neural network looks like this: Each layer transforms the representation produced by the previous layer. Earlier layers often learn relatively simple patterns, while later layers can combine them into more task-specific representations. For example, in an image-recognition system, learned representations may progress from simple visual patterns toward shapes, object parts, and features useful for identifying an object. The exact representations are learned from data rather than explicitly programmed. What happens inside a neural network? A neuron typically receives several inputs, multiplies them by learned weights , adds a bias , and applies an activation function . A simplified neuron can be written as: Where: - x values are inputs. - w values are learned weights. - b is a learned bias. - The activation function introduces non-linearity. A network contains many such computations organized into layers. How does a deep learning model learn? Training usually repeats four main steps: During the forward pass , the network produces a prediction. A loss function measures how far that prediction is from the desired output. Backpropagation computes gradients that describe how the model parameters contributed to the loss. An optimizer, such as stochastic gradient descent or Adam, then uses those gradients to update the parameters. Repeating this process across training examples gradually improves the model for its objective. Why is it called "deep"? The word deep refers to the use of multiple computational layers between the input and output. A network with many learned layers is commonly called a deep neural network . There is no universal number of layers at which a neural network suddenly becomes "deep." The term generally distinguishes multi-layer architectures from simpler or shallower models. Where is deep learning used? Deep learning is widely used in areas such as: - Computer vision - Speech recognition - Natural language processing - Recommendation systems - Generative models - Time-series and signal processing - Robotics and perception Different problems use different architectures, including convolutional neural networks, recurrent neural networks, Transformers, autoencoders, and other specialized neural architectures. Deep learning vs traditional machine learning A useful distinction is representation learning . In many traditional machine-learning workflows, humans spend substantial effort designing features before training a model. Deep learning can learn useful representations directly from raw or lightly processed data, although data preparation and domain knowledge still matter. Deep learning is especially effective when large datasets and sufficient compute are available, but it is not automatically the best solution for every problem. Simpler models can be faster, easier to interpret, and more effective on small or structured datasets. In short, deep learning trains multi-layer neural networks to learn representations from data and use those representations to make predictions or generate outputs.

Deep learning works by passing data through a multi-layer neural network , measuring how good the network's output is, and repeatedly adjusting its learned parameters to improve performance. At a high level, training follows this loop: 1. Data enters the network The input must first be represented numerically. Depending on the problem, this might be image pixels, audio samples, numerical features, or vector representations of tokens. The input is passed into the first layer of the neural network. 2. Layers transform the data Each layer receives values from the previous layer and computes new representations. A simplified neuron computes: Here, x represents inputs, w represents weights, b is a bias, and the activation function produces the neuron's output. Weights and biases are the parameters the network learns during training. 3. The forward pass produces a prediction Information moves from the input through the network's layers toward the output. For an image classifier: This computation from input to output is called the forward pass . 4. A loss function measures the error During supervised training, the prediction is compared with the correct target using a loss function . For example: The appropriate loss depends on the task. Classification commonly uses cross-entropy-based losses, while regression often uses losses such as mean squared error. 5. Backpropagation calculates gradients The network needs to determine how changing each parameter would affect the loss. Backpropagation efficiently computes gradients of the loss with respect to the network's parameters by applying the chain rule through the computational graph. Conceptually: Backpropagation calculates gradients; it does not itself decide the parameter-update rule. 6. The optimizer updates the parameters An optimizer uses the gradients to modify weights and biases. A simplified gradient-descent update is: The learning rate controls the size of each update. Common optimizers include stochastic gradient descent SGD and Adam. 7. Training repeats over many examples Training data is commonly processed in mini-batches . After many parameter updates across the dataset, the network can become better at minimizing its training objective. One complete pass through the training dataset is called an epoch . Training usually continues for multiple epochs while performance is also evaluated on data that is not being used for parameter updates. 8. The trained model performs inference After training, the model can process new inputs using the parameters it learned. During ordinary inference, the model generally performs a forward computation but does not run backpropagation or update its weights. Why multiple layers help Deep networks can learn hierarchical or compositional representations. A layer transforms the representation produced by previous layers, allowing later computations to build on earlier ones. The exact representations depend on the architecture, training objective, and data. In short, deep learning works by repeatedly making predictions, measuring error, computing gradients with backpropagation, and updating neural-network parameters with an optimizer. After training, the learned parameters are used to make predictions on new data.

Deep learning is a subset of machine learning. The main difference is that deep learning uses multi-layer neural networks to learn representations from data, while machine learning is a broader field that also includes algorithms such as linear regression, decision trees, random forests, support vector machines, and gradient-boosted trees. Machine learning Machine learning develops models that learn patterns from data and use those patterns to make predictions or decisions. A traditional supervised machine-learning workflow may look like: For example, when predicting house prices, useful inputs might include floor area, number of bedrooms, location-derived variables, and property age. These features can be supplied to models such as linear regression or gradient-boosted trees. Deep learning Deep learning uses neural networks containing multiple computational layers. These networks can learn useful internal representations as part of training. This is particularly valuable for high-dimensional unstructured data such as images, audio, and text. For example, an image model can learn representations useful for recognizing objects without a developer manually specifying every visual feature. Key differences | Aspect | Machine learning | Deep learning | |---|---|---| | Scope | Broad field | Subfield of machine learning | | Models | Regression, trees, SVMs, boosting, neural networks, etc. | Primarily multi-layer neural networks | | Feature engineering | Often important, depending on model and data | Can learn many useful representations automatically | | Data | Can work very well with smaller or structured datasets | Often benefits substantially from larger datasets | | Compute | Many models train efficiently on CPUs | Training commonly benefits from GPUs or other accelerators | | Training time | Often relatively short | Can be computationally expensive | | Unstructured data | Requires appropriate representations/features | Particularly effective for images, audio, text, and similar data | | Interpretability | Some model families are relatively interpretable | Deep networks are often harder to interpret directly | | Scaling | Depends strongly on algorithm | Many architectures benefit from additional data, parameters, and compute | Example: image classification A traditional ML pipeline might involve explicitly extracting useful image features before passing them to a classifier: A deep-learning approach can train a neural network to learn task-relevant representations and the final prediction function together: This does not mean deep learning requires no preprocessing or domain knowledge. Data quality, architecture, objective design, evaluation, and training strategy remain important. When traditional machine learning may be preferable Deep learning is not automatically the best choice. Traditional ML models can be strong choices when: - The dataset is relatively small. - The data is primarily structured or tabular. - Training speed or computational cost is important. - Model behavior needs to be easier to inspect. - A simpler model already provides sufficient performance. Tree-based methods, for example, are often strong baselines for tabular prediction problems. When deep learning is especially useful Deep learning is commonly used when working with: - Images and video - Speech and audio - Natural language - Large-scale representation learning - Generative modeling - Complex high-dimensional signals Modern systems may also combine both approaches. A deep model can generate embeddings, for example, and those embeddings can then become features for another machine-learning model. In short, machine learning is the broader discipline of learning predictive patterns from data, while deep learning is a machine-learning approach centered on multi-layer neural networks and learned representations.

A neural network is a machine-learning model made of interconnected computational units, commonly called neurons , organized into layers. It learns a mapping from inputs to outputs by adjusting numerical parameters called weights and biases . A simple feed-forward neural network can be visualized as: What does a neuron do? A neuron receives one or more input values. Each input is multiplied by a weight, the results are added together with a bias, and an activation function is applied. Where: - x₁ ... xₙ are input values. - w₁ ... wₙ are learned weights. - b is a learned bias. - activation is an activation function. Weights determine how strongly different inputs affect the computation, while the bias allows the neuron to shift its response. Layers in a neural network A typical neural network contains three kinds of layers. Input layer The input layer receives the numerical representation of the data. For example, a model predicting a house price might receive: Hidden layers Hidden layers perform learned transformations of the data. Networks with multiple hidden or computational layers are commonly associated with deep learning . Output layer The output layer produces the model's final result. Its structure depends on the task. Examples include: - A numerical value for regression. - Scores or probabilities for classification. - Multiple output values for multi-label or multi-output tasks. How does a neural network learn? Initially, many network parameters are initialized without encoding the final solution. Training repeatedly adjusts them based on data. A simplified training cycle is: The forward pass generates a prediction. The loss function measures how well that prediction matches the training objective. Backpropagation computes gradients of the loss with respect to the network parameters. An optimizer , such as SGD or Adam, uses those gradients to update the parameters. Repeating this process across many training examples allows the network to learn patterns useful for its task. Why are activation functions important? If every layer performed only linear transformations, stacking many such layers would still represent a linear transformation. Nonlinear activation functions allow neural networks to represent much more complex functions. Common activation functions include: - ReLU - Sigmoid - Tanh - GELU - Softmax, commonly used at an output layer for multiclass probability distributions Example Suppose a network classifies an image as a cat or dog: During training, the network adjusts its parameters so its outputs better match the correct labels according to the chosen loss function. Neural networks and deep learning A neural network does not necessarily have to be very deep. Deep learning generally refers to neural-network systems containing multiple learned layers that build increasingly useful representations. Different neural-network architectures are designed for different kinds of problems, including convolutional neural networks, recurrent neural networks, and Transformers. In short, a neural network is a layered mathematical model that learns from data by adjusting weights and biases so that its inputs are transformed into useful outputs.

An artificial neuron is a basic computational unit used in artificial neural networks. It receives one or more numerical inputs, combines them using learned weights and a bias , and usually applies an activation function to produce an output. It is inspired loosely by the idea of biological neurons, but it is fundamentally a mathematical computation rather than a detailed simulation of a biological neuron. Basic structure A simplified artificial neuron works like this: Mathematically: Where: - x₁, x₂, ..., xₙ are the inputs. - w₁, w₂, ..., wₙ are the weights. - b is the bias. - f is an activation function. - y is the neuron's output. What do the weights do? A weight controls how strongly an input contributes to the neuron's calculation. For example: The weighted sum is: The activation function then transforms z into the neuron's output. Why is the bias needed? The bias is an additional learned parameter. It shifts the neuron's pre-activation value independently of the inputs. Without a bias: With a bias: This gives the model additional flexibility when fitting relationships in data. What does the activation function do? The activation function transforms the weighted sum before it is passed onward. For example, ReLU is: So: Nonlinear activation functions are important because they allow networks composed of many neurons to represent nonlinear relationships. Without nonlinear activations between layers, stacking linear transformations would still result in an overall linear transformation. How neurons form a neural network A single artificial neuron performs a relatively simple computation. Neural networks become powerful by organizing many such units into layers. Outputs from neurons in one layer become inputs to neurons or other computational units in subsequent layers. How does an artificial neuron learn? The neuron does not independently decide its weights. During neural-network training, the model calculates a loss and uses backpropagation to compute gradients for parameters throughout the network. An optimizer then updates weights and biases. A simplified gradient-descent update is: Repeated updates allow the network to learn parameter values that better minimize its training objective. Simple Python example This code implements the core computation of a simple artificial neuron: weighted inputs + bias + activation . In short, an artificial neuron takes numerical inputs, combines them using learned weights and a bias, applies an activation function, and passes the resulting value to the next part of a neural network.

Neural networks come in many architectures, each designed around particular data structures, computational patterns, or learning tasks. Some of the most important types include feed-forward neural networks, convolutional neural networks, recurrent neural networks, Transformers, autoencoders, generative adversarial networks, and graph neural networks . 1. Feed-forward neural networks A feed-forward neural network FNN passes information from the input toward the output without recurrent feedback connections. A common example is the multilayer perceptron MLP , usually composed of fully connected layers. Typical uses include: - Classification - Regression - Learning nonlinear mappings from fixed-size feature vectors 2. Convolutional neural networks A convolutional neural network CNN uses convolution operations to learn local patterns while sharing parameters across positions. CNNs are particularly well suited to grid-like data and have been widely used for: - Image classification - Object detection - Image segmentation - Medical-image analysis - Audio and signal processing Their local receptive fields and shared filters make them effective at detecting patterns such as edges, textures, and increasingly complex spatial features. 3. Recurrent neural networks A recurrent neural network RNN maintains a hidden state that carries information across sequence steps. RNNs were widely used for sequential data such as: - Text - Speech - Time series - Sequential sensor data Standard RNNs can struggle with long-range dependencies because of issues such as vanishing or exploding gradients. 4. LSTM networks Long Short-Term Memory LSTM networks are a type of recurrent neural network designed with gating mechanisms that help control information flow through the recurrent state. An LSTM cell includes gates that regulate what information is retained, updated, and exposed. LSTMs have historically been used for: - Sequence classification - Speech recognition - Time-series modeling - Language modeling They generally handle longer dependencies better than basic RNNs, although many sequence-modeling applications now also use Transformer-based architectures. 5. GRU networks A Gated Recurrent Unit GRU is another gated recurrent architecture. GRUs use a simpler gating structure than LSTMs while serving a similar purpose: controlling how information is retained and updated over a sequence. 6. Transformers A Transformer is a neural-network architecture built around attention mechanisms rather than recurrence as its primary method for modeling relationships between sequence elements. A simplified Transformer block contains: Transformers are widely used in: - Natural language processing - Large language models - Computer vision - Speech and audio - Multimodal systems Self-attention allows tokens or other elements to directly incorporate information from other positions in the input. 7. Autoencoders An autoencoder learns to encode an input into a latent representation and then reconstruct the original input. Autoencoders can be used for: - Representation learning - Dimensionality reduction - Denoising - Anomaly detection A variational autoencoder VAE extends this idea with a probabilistic latent-variable formulation and can be used as a generative model. 8. Generative adversarial networks A Generative Adversarial Network GAN contains two neural networks trained in competition: - A generator creates synthetic samples. - A discriminator attempts to distinguish real samples from generated samples. GANs have been used extensively for image generation, image-to-image translation, and other generative tasks. 9. Graph neural networks A Graph Neural Network GNN is designed to operate on graph-structured data containing nodes and edges. GNNs commonly aggregate information from neighboring nodes to create learned node, edge, or graph representations. Applications include: - Molecular modeling - Recommendation systems - Knowledge graphs - Fraud detection - Network analysis 10. Specialized and hybrid architectures Modern neural-network systems frequently combine architectural ideas rather than belonging to exactly one category. Examples include: - CNNs combined with attention - Transformer-based vision models - Graph Transformers - Encoder-decoder architectures - Neural networks with convolutional, recurrent, and attention components The architecture should be chosen based on the structure of the data, the learning objective, computational constraints, and empirical performance. Quick comparison | Neural network type | Main idea | Common applications | |---|---|---| | FNN / MLP | Forward flow through fully connected layers | Classification, regression | | CNN | Local filters and parameter sharing | Images, vision, signals | | RNN | Recurrent hidden state | Sequential data | | LSTM | Gated recurrent memory | Longer sequence dependencies | | GRU | Simplified gated recurrence | Sequence modeling | | Transformer | Attention-based interactions | Language, vision, multimodal | | Autoencoder | Encode and reconstruct | Representation learning, denoising | | GAN | Generator vs discriminator | Generative modeling | | GNN | Message passing over graphs | Graph-structured data | In short, different neural-network architectures introduce different computational structures for learning from different kinds of data. There is no single best neural-network type; the appropriate architecture depends on the problem and its constraints.

A neural network is commonly described as having input, hidden, and output layers . These terms describe different stages in the flow of information through the network. The input layer receives the data, hidden layers transform it into useful internal representations, and the output layer produces the model's final prediction or representation. 1. Input layer The input layer represents the features supplied to the neural network. Suppose a model predicts the price of a house using four features: The input vector might be: Each value represents one input feature after whatever preprocessing the model requires. For an image, inputs might originate from pixel values. For text, the network may receive token embeddings. The exact form depends on the architecture. 2. Hidden layers A hidden layer sits between the input and output stages and performs learned transformations. A simple fully connected hidden layer can be written as: Where: - x is the input vector. - W is a matrix of learned weights. - b is a learned bias vector. - f is an activation function. - h is the resulting hidden representation. With several hidden layers: Each layer receives a representation and transforms it for the next stage. The word hidden does not mean the layer is secret. It means its values are internal to the model rather than being the original input or the final output. What do hidden layers learn? During training, the parameters in hidden layers are adjusted so the network learns representations useful for minimizing its objective. For an image-recognition model, different stages may learn representations associated with increasingly complex visual patterns. A simplified intuition is: This hierarchy is not guaranteed to map neatly to human-defined concepts, but it illustrates how deeper networks can compose transformations across layers. 3. Output layer The output layer produces the final values required by the task. Its design depends strongly on the problem. Binary classification A model answering a two-class question might produce a single score transformed with a sigmoid function: Multiclass classification For several mutually exclusive classes, the output may contain one logit per class, often followed by softmax when probabilities are required. Regression A house-price model might output one numerical prediction: Some tasks require multiple output values instead. Example of information flow Consider a network that recognizes handwritten digits: During a forward pass, information moves through these stages to generate the prediction. During training, backpropagation computes gradients through the network so the optimizer can update trainable parameters. Are all neural networks organized the same way? No. The input-hidden-output terminology is easiest to visualize with a standard feed-forward network, but modern architectures can contain more specialized structures. Examples include: - CNNs with convolutional layers - RNNs with recurrent connections - Transformers with attention and feed-forward blocks - Autoencoders with encoder and decoder components - Graph neural networks with message-passing layers Even in these architectures, the general idea remains useful: data enters the model, passes through learned internal transformations, and produces an output. Summary | Layer | Main role | Example | |---|---|---| | Input | Represents data supplied to the model | Pixels, numerical features, embeddings | | Hidden | Learns intermediate transformations and representations | Dense, convolutional, recurrent, attention-based computations | | Output | Produces task-specific results | Class scores, probabilities, numerical predictions | In short, the input layer receives the model's data, hidden layers transform that information through learned parameters, and the output layer produces the result required by the task.