Deep Learning

Deep learning is a branch of machine learning that uses neural networks with many layers. These layers learn representations directly from data, making deep learning especially useful for images, language, audio, and multimodal systems.

How does backpropagation through time (BPTT) train recurrent neural networks?

Backpropagation through time (BPTT) trains a recurrent neural network (RNN) by unfolding its repeated hidden state calculations across time and applying the chain rule backward through that unfolded sequence. Each time step uses the same learned weights , so gradient contributions from their different uses are added before the optimizer updates those weights. How BPTT works, step by step 1. Unroll the RNN: Represent time steps 1, 2, …, T as a chain of computations. They are not separate networks with separately learned weights. 2. Forward pass: Compute each hidden state from the current input and the previous state, such as hₜ = tanh(Wₓₕxₜ + Wₕₕhₜ₋₁ + b). Produce predictions and calculate a loss (at the final step or at multiple steps). 3. Backward pass: Starting from the loss, propagate derivatives backward along the unrolled connections using the chain rule. Later step losses can send learning signals to earlier states. 4. Accumulate shared weight gradients: Because Wₕₕ is reused at every step, its total gradient sums the contributions of all its uses , including effects passing through future states. 5. Update once: An optimizer uses the accumulated gradient to update the shared weights. The updated weights are then reused at every time step of the next forward pass. Unrolled view: h₀ → [step 1, W] → h₁ → [step 2, W] → h₂ → [step 3, W] → h₃ → loss. The label W refers to one shared parameter , not three separately trained parameters. Backward gradients follow this chain in the opposite direction. Beginner friendly numerical example: three time steps Use an intentionally simple scalar, linear RNN so every gradient contribution is visible. Let hₜ = w × hₜ₋₁ + xₜ, with initial state h₀ = 1, inputs x₁ = x₂ = x₃ = 0, and shared recurrent weight w = 0.5 . Forward pass Step 1: h₁ = 0.5 × 1 = 0.5. Step 2: h₂ = 0.5 × 0.5 = 0.25. Step 3: h₃ = 0.5 × 0.25 = 0.125. Suppose the target at step 3 is 0. The final step squared error loss is L = ½(h₃ − 0)² = 0.0078125. Backward pass: how the SAME weight receives three contributions The starting gradient is dL/dh₃ = h₃ = 0.125. Following the three paths where w was used: Use at step 3: 0.125 × h₂ = 0.125 × 0.25 = 0.03125. Use at step 2: 0.125 × w × h₁ = 0.125 × 0.5 × 0.5 = 0.03125. Use at step 1: 0.125 × w² × h₀ = 0.125 × 0.25 × 1 = 0.03125. Total shared weight gradient: dL/dw = 0.03125 + 0.03125 + 0.03125 = 0.09375. With learning rate 0.1, one gradient descent update gives w new = 0.5 − 0.1 × 0.09375 = 0.490625 . There is one update to w , not three independent updates. Real RNNs have vectors, matrices, nonlinear activations, and often losses at many steps; automatic differentiation performs the same chain rule bookkeeping. Why can gradients vanish or explode? To reach an earlier time step, a gradient passes through repeated recurrent weight and activation derivatives . Multiplying factors with small magnitude over many steps can make the signal vanish ; repeatedly large factors can make it explode . In this toy linear model, the backward signal crossing each connection gets another factor of w = 0.5. With nonlinear matrix valued RNNs, the behavior depends on products of Jacobians—not simply one weight. Small gradients make long range relationships hard to learn; very large gradients can destabilize training. Gradient clipping can mitigate exploding gradients, while gated architectures such as LSTMs and GRUs help preserve useful signals over time. See the existing vanishing/exploding gradient questions for more detail. Key takeaway: BPTT is ordinary backpropagation applied to an RNN unfolded across time . Its distinctive feature is that all time steps contribute to gradients of the same recurrent parameters . Related: “What is a recurrent neural network (RNN)?”, “What is the vanishing gradient problem?”, and “What is the exploding gradient problem?” The next question covers truncated backpropagation through time as a separate topic. Research: Werbos (1990), Backpropagation through time: what it does and how to do it ; Pascanu, Mikolov & Bengio (2013), On the difficulty of training recurrent neural networks ; Goodfellow, Bengio & Courville, Deep Learning , Chapter 10.

What is truncated backpropagation through time?

Truncated backpropagation through time (TBPTT) trains a recurrent neural network (RNN) on a long sequence by limiting each backward pass to a shorter window of time steps. Crucially, it can carry the hidden state value forward across windows while stopping gradients from flowing back through earlier windows . Why truncate the backward pass? Processing a very long sequence with one connected training graph requires keeping many intermediate activations for gradient calculation. TBPTT controls this cost by choosing a window length K and handling the sequence in manageable pieces. How does TBPTT work? 1. Split the sequence: Choose a truncation length K, such as 3, 20, or 50 time steps, based on the task and memory budget. 2. Run one window forward: Feed its inputs through the RNN, starting from the hidden state produced by the preceding window (or an initial state for the first window). 3. Compute loss and backpropagate: Use losses in the current window to calculate gradients through at most that window's recurrent computations. Update the same model parameters with the optimizer. 4. Carry the hidden state value forward: Reuse the last hidden state value when processing the next window; this preserves available sequence context. 5. Detach its gradient history: Break the automatic differentiation link to the previous window before the next one begins. In PyTorch, a tensor operation such as hidden = hidden.detach() expresses this idea. Detach is not the same as resetting hidden to zero. Beginner friendly example: 12 steps, windows of 3 Suppose an RNN processes 12 consecutive words and we choose K = 3 : Window 1: Steps 1–3 → compute loss and gradients over those 3 steps → carry hidden state h₃. Window 2: Steps 4–6 → start from the value of h₃, but detach its old computation graph → train over steps 4–6 → carry h₆. Window 3: Steps 7–9 → start from detached h₆ → train over steps 7–9 → carry h₉. Window 4: Steps 10–12 → start from detached h₉ → train over steps 10–12. That is 12 ÷ 3 = 4 windows . At the boundary between windows 1 and 2, the model does not forget the numerical value of h₃ ; it merely stops backpropagating from later losses into the computations that created h₃. Tiny numeric illustration: To see the difference, temporarily hold a simple RNN's weight fixed and use hₜ = 0.5 × hₜ₋₁ + xₜ, with h₀ = 0, x₁ = 1, and all later inputs 0. Window 1: h₁ = 1, h₂ = 0.5, h₃ = 0.25. Window 2: Start from the carried h₃ = 0.25 , giving h₄ = 0.125, h₅ = 0.0625, h₆ = 0.03125. Even though the step 6 prediction still uses information passed forward from step 1, the step 6 loss cannot send a gradient through detached h₃ back into steps 1–3 . Within window 2, the shared RNN weights still receive gradients from steps 4–6. In real training, weight updates between windows may change the later numerical values; the fixed weight here only makes the forward carry easy to follow. Benefits and tradeoffs Lower memory per update: A conventional nonoverlapping TBPTT implementation retains an autograd history of about K steps instead of all T steps, where K is much smaller than T. Manageable updates: Training can update parameters regularly rather than constructing one huge backward graph. Processing an entire long sequence still requires work across all its time steps. Limited long range credit assignment: Distant information may influence later hidden state values, but losses cannot directly push gradients across a detached boundary. Very short windows may miss important long term learning signals. Window size choice: Larger K gives longer gradient reach at greater memory and computation cost per update; smaller K is cheaper but makes a stronger approximation. Practical note: The example uses one update every K time steps with nonoverlapping windows. More advanced TBPTT schedules can choose the update interval and backward horizon separately. Hidden states should also be reset at genuine sequence boundaries when unrelated sequences must not share context. Key takeaway: TBPTT preserves the forward flow of hidden state information across a long sequence, but limits the backward flow of gradients to a practical window. It is a computational tradeoff, not a change to the RNN architecture. Related: “How does backpropagation through time (BPTT) train recurrent neural networks?” covers full sequence gradient propagation. “What is a recurrent neural network (RNN)?” introduces hidden states. Research: Williams and Peng (1990), An Efficient Gradient Based Algorithm for On Line Training of Recurrent Network Trajectories ; Dive into Deep Learning — Backpropagation Through Time. Implementation references: Lightning TBPTT guide; PyTorch autograd and detach.

How do the input, forget, and output gates work in an LSTM?

An LSTM (Long Short Term Memory) uses three learned gates—forget, input, and output—to control which information it retains, writes, and exposes at each time step. These gates make the recurrent memory update selective rather than automatically overwriting the previous state. What does each gate do? An LSTM receives the current input xₜ and the previous hidden state hₜ₋₁. Learned weights and biases produce gate values through the sigmoid function, so each gate value is between 0 and 1 (per memory dimension). These are soft, differentiable controls , not necessarily hard on/off switches. 1. Forget gate (fₜ) — keep or discard old memory: Multiplies the previous cell state cₜ₋₁. A value near 1 retains most of that component; near 0 removes most of it. fₜ = σ(W f[xₜ, hₜ₋₁] + b f). 2. Input gate (iₜ) — decide how much new information to write: Scales a candidate memory update gₜ, which is usually created with tanh and lies between −1 and 1. iₜ = σ(W i[xₜ, hₜ₋₁] + b i) and gₜ = tanh(W g[xₜ, hₜ₋₁] + b g). 3. Output gate (oₜ) — decide how much updated memory to expose: Controls the hidden state output after the cell state has been updated: oₜ = σ(W o[xₜ, hₜ₋₁] + b o). The output gate does not directly erase the cell state. Here W and b represent trainable weights and biases; [xₜ, hₜ₋₁] means concatenating the current input and the previous hidden state. σ denotes sigmoid. Specific implementations may use separate input and recurrent weight matrices for the same operations. How do the gates update memory? Cell state update: cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ gₜ Hidden state output: hₜ = oₜ ⊙ tanh(cₜ) The symbol ⊙ means element wise multiplication . The forget and input gates jointly determine the new cell state ; the output gate determines the exposed hidden state from that updated memory. Beginner friendly numerical example Imagine one component of an LSTM's memory. Suppose its previous cell state value is cₜ₋₁ = 0.8 , and the learned gate computations return: Forget gate: fₜ = 0.5 (keep half the old memory component). Input gate: iₜ = 0.75 (write 75% of the candidate update). Candidate update: gₜ = 0.4. Output gate: oₜ = 0.6. Step 1 — retain old information: fₜ × cₜ₋₁ = 0.5 × 0.8 = 0.4. Step 2 — write new information: iₜ × gₜ = 0.75 × 0.4 = 0.3. Step 3 — calculate the updated cell state: cₜ = 0.4 + 0.3 = 0.7 . Step 4 — calculate the exposed hidden state: hₜ = 0.6 × tanh(0.7) ≈ 0.6 × 0.6044 ≈ 0.3626. Interpretation: The updated memory value is 0.7 , but the hidden state output is only about 0.3626 because the output gate regulates what is exposed. The gate values are illustrative; a trained LSTM learns them from the data and preceding hidden state. Why are gates useful? Selective retention: The forget gate can keep relevant context while reducing outdated information. Selective updates: The input gate can prevent every new input from overwriting useful memory. Selective exposure: The output gate can limit what the current hidden state output reveals while the cell state retains other information. Trainable behavior: Gates adapt through learning and can behave differently across time steps and memory dimensions. Important distinction: The cell state (cₜ) is the updated memory carried forward. The hidden state (hₜ) is the gated output derived from that memory. This is enough to understand the gates; the next question explains cell state versus hidden state in greater depth. Key takeaway: Forget = retain old information; input = write candidate information; output = expose updated information. All three are learned continuous controls. Related: “What is an LSTM network?” introduces the architecture. “How does backpropagation through time (BPTT) train recurrent neural networks?” covers recurrent training. The following question focuses on the distinction between cell and hidden states. Research: Hochreiter & Schmidhuber (1997), Long Short Term Memory ; Gers, Schmidhuber & Cummins (2000), Learning to Forget: Continual Prediction with LSTM . Implementation: PyTorch torch.nn.LSTM equations.

What is the difference between an LSTM hidden state and cell state?

In an LSTM (Long Short Term Memory), the cell state (cₜ) is the internal memory carried across time steps, while the hidden state (hₜ) is the current, exposed representation of that memory. They are separate vectors: the LSTM can preserve information in its cell state without exposing all of it in its hidden output. What does each state do? Cell state (cₜ) — carry memory: Holds an internal representation that the LSTM can retain or modify as it processes a sequence. Its update has a pathway from the previous cell state cₜ₋₁, helping useful information survive across multiple time steps. It is not normally the feature vector sent directly to the prediction layer. Hidden state (hₜ) — provide the current output: Represents the part of the updated memory made available at the current step. It can be used by the next recurrent step and passed to another layer or a prediction head. It is sometimes called the LSTM's output state . Both states are learned numerical representations, not human readable facts stored in separate compartments. How are the two related? For a standard LSTM, the hidden state is derived from the updated cell state: hₜ = oₜ ⊙ tanh(cₜ) Here tanh squashes each cell state component into the range −1 to 1 , oₜ is the learned output control value between 0 and 1 for each component, and ⊙ means element wise multiplication. The cell state itself need not be confined to −1 to 1 . How the gates calculate and update memory is covered separately in “How do the input, forget, and output gates work in an LSTM?” Beginner friendly numerical example Consider one component of an LSTM's state over two successive time steps. Assume these illustrative values: Time step t: Updated cell state: cₜ = 1.2 Output control value: oₜ = 0.25 Hidden state: hₜ = 0.25 × tanh(1.2) ≈ 0.25 × 0.8337 ≈ 0.2084 Time step t+1: Updated cell state: cₜ₊₁ = 1.1 Output control value: oₜ₊₁ = 0.80 Hidden state: hₜ₊₁ = 0.80 × tanh(1.1) ≈ 0.80 × 0.8005 ≈ 0.6404 What changed? The stored memory component changed only slightly, from 1.2 to 1.1 , but the exposed hidden state component increased from about 0.2084 to 0.6404 . The model can therefore keep similar internal memory while producing a very different current output. These values are chosen to illustrate the relationship, not computed from a particular trained LSTM. Which state is passed to the next step? At the next time step, the LSTM uses both cₜ and hₜ , along with the new input xₜ₊₁. The cell state supplies the continuing memory; the hidden state influences the next computation and is the usual sequence output available to subsequent layers. For example, PyTorch's torch.nn.LSTM returns the sequence output (hidden state features at each time step) and final states (h n, c n) . The cell state is returned as recurrent state, not as the standard per step output feature. If a classifier follows the LSTM, it commonly turns a selected hidden state into prediction scores—hₜ itself is not automatically the final class prediction. Key takeaway: Cell state = information the LSTM carries forward. Hidden state = information it exposes now. The two work together, but they are not interchangeable. Related: “How do the input, forget, and output gates work in an LSTM?” explains the gate mechanics. The next question introduces the different state update design of a GRU. References: PyTorch LSTM documentation and Dive into Deep Learning — Long Short Term Memory.

How do the update and reset gates work in a GRU?

A GRU (Gated Recurrent Unit) uses two learned gates to control how its hidden state changes as a sequence is processed. The reset gate decides how much previous information contributes to a candidate hidden state. The update gate decides how much of the old hidden state to retain versus replace with that candidate. What does each gate do? At step t, the GRU receives the current input xₜ and previous hidden state hₜ₋₁. Both gates use a sigmoid activation, giving continuous values between 0 and 1 for each hidden state component. Reset gate (rₜ): A value near 0 makes the proposed new content depend less on the previous hidden state; near 1 lets the candidate use more of that history. Resetting the candidate's access to history does not directly erase stored memory. Update gate (zₜ): Using the convention below, a value near 1 mostly retains the previous hidden state; near 0 mostly takes the new candidate. One gate controls both retention and replacement. GRU equations rₜ = σ(Wᵣxₜ + Uᵣhₜ₋₁ + bᵣ) — reset gate zₜ = σ(W𝓏xₜ + U𝓏hₜ₋₁ + b𝓏) — update gate nₜ = tanh(Wₙxₜ + Uₙ(rₜ ⊙ hₜ₋₁) + bₙ) — candidate hidden state hₜ = zₜ ⊙ hₜ₋₁ + (1 − zₜ) ⊙ nₜ — updated hidden state Here σ is sigmoid; tanh squashes the candidate into −1 to 1 ; W, U, and b are trainable weights and biases; and ⊙ means element wise multiplication. All equations act on vectors in a real GRU, but one scalar component is enough for a numerical example. The reset gate acts first , while forming the candidate; the update gate acts second , mixing candidate and prior state. Beginner friendly numerical example Suppose one hidden state component is hₜ₋₁ = 0.8 . For illustration, learned gate calculations yield pre sigmoid scores of −1.0986 for reset and +1.0986 for update. Assume the candidate's current input contribution is 0.4 , its recurrent weight is 0.5 , and its bias is zero. 1. Reset gate: rₜ = σ(−1.0986) ≈ 0.25. 2. Update gate: zₜ = σ(+1.0986) ≈ 0.75. 3. Candidate: The reset scaled prior state is 0.25 × 0.8 = 0.2. Therefore, nₜ = tanh(0.4 + 0.5 × 0.2) = tanh(0.5) ≈ 0.4621. 4. Final state: hₜ = 0.75 × 0.8 + (1 − 0.75) × 0.4621 = 0.6 + 0.1155 ≈ 0.7155. Interpretation: The reset gate allowed only 25% of the old hidden state component into the candidate calculation . The update gate then retained 75% of the old hidden state in the final result , mixing in 25% candidate. So the model can propose new content using little history while still retaining most of its previous representation. All sample values are illustrative, not measured from a trained model. How is a GRU different from an LSTM? A GRU has one recurrent hidden state , which it uses for both memory and output, rather than the separate hidden state and cell state of an LSTM. Its reset and update gates provide a compact mechanism for selective candidate formation and retention. For the dedicated LSTM details, see “How do the input, forget, and output gates work in an LSTM?” (Question 243) and “What is the difference between an LSTM hidden state and cell state?” (Question 244). Implementation detail: PyTorch computes the GRU candidate with the reset gate applied after the recurrent linear transformation, whereas the original GRU equations above apply it before. The one dimensional example with zero candidate bias yields the same number either way, but multidimensional implementations may differ. Key takeaway: Reset controls what past information influences the proposed update; update determines how much old information survives in the final state. Research: Cho et al. (2014), Learning Phrase Representations using RNN Encoder Decoder for Statistical Machine Translation . Implementation/reference: PyTorch torch.nn.GRU and Dive into Deep Learning — GRU.

What is exposure bias in sequence-to-sequence neural networks?

Exposure bias is the training–inference mismatch that can arise in an autoregressive sequence to sequence (seq2seq) model when it learns to predict the next token from correct previous tokens during training, but must use its own earlier predictions at inference time. If an early prediction is wrong, later predictions may receive an unfamiliar or misleading prefix, making recovery harder. Where does the mismatch come from? During teacher forced training: The decoder predicts the next target token while conditioning on the reference (ground truth) prefix , even if its own previous prediction was wrong. During autoregressive inference: The correct target is unavailable. The decoder must feed its generated prefix back into subsequent predictions; that prefix may already contain mistakes. The decoder therefore practices mostly on reference token histories but is evaluated on histories it generates itself. The dedicated “What is teacher forcing in neural networks?” article explains the training technique; the issue here is what changes when the model runs without the teacher. Beginner friendly example: one mistake changes what comes next Imagine a translation model whose correct English target is “I love tea” . Teacher forced training 1. Given <start , predict I. 2. Given the correct prefix I, predict love. 3. Given the correct prefix I love, predict tea. Even if the model accidentally predicts “like” at step 2, step 3 is still trained using the correct word “love” . Autoregressive inference 1. Given <start , the model generates I. 2. Given I, it mistakenly generates like . 3. It now sees I like (not I love) and might generate coffee , producing “I like coffee” rather than “I love tea”. For a toy numerical illustration, suppose the model assigns 85% probability to tea after the correct prefix I love, but only 10% after the generated prefix I like. The numbers are hypothetical: they show how a changed prefix can change the next token distribution, not a guaranteed outcome. Why does exposure bias matter? Errors may compound: A mistaken token changes the context for the next step; another mistake can then change the following context. Training metrics can look better than generated sequences: A low teacher forced next token loss does not ensure reliable complete sequence generation. Longer outputs offer more opportunities for drift: For translation, summarization, or other multi step generation, the model may gradually move away from the intended sequence. Important nuance: Exposure bias is a potential source of sequence generation errors, not proof that every mistake will cascade or that every autoregressive model must fail. Models can sometimes recover, and the extent to which this mismatch hurts generation depends on generalization and the task. How should it be assessed? Alongside teacher forced token level evaluation, run the decoder autoregressively on held out inputs , inspect full generated sequences, and check whether it can recover after an incorrect token. Techniques that change training time exposure to generated prefixes are studied separately; the next question covers scheduled sampling . Key takeaway: Teacher forcing trains on correct history; inference runs on the model's own history. Exposure bias is the gap between those conditions and the risk of error accumulation. Research: Schmidt (2019), Generalization in Generation: A closer look at Exposure Bias and Nicolai & Silfverberg (2020), Noise Isn’t Always Negative: Countering Exposure Bias in Sequence to Sequence Inflection Models .

What is scheduled sampling in sequence model training?

Scheduled sampling is a training strategy for autoregressive sequence models that gradually switches the decoder's previous token input from the correct (ground truth) token to a token generated by the model itself. Early in training, the model gets more guidance; later, it practices continuing from its own predictions. How does scheduled sampling work? At each decoder step in training: 1. Predict a token using the decoder's current state. 2. Choose the next input: With probability p truth, feed the correct previous target token to the next step. With probability 1 − p truth, feed a model generated token instead. 3. Calculate training loss against the correct target token , whichever input was selected. Sampling changes the decoder's input history , not the ground truth label. 4. Gradually lower p truth across training iterations or epochs. This is the schedule . The choice is typically randomized separately at each decoder step. If p truth = 1, training uses ground truth previous tokens throughout (ordinary teacher forcing). If p truth = 0, subsequent steps use model generated tokens. The first decoder input can still be a start of sequence token. Beginner friendly numerical example Suppose a decoder is learning the target sequence I am learning . After the prefix I, its prediction is mistakenly is when the correct token is am . For the following step, scheduled sampling chooses which token the decoder receives: Correct previous token: am Model generated previous token: is Correct next target for the loss: learning (unchanged in both cases) Imagine a simple schedule for the probability p truth: Training start: p truth = 1.00 → ground truth on 100% of such choices. Early training: p truth = 0.80 → about 80% ground truth, 20% model generated. Mid training: p truth = 0.50 → about 50% each. Late training: p truth = 0.20 → about 20% ground truth, 80% model generated. These percentages are probabilities , not guaranteed proportions in a short batch. For one illustrative random draw of u = 0.60 , choose ground truth when u < p truth, otherwise choose the generated token: At 1.00 or 0.80 , the next input is am (ground truth). At 0.50 or 0.20 , the next input is is (model generated). So as training progresses, the decoder increasingly sees its own possible mistakes while the desired prediction remains learning . How is the schedule chosen? Linear decay: Reduce p truth by a fixed amount over training. Exponential or inverse sigmoid decay: Reduce it according to a smooth, nonlinear curve. Practical choice: Start near 1, decrease gradually, and tune the endpoint and speed using validation results from actual generated sequences. A schedule need not reach 0. Benefits and limitations Potential benefit: The decoder practices with generated histories instead of relying only on correct previous tokens; some sequence tasks improved in the original experiments. Tradeoff: Incorrect generated tokens can make training inputs harder or noisier, especially if introduced too early. Important theoretical limitation: Scheduled sampling is not guaranteed to improve generation . Its conventional learning objective can be statistically inconsistent ; an early critique showed that the optimum can differ from the true sequence distribution. Evaluate complete autoregressive outputs, not only teacher forced loss. The reason training and inference inputs can differ is addressed in “What is exposure bias in sequence to sequence neural networks?” (Question 246). This question is about the specific training procedure and probability schedule . Key takeaway: Scheduled sampling gradually replaces ground truth decoder inputs with model generated inputs during training, while still supervising predictions against correct targets. Research: Bengio et al. (2015), Scheduled Sampling for Sequence Prediction with Recurrent Neural Networks ; Huszár (2015), How (not) to Train your Generative Model: Scheduled Sampling, Likelihood, Adversary? .

What is a Temporal Convolutional Network (TCN)?

A Temporal Convolutional Network (TCN) is a neural network that learns patterns across an ordered sequence using convolutions along the time axis, instead of updating a recurrent hidden state one step at a time. A common TCN design combines causal convolutions , dilations , and residual connections so it can use a long history while processing positions in parallel. How does a TCN process a sequence? Imagine hourly electricity readings x₁, x₂, …, xₜ, with a model predicting demand at each time. Causal convolution — no future leakage: To produce the output at time t, a causal filter can read xₜ and earlier readings, not xₜ₊₁ . Left side padding preserves output length without revealing future readings. Dilated convolution — look farther back: Instead of always reading adjacent time steps, a filter samples positions separated by its dilation d. With kernel size 2, it reads xₜ and xₜ₋d. Stacked layers — enlarge historical context: Successive layers combine nearby and more distant information. Dilation often increases 1, 2, 4, 8, …. Residual connections — train deeper networks: A residual block adds its input to a learned transformation, output = F(input) + input, sometimes projecting the input with a 1×1 convolution if channel counts differ. This gives gradients a more direct path through the stack. Causality is essential for online forecasting, where future values are unknown. Not every model called a TCN must be causal ; offline sequence labeling designs can use future context when the whole sequence is available. Beginner friendly numerical example Take eight hourly measurements: x₁…x₈ = [1, 2, 3, 4, 5, 6, 7, 8] Use three stacked causal 1D convolution layers , each with a two position kernel [1, 1] , zero bias, no activation, and dilations 1, 2, 4 . This stripped down example shows which time steps contribute; a practical TCN also learns weights and usually adds nonlinearities and residual blocks. 1. Layer 1, dilation 1: At time 8, read x₈ and x₇. Thus y¹₈ = 8 + 7 = 15. At time 6, y¹₆ = 6 + 5 = 11. 2. Layer 2, dilation 2: At time 8, read layer 1 positions 8 and 6: y²₈ = y¹₈ + y¹₆ = 15 + 11 = 26. Similarly, y²₄ = y¹₄ + y¹₂ = (4 + 3) + (2 + 1) = 10. 3. Layer 3, dilation 4: At time 8, read layer 2 positions 8 and 4: y³₈ = y²₈ + y²₄ = 26 + 10 = 36 . The final value 36 is the sum of x₁ through x₈, because this chosen toy kernel gives each of the eight historical positions a path to the output. No x₉ is used . What is the receptive field? The receptive field is how many time positions can influence an output. For L stride 1 causal convolution layers with kernel size k and dilations d₁…dL: R = 1 + (k − 1) × (d₁ + … + dL) For this one convolution per layer example: R = 1 + (2 − 1) × (1 + 2 + 4) = 8 time steps. This is the theoretical maximum span for an output away from the sequence boundary. Important architecture detail: Many standard residual TCN blocks contain two convolutions at the same dilation. Three such blocks with k=2, dilations 1,2,4, and stride 1 would instead have R = 1 + 2 × (1+2+4) = 15 steps. Always count the actual convolutions rather than applying a single receptive field formula to every TCN. For the general ideas behind dilations and receptive fields, see “What is a dilated convolution and when is it useful?” and “What is the receptive field of a neuron in a CNN?” ; the key TCN specific point is how they act across time . When is a TCN useful? Forecasting and sensor data: Predict electricity demand, machine readings, or other time dependent signals using past observations. Audio and event streams: Learn local and long range patterns over sampled sequences. Action or sequence labeling: Classify events at successive video or signal positions. Benefits: Convolution positions within a layer can be calculated in parallel during training; dilations expand the receptive field without requiring proportionally larger kernels; residual paths help train deeper stacks. Limitations: A fixed architecture sees only its designed receptive field; very long dependencies may require more layers or a different dilation plan. Padding, dilation choice, and causal versus noncausal setup must match the task. A TCN is not universally more accurate or efficient than recurrent or attention based alternatives. Key takeaway: A TCN uses time axis convolutions to model sequences: causality prevents future leakage, dilation extends the available past, and residual connections support deeper temporal stacks. Research: Bai, Kolter & Koltun (2018), An Empirical Evaluation of Generic Convolutional and Recurrent Networks for Sequence Modeling ; Lea et al. (2016), Temporal Convolutional Networks for Action Segmentation and Detection .

What is stochastic depth (DropPath) in deep networks?

Stochastic depth (often called DropPath in modern residual networks) is a training time regularization technique that randomly skips a residual branch while preserving its shortcut connection. A deep model therefore trains with different effective paths, but normally uses all its blocks at inference. How does DropPath work? A residual block combines an input x with a learned transformation F(x): output = x + F(x) With DropPath, the model randomly decides whether the whole residual branch contributes on a training pass: Survives: Keep the residual transformation F(x). Dropped: Set that branch's contribution to zero, leaving output = x. The shortcut still carries the input and gradients. During inference: Stop randomly dropping branches and use the complete network. In common modern implementations, one random decision is made per sample and residual branch , shared across the branch's channels and spatial positions. Some variants make the decision for the whole mini batch. The survival probability may vary by network depth, with later blocks often assigned higher drop rates. Beginner friendly numerical example Suppose one residual block has: Input: x = 2 Residual branch output: F(x) = 4 Survival probability: p = 0.8 (80% chance the branch stays) Drop probability: 1 − p = 0.2 (20% chance it is skipped) A common inverted scaling implementation uses: y train = x + (b / p) × F(x), where b is 1 with probability p , otherwise 0 . Case 1 — branch survives (b = 1): y train = 2 + (1 / 0.8) × 4 = 2 + 5 = 7 Case 2 — branch is dropped (b = 0): y train = 2 + 0 = 2 Expected training output: E[y train] = 0.8 × 7 + 0.2 × 2 = 6 At inference: DropPath is disabled, so the block uses y eval = 2 + 4 = 6 . Inverted scaling during training makes the expected branch contribution equal to its full inference time contribution. It does not mean every stochastic training output is 6. Implementation nuance: The original stochastic depth paper used an alternative convention: it skipped unscaled branches during training and scaled surviving branches by their survival probabilities at test time. Modern libraries such as timm and torchvision commonly divide a surviving branch by p during training , then leave the entire branch active without scaling at inference. Use the convention implemented by your framework rather than mixing them. Why can this help very deep networks? Shorter active paths during training: On some passes, deep residual transformations are bypassed, reducing the number of transformations on the active path. Regularization: Varying which residual branches contribute can reduce dependence on any particular deep path and may improve generalization. Preserved shortcut: The identity path keeps carrying information when a residual branch is skipped, helping gradient flow. For example, a model with 100 eligible residual branches might keep about 80 active on average if each independently survives with probability 0.8. That is an expected count , not a guarantee for an individual sample. Limitations and practical considerations Too much dropping can cause underfitting: If important branches rarely train, accuracy may suffer. Tune probabilities: Suitable drop rates depend on depth, architecture, dataset size, and training duration; stronger dropping is not always better. Use genuine residual paths: Skipping a branch is simplest when its shortcut is a compatible identity or projection; do not remove the shortcut itself. No guarantee of faster training: Actual wall clock savings depend on the implementation, batching, and whether skipped computations are truly omitted. Key takeaway: DropPath randomly removes residual transformations during training , while keeping shortcuts intact; full depth inference is normally restored. The next question compares stochastic depth with ordinary dropout. For background, see “What are residual connections in neural networks?” and “What is dropout in deep learning?” ; their mechanisms are not repeated here. Research: Huang et al. (2016), Deep Networks with Stochastic Depth . Implementation: Torchvision stochastic depth and timm DropPath.

How does stochastic depth differ from dropout?

Dropout and stochastic depth (DropPath) both regularize neural networks by randomly removing contributions during training, but they operate at different levels: ordinary dropout masks individual activation values, while DropPath masks an entire learned residual branch. The residual shortcut remains available even if its branch is dropped. Key differences What gets removed? Ordinary dropout sets selected activation elements to zero. DropPath removes all components of one residual branch output together for a given training sample. Mask granularity: Standard dropout independently samples masks for individual elements. Modern DropPath typically uses one keep/drop decision per sample per residual branch , broadcast across the branch's channels and positions. Some implementations support a batch wide mask. Where used? Dropout is broadly applicable to layers in different networks. DropPath requires a suitable bypass, so it is common in deep residual CNNs and Vision Transformers. What changes during training? Dropout changes which individual features are available; DropPath changes which residual transformations contribute, producing effectively shorter paths. What happens at inference? Both are normally turned off: all activation elements stay available and every residual branch contributes. How is scaling handled? Common inverted implementations divide surviving elements or branches by the keep probability during training . Then evaluation uses the unmasked values or branches without extra scaling. The original stochastic depth paper used a different convention, covered in “What is stochastic depth (DropPath) in deep networks?” . Beginner friendly numerical example Use a 50% drop probability for each method (keep probability q = 0.5). These are chosen random outcomes: a 50% rate does not force exactly half the elements to disappear in every small example. 1. Dropout — remove selected values Suppose an activation vector is a = [2, 4]. The sampled element wise keep mask is m = [1, 0]. Training: a train = (m / q) ⊙ a = [4, 0]. The first element stays, scaled by 1/0.5 = 2; the second becomes zero. Inference: a eval = [2, 4]. Nothing is randomly masked. 2. Stochastic depth — remove the whole residual contribution Consider a residual block receiving x = [2, 4], whose learned branch produces F(x) = [1, 3]. DropPath makes one binary choice b for the entire branch: Branch dropped (b = 0): y train = x + (b / q)F(x) = [2, 4]. The shortcut still passes x. Branch kept (b = 1): y train = [2, 4] + 2[1, 3] = [4, 10]. Both branch components participate together. Inference: y eval = x + F(x) = [3, 7], with the full branch active. The two equally likely DropPath outcomes have expected output 0.5[2, 4] + 0.5[4, 10] = [3, 7]. Inverted scaling preserves the expected contribution, not the exact output of every training pass. Notice: Dropout removed a single activation element ; DropPath removed the entire residual contribution but kept the shortcut. Both can have the same numerical drop probability while applying very different masks. When should you choose each? Dropout: Useful when feature level randomness helps prevent overreliance on particular activations; it is common in fully connected and other network layers. DropPath: Useful in sufficiently deep residual networks where bypassing whole transformations during training can support regularization and optimization. Both together? Possible when an architecture supports them, but not automatically better. Excessive dropping of either kind can cause underfitting; validate rates and performance rather than assuming faster training or higher accuracy. Important nuance: This compares DropPath with ordinary element wise dropout . Spatial dropout removes entire channels; it still does not necessarily drop a complete residual branch . See the existing “What is spatial dropout?” article for that separate method. For each mechanism in depth, see “What is dropout in deep learning?” , “What happens to dropout during inference?” , and “What is stochastic depth (DropPath) in deep networks?” . Key takeaway: Dropout masks activation elements; stochastic depth masks residual paths. Both are usually training only regularizers, but their masks operate at different structural scales. References: Huang et al. (2016), Deep Networks with Stochastic Depth ; PyTorch nn.Dropout documentation; Torchvision stochastic depth documentation.