What is truncated backpropagation through time?

Truncated backpropagation through time (TBPTT) trains a recurrent neural network (RNN) on a long sequence by limiting each backward pass to a shorter window of time steps. Crucially, it can carry the hidden-state value forward across windows while stopping gradients from flowing back through earlier windows.

Why truncate the backward pass?

Processing a very long sequence with one connected training graph requires keeping many intermediate activations for gradient calculation. TBPTT controls this cost by choosing a window length `K` and handling the sequence in manageable pieces.

How does TBPTT work?

1. Split the sequence: Choose a truncation length K, such as 3, 20, or 50 time steps, based on the task and memory budget.
2. Run one window forward: Feed its inputs through the RNN, starting from the hidden state produced by the preceding window (or an initial state for the first window).
3. Compute loss and backpropagate: Use losses in the current window to calculate gradients through at most that window's recurrent computations. Update the same model parameters with the optimizer.
4. Carry the hidden-state value forward: Reuse the last hidden-state value when processing the next window; this preserves available sequence context.
5. Detach its gradient history: Break the automatic-differentiation link to the previous window before the next one begins. In PyTorch, a tensor operation such as `hidden = hidden.detach()` expresses this idea. Detach is not the same as resetting `hidden` to zero.

Beginner-friendly example: 12 steps, windows of 3

Suppose an RNN processes 12 consecutive words and we choose `K = 3`:

• Window 1: Steps 1–3 → compute loss and gradients over those 3 steps → carry hidden state h₃.
• Window 2: Steps 4–6 → start from the value of h₃, but detach its old computation graph → train over steps 4–6 → carry h₆.
• Window 3: Steps 7–9 → start from detached h₆ → train over steps 7–9 → carry h₉.
• Window 4: Steps 10–12 → start from detached h₉ → train over steps 10–12.

That is `12 ÷ 3 = 4` windows. At the boundary between windows 1 and 2, the model does not forget the numerical value of `h₃`; it merely stops backpropagating from later losses into the computations that created h₃.

Tiny numeric illustration: To see the difference, temporarily hold a simple RNN's weight fixed and use hₜ = 0.5 × hₜ₋₁ + xₜ, with h₀ = 0, x₁ = 1, and all later inputs 0.

• Window 1: h₁ = 1, h₂ = 0.5, h₃ = 0.25.
• Window 2: Start from the carried `h₃ = 0.25`, giving h₄ = 0.125, h₅ = 0.0625, h₆ = 0.03125.

Even though the step-6 prediction still uses information passed forward from step 1, the step-6 loss cannot send a gradient through detached `h₃` back into steps 1–3. Within window 2, the shared RNN weights still receive gradients from steps 4–6. In real training, weight updates between windows may change the later numerical values; the fixed weight here only makes the forward carry easy to follow.

Benefits and tradeoffs

• Lower memory per update: A conventional nonoverlapping TBPTT implementation retains an autograd history of about K steps instead of all T steps, where K is much smaller than T.
• Manageable updates: Training can update parameters regularly rather than constructing one huge backward graph. Processing an entire long sequence still requires work across all its time steps.
• Limited long-range credit assignment: Distant information may influence later hidden-state values, but losses cannot directly push gradients across a detached boundary. Very short windows may miss important long-term learning signals.
• Window-size choice: Larger K gives longer gradient reach at greater memory and computation cost per update; smaller K is cheaper but makes a stronger approximation.

Practical note: The example uses one update every K time steps with nonoverlapping windows. More advanced TBPTT schedules can choose the update interval and backward horizon separately. Hidden states should also be reset at genuine sequence boundaries when unrelated sequences must not share context.

Key takeaway: TBPTT preserves the forward flow of hidden-state information across a long sequence, but limits the backward flow of gradients to a practical window. It is a computational tradeoff, not a change to the RNN architecture.

Related: “How does backpropagation through time (BPTT) train recurrent neural networks?” covers full-sequence gradient propagation. “What is a recurrent neural network (RNN)?” introduces hidden states.

Research: Williams and Peng (1990), *An Efficient Gradient-Based Algorithm for On-Line Training of Recurrent Network Trajectories*; Dive into Deep Learning — Backpropagation Through Time. Implementation references: Lightning TBPTT guide; PyTorch autograd and detach.