How do the input, forget, and output gates work in an LSTM?

An LSTM (Long Short-Term Memory) uses three learned gates—forget, input, and output—to control which information it retains, writes, and exposes at each time step. These gates make the recurrent memory update selective rather than automatically overwriting the previous state.

What does each gate do?

An LSTM receives the current input xₜ and the previous hidden state hₜ₋₁. Learned weights and biases produce gate values through the sigmoid function, so each gate value is between 0 and 1 (per memory dimension). These are soft, differentiable controls, not necessarily hard on/off switches.

1. Forget gate (`fₜ`) — keep or discard old memory: Multiplies the previous cell state cₜ₋₁. A value near 1 retains most of that component; near 0 removes most of it. fₜ = σ(W_f[xₜ, hₜ₋₁] + b_f).
2. Input gate (`iₜ`) — decide how much new information to write: Scales a candidate memory update gₜ, which is usually created with tanh and lies between −1 and 1. iₜ = σ(W_i[xₜ, hₜ₋₁] + b_i) and gₜ = tanh(W_g[xₜ, hₜ₋₁] + b_g).
3. Output gate (`oₜ`) — decide how much updated memory to expose: Controls the hidden-state output after the cell state has been updated: oₜ = σ(W_o[xₜ, hₜ₋₁] + b_o). The output gate does not directly erase the cell state.

Here W and b represent trainable weights and biases; [xₜ, hₜ₋₁] means concatenating the current input and the previous hidden state. σ denotes sigmoid. Specific implementations may use separate input and recurrent weight matrices for the same operations.

How do the gates update memory?

Cell-state update: cₜ = fₜ ⊙ cₜ₋₁ + iₜ ⊙ gₜ

Hidden-state output: hₜ = oₜ ⊙ tanh(cₜ)

The symbol ⊙ means element-wise multiplication. The forget and input gates jointly determine the new cell state; the output gate determines the exposed hidden state from that updated memory.

Beginner-friendly numerical example

Imagine one component of an LSTM's memory. Suppose its previous cell-state value is `cₜ₋₁ = 0.8`, and the learned gate computations return:

• Forget gate: fₜ = 0.5 (keep half the old memory component).
• Input gate: iₜ = 0.75 (write 75% of the candidate update).
• Candidate update: gₜ = 0.4.
• Output gate: oₜ = 0.6.

Step 1 — retain old information: fₜ × cₜ₋₁ = 0.5 × 0.8 = 0.4.

Step 2 — write new information: iₜ × gₜ = 0.75 × 0.4 = 0.3.

Step 3 — calculate the updated cell state: `cₜ = 0.4 + 0.3 = 0.7`.

Step 4 — calculate the exposed hidden state: hₜ = 0.6 × tanh(0.7) ≈ 0.6 × 0.6044 ≈ 0.3626.

Interpretation: The updated memory value is `0.7`, but the hidden-state output is only about `0.3626` because the output gate regulates what is exposed. The gate values are illustrative; a trained LSTM learns them from the data and preceding hidden state.

Why are gates useful?

• Selective retention: The forget gate can keep relevant context while reducing outdated information.
• Selective updates: The input gate can prevent every new input from overwriting useful memory.
• Selective exposure: The output gate can limit what the current hidden-state output reveals while the cell state retains other information.
• Trainable behavior: Gates adapt through learning and can behave differently across time steps and memory dimensions.

Important distinction: The cell state (`cₜ`) is the updated memory carried forward. The hidden state (`hₜ`) is the gated output derived from that memory. This is enough to understand the gates; the next question explains cell state versus hidden state in greater depth.

Key takeaway: Forget = retain old information; input = write candidate information; output = expose updated information. All three are learned continuous controls.

Related: “What is an LSTM network?” introduces the architecture. “How does backpropagation through time (BPTT) train recurrent neural networks?” covers recurrent training. The following question focuses on the distinction between cell and hidden states.

Research: Hochreiter & Schmidhuber (1997), *Long Short-Term Memory*; Gers, Schmidhuber & Cummins (2000), *Learning to Forget: Continual Prediction with LSTM*. Implementation: PyTorch `torch.nn.LSTM` equations.