What is teacher forcing in neural networks?

Teacher forcing is a training technique used in sequence models where the model is given the correct previous output as input when predicting the next step.

It is commonly used in encoder-decoder and autoregressive sequence models.

How teacher forcing works

Suppose a decoder is learning to generate:

I am learning

During training, instead of feeding the model its own previous prediction, teacher forcing gives it the correct previous token.

For example:

Correct token "I" → predict "am"

Correct token "am" → predict "learning"

This helps the model learn the relationship between correct sequence positions more efficiently.

Why is teacher forcing useful?

It can:

• Speed up training.
• Make optimization more stable.
• Prevent early prediction mistakes from immediately affecting every later training step.

What is the main drawback?

During inference, the correct previous output is usually not available.

The model must instead use its own prediction:

Model prediction → next model prediction → next model prediction

This creates a mismatch between training and inference. If the model makes an early mistake, later predictions may become worse because it has not always been trained on its own imperfect outputs.

This issue is often called exposure bias.

Is teacher forcing always used?

No. Some training strategies use teacher forcing only part of the time or gradually reduce how often the correct previous output is provided.

For the broader architecture, see “What is sequence-to-sequence learning?” and “What is an encoder-decoder architecture?”.

In short, teacher forcing trains a sequence model by feeding the correct previous output into the next prediction step, which can speed learning but creates a difference between training and inference.