What is inference in deep learning?

Inference in deep learning is the process of using a trained neural network to make predictions or generate outputs from new, unseen input data.

Training is when the model learns. Inference is when the trained model is actually used.

How inference works

A simplified flow is:

New input → Trained model → Prediction or generated output

For example:

• An image classifier receives a new image and predicts its class.
• A speech model receives audio and produces text.
• A generative model receives a prompt and produces an output.

What happens during inference?

During inference:

• The model's learned weights are typically fixed.
• A forward pass is performed through the network.
• No backpropagation is needed.
• Gradients are usually disabled because the model is not being trained.

This generally makes inference less computationally expensive than training.

Why does inference performance matter?

In production systems, inference often needs to be fast and efficient.

Important factors can include:

• Latency
• Throughput
• Memory usage
• Model size
• Hardware efficiency

Techniques such as quantization, pruning, batching, and specialized accelerators can be used to improve inference efficiency.

Is inference always deterministic?

No. Some models produce the same output for the same input, while generative models may use sampling and therefore produce different outputs across runs.

The next article explains the distinction in more detail in “What is the difference between training and inference?”.

In short, inference is the stage where a trained deep learning model uses what it has learned to make predictions or generate outputs for new inputs.