What is mixed-precision training in deep learning?

Mixed-precision training uses more than one floating-point precision within the same neural-network training run. It performs suitable operations with lower-precision formats, commonly 16-bit FP16 or BF16, while retaining FP32 for numerically sensitive calculations, parameter updates or optimizer state as appropriate. The goal is faster training and/or lower memory use without unacceptable accuracy or stability loss.

A foundational paper is Micikevicius et al., *Mixed Precision Training* (ICLR 2018). Modern frameworks such as PyTorch provide automatic mixed precision (AMP) so programmers usually do not have to manually cast every layer.

1. Why mix precisions instead of using only FP32 or only 16-bit?

• FP32 means 32 bits = 4 bytes per scalar. It is a well-established training baseline with relatively wide numerical range and useful precision.
• 16-bit formats use 16 bits = 2 bytes per scalar. They can reduce tensor storage and memory bandwidth, and may enable faster matrix multiplications and convolutions on compatible hardware.
• Not every operation tolerates reduced precision equally well. Some reductions, normalization calculations, exponentials, loss functions or accumulations may need a safer or higher-precision computation path.
• Mixed precision chooses the format by role. For example, a convolution might use lower-precision inputs and specialized accelerated kernels, while sensitive reductions or parameter updates use FP32.
• Full half-precision conversion is different from AMP. Simply running model.half() converts parameters too and can create instability or incompatibilities; it is not a substitute for selective autocasting.

Core intuition: Use smaller, faster arithmetic where it is safe, but avoid sacrificing the numerical information that keeps learning reliable.

2. Beginner-friendly memory calculation: One tensor

Take one tensor containing 10,000,000 floating-point values, such as a model parameter tensor or a saved activation collection.

Step 1 — FP32 storage

• 10,000,000 × 4 bytes = 40,000,000 bytes = 40 MB in decimal units.
• Each element uses 32 bits.

Step 2 — A 16-bit representation

• 10,000,000 × 2 bytes = 20,000,000 bytes = 20 MB.
• Each element uses 16 bits.

Step 3 — Compare

• Savings for that tensor: 40 MB − 20 MB = 20 MB.
• Percentage saving: 20/40 × 100% = 50%.
• The tensor contains the same number of elements, but the lower-precision representation offers fewer representable numerical values.

This 50% is a tensor-storage property—not a guaranteed reduction in the overall memory footprint of a training job.

3. Numerical example: Parameters, activations and gradients

Imagine a fictional neural network with:

• 10 million trainable parameters.
• 20 million simultaneously stored activation elements at the illustrative point in training.
• 10 million parameter-gradient elements, one gradient per trainable parameter.
• An Adam-like optimizer maintaining two FP32 state values per parameter (e.g., first and second moments).

We deliberately exclude CUDA workspaces, additional temporary tensors, input/output buffers, data-loader storage, allocator fragmentation, certain buffers and other implementation-specific state. These are illustrative accounting assumptions, not measured GPU profiler results.

Step A — Compute the FP32 baseline

• Model parameters: 10M × 4 bytes = 40 MB.
• Saved activations: 20M × 4 bytes = 80 MB.
• Parameter gradients: 10M × 4 bytes = 40 MB.
• Two FP32 optimizer states: 2 × 10M × 4 bytes = 80 MB.
• Illustrative total: `40+80+40+80 = 240 MB`.

Step B — Calculate the theoretical 16-bit sizes for the three major tensors

If each listed tensor were represented independently using 16-bit storage:

• Parameters: 10M × 2 = 20 MB instead of 40 MB.
• Activations: 20M × 2 = 40 MB instead of 80 MB.
• Parameter gradients: 10M × 2 = 20 MB instead of 40 MB.

These are potential per-tensor sizes, not an assertion that ordinary PyTorch AMP stores all three classes of tensors in 16-bit.

Step C — Consider a common PyTorch autocast setup

Suppose we create a regular FP32 model and FP32 Adam optimizer, enable autocast on appropriate operations, and—*as an illustrative simplification*—half the bytes are needed for all 20 million retained activation elements, while parameters, parameter-gradient buffers and optimizer states remain FP32.

• Persistent FP32 model parameters: 40 MB.
• Saved lower-precision activations: 40 MB.
• FP32 parameter gradients: 40 MB.
• FP32 Adam state: 80 MB.
• Illustrative AMP total: `40+40+40+80 = 200 MB`.
• Savings versus FP32 baseline: 240−200 = 40 MB, or 40/240×100% ≈ 16.7%.

Critical caveat: PyTorch autocast does not promise every saved activation becomes 16-bit. Some operations remain FP32; temporary casts and cached weights can add memory. The 200 MB is a *hypothetical budget*, not a PyTorch guarantee.

Step D — A different master-weight implementation

Some explicitly designed mixed-precision systems keep:

• Working FP16 parameter copy: 20 MB.
• Working FP16 activation tensors: 40 MB.
• FP16 parameter-gradient copy: 20 MB.
• An additional FP32 master parameter copy: 40 MB.
• Two FP32 Adam states: 80 MB.

Total for this specific hypothetical layout: 20+40+20+40+80 = 200 MB. Here the FP32 master weights are a *separate storage copy*. Actual implementations may hold additional gradient buffers, casts, caches and other tensors.

Why calculate both layouts? The first uses FP32 persistent model weights with autocast, while the second uses explicit FP16 working weights plus FP32 master weights. Both are valid styles of mixed precision, but their storage details differ. Do not add a second FP32 master-weight copy to an ordinary FP32-parameter AMP run unless the implementation actually has one.

4. How does automatic mixed precision (AMP) work?

An automatic mixed precision system classifies eligible operations by what precision is useful and safe:

1. Start with model and optimizer. In a typical PyTorch autocast training recipe, the model parameters are created and held in FP32.
2. Enable an autocast region. Operations such as suitable linear layers and convolutions may execute using 16-bit arithmetic or operands.
3. Keep sensitive operations safe. Eligible reductions and other numerically sensitive operations may be selected for FP32 execution; exact eligibility depends on framework, device and operator.
4. Compute a training loss. Operations run at the framework-selected dtypes; do not assume every output or loss is 16-bit.
5. Backpropagate. Compute gradients from that loss, typically outside the autocast context; backward uses dtypes determined by the forward operations.
6. Update parameters through the optimizer. If parameters and optimizer state are FP32, the persistent updates occur in FP32 even though eligible forward operations used lower precision.
7. Repeat while monitoring stability and performance. Compare to a reliable FP32 baseline rather than assuming enabling AMP automatically improves everything.

Autocast is an operation-level casting policy, not a global tensor conversion switch.

5. Why can FP32 accumulation or FP32 master weights help?

There are two separate numerical concerns.

Accumulating many products

• Matrix multiplication and convolution calculate sums of many products.
• Lower-precision operands can increase arithmetic throughput on supported hardware.
• Some specialized kernels accumulate products in FP32 to reduce rounding error before producing their outputs. The exact accumulation type depends on the hardware and selected kernel; it is not guaranteed for every operation.
• Similarly, selected reductions, normalization statistics and related calculations can benefit from FP32.

Updating trainable weights

• If all weights and tiny updates are stored only in low precision, a sufficiently small parameter change may be rounded away.
• An explicit FP32 master-weight copy can preserve smaller accumulated updates while reduced-precision copies are used for forward/backward computation.
• In typical PyTorch AMP with an FP32 model, the FP32 parameters themselves play that higher-precision update role; there is not automatically a separate master copy.
• Whether an additional FP32 copy is used is an implementation choice, not a mandatory feature of every mixed-precision pipeline.

6. FP16, BF16 and numerical stability: The overview

Two common 16-bit types are FP16 and BF16. They occupy the same 2 bytes per element, but differ in representable range and precision:

• FP16: Often useful on supported accelerators, but its limited range can make very small gradients underflow or large values overflow.
• BF16: Has a wider exponent range but coarser precision than FP16; it is often easier to use in deep learning workloads on supported hardware.
• FP32: Still matters for numerically sensitive operations and optimizer state.

Question 294 will give the detailed bit-layout, range and precision comparison between FP16 and BF16, so those numerical-format details are reserved for that lesson.

Gradient loss scaling: FP16 mixed-precision training often uses a framework-provided gradient scaler to reduce underflow problems, and the scaler may skip an optimizer step if nonfinite gradients are detected. BF16 often does not need the same approach. The mathematics and correct handling of gradient loss scaling belong to Question 295; the purpose here is only to identify it as a stability tool.

Important: BF16 support, FP16 support, autocast operator coverage and accelerator performance vary. Check actual hardware and framework capabilities rather than assuming all 16-bit workloads are accelerated.

7. Practical PyTorch training procedure

A representative CUDA FP16 AMP training step using the current PyTorch API is:

PYTHON
import torch

device = "cuda"  # requires a supported CUDA device
model = MyModel().to(device)  # model parameters stay FP32 by default
optimizer = torch.optim.AdamW(model.parameters(), lr=1e-3)
criterion = torch.nn.CrossEntropyLoss()

scaler = torch.amp.GradScaler("cuda")  # FP16 stability helper

for inputs, labels in train_loader:
    inputs = inputs.to(device)
    labels = labels.to(device)

    optimizer.zero_grad(set_to_none=True)

    # Forward and loss: choose dtypes per eligible operation.
    with torch.autocast(device_type="cuda", dtype=torch.float16):
        predictions = model(inputs)
        loss = criterion(predictions, labels)

    # Backward and update happen outside autocast.
    scaler.scale(loss).backward()
    scaler.step(optimizer)
    scaler.update()

What this does and does not mean:

• The framework may autocast selected operations, not every operation or tensor.
• Persistent model parameters remain FP32 in this example; torch.autocast did not call model.half().
• Backward is performed outside the autocast context, as PyTorch recommends.
• GradScaler is a standard FP16-related stability component, not a speedup mechanism by itself. The implementation details are covered in Question 295.
• This sketch assumes that MyModel and train_loader are defined and that a supported GPU is available.
• CPU/BF16 or other accelerator configurations may use different device settings and may not require gradient scaling.

The older torch.cuda.amp.autocast and torch.cuda.amp.GradScaler APIs have been superseded by torch.autocast and torch.amp.GradScaler in the documented modern usage.

8. How does mixed precision improve runtime?

There are three possible sources of benefit:

• Arithmetic throughput: Compatible GPUs may execute lower-precision matrix multiplication or convolution substantially faster using specialized hardware.
• Memory bandwidth: Moving two bytes instead of four for a given tensor can reduce data-transfer cost when that tensor is actually stored or transferred in the lower-precision representation.
• Smaller saved activations: Some operations keep less activation memory, potentially enabling a larger batch size, higher resolution or a model that otherwise would not fit.

But speedup is workload dependent:

• CPU or unsupported hardware may see no benefit or slower performance because casts and implementation overhead outweigh arithmetic gains.
• Small models can be dominated by kernel-launch overhead, data loading or synchronization rather than matrix arithmetic.
• Ops not supported by accelerated low-precision kernels may still run in FP32.
• Memory allocation, temporary casts and optimizer states can dominate even when some activations are halved.
• A faster per-step compute kernel may not translate directly to equal training-epoch or end-to-end speedup.

Correct measurement: Benchmark GPU peak reserved/allocated memory, images or tokens per second, step latency and total training time with the same data, batch, model and validation protocol. Reset and synchronize GPU measurements appropriately.

9. Practical applications

• Vision: CNNs, ViTs, segmentation models and image-generation networks with large matrix/convolution workloads.
• Transformers and language models: High-throughput token training, when the target accelerator and model implementation support suitable lower-precision kernels.
• Large-batch experiments: Savings in activations may allow more samples per GPU without additional hardware.
• Resource-constrained fine-tuning: Some models can train or fine-tune on a given GPU only after reducing per-step memory.
• Distributed training: Reduced-precision communication and selected activation formats may lower bandwidth pressure, but distributed optimizer and sharding conventions determine actual savings.

10. Common mistakes, edge cases and limitations

• Assuming all tensors become FP16: AMP is *mixed* precision. Default FP32 model weights and their gradients can remain FP32.
• Assuming 50% overall memory savings: Half the bytes for a 16-bit tensor does not halve FP32 optimizer states, data inputs, workspaces or temporary tensors.
• Casting the entire model to half: Blindly using model.half() bypasses per-operation dtype selection and may break numerically sensitive layers.
• Backward inside autocast: PyTorch recommends placing forward/loss under autocast and running backward outside.
• Manual casts inside autocast: Excessive .half()/.float() calls can defeat safe operator choices and introduce type mismatches.
• Unsupported kernels/hardware: Some devices, kernels and shapes do not benefit or support the chosen lower-precision format.
• Nonfinite loss or gradients: Monitor NaNs/Infs and unexpectedly skipped steps; compare to FP32. Do not assume every numerical failure is solved by enabling AMP.
• Gradient clipping without the correct scaler handling: FP16 training with a scaler requires gradients to be brought back to their proper scale before clipping. Detailed ordering is a topic for Question 295.
• Wrong performance metric: Compare throughput and peak memory on the same hardware and batch, not theoretical dtype byte counts alone.
• Silent quality degradation: Check training convergence and validation metrics, because reduced precision can affect sensitive models or tasks.
• Inconsistent checkpoints: Save the necessary model and optimizer state; keep dtype/scaler expectations consistent when resuming.

Curriculum boundary: Question 292 concluded latent diffusion and Advanced Generative Neural Networks. Question 293 opens Efficient Training and Model Interpretation, introducing mixed-precision operation selection, memory accounting, benefits and implementation. Question 294 will compare FP16 and BF16 formats in detail; Question 295 will cover gradient loss scaling. Subsequent lessons will address other training efficiency techniques.

Key takeaway: Mixed-precision training accelerates suitable operations and may reduce memory by using 16-bit computation selectively while retaining FP32 where needed. Per-tensor byte savings are straightforward, but actual training memory, speed and stability depend on weights, activations, optimizer states, kernels and hardware.

Research: Micikevicius et al., *Mixed Precision Training* (ICLR 2018); PyTorch, *Automatic Mixed Precision package*; PyTorch, *Automatic Mixed Precision examples*; NVIDIA, *Train With Mixed Precision*.