What is the difference between FP16 and BF16 for neural network training?

FP16 and BF16 are both 16-bit floating-point number formats, but they make different trade-offs between numerical precision and numerical range. FP16 preserves more fine-grained digits near a given value; BF16 represents vastly smaller and larger magnitudes because it uses the same exponent width as FP32. For neural-network training, BF16 often simplifies the handling of very large or tiny activations and gradients, while FP16 can preserve more small relative differences when its narrower range is sufficient.

The important distinction is range versus precision. They both occupy two bytes per stored element; neither is automatically more accurate or faster for every training task.

1. The bit layout: What fits in each number?

Floating-point numbers are represented with a sign bit, an exponent field that controls magnitude, and a fraction (mantissa) field that controls numerical detail.

• FP16, IEEE binary16: 1 sign bit + 5 exponent bits + 10 stored fraction bits = 16 bits. The normal significand also has an implicit leading 1, giving 11 significant binary bits. Exponent bias = 15.
• BF16, bfloat16: 1 sign bit + 8 exponent bits + 7 stored fraction bits = 16 bits. With the implicit leading 1, there are 8 significant binary bits. Exponent bias = 127.
• FP32, IEEE binary32: 1 sign bit + 8 exponent bits + 23 stored fraction bits = 32 bits. With the implicit leading 1, there are 24 significant binary bits. Exponent bias = 127.

Simple way to remember it: BF16 borrows the wide exponent of FP32 but sacrifices more fraction bits to fit in 16 bits. FP16 uses fewer exponent bits and allocates more to the fraction, allowing finer steps within its smaller numerical range.

For an ordinary positive normalized number, the general form is:

value = (1 + fraction) × 2^(stored_exponent − bias).

This is a simplified positive-number formula; negative values use the sign bit, while zeros, subnormal numbers, infinities and NaNs have special encodings.

Concrete bit-pattern illustration for the exact value 1.0:

• FP16: 0 | 01111 | 0000000000 (sign | 5-bit exponent | 10 fraction bits).
• BF16: 0 | 01111111 | 0000000 (sign | 8-bit exponent | 7 fraction bits).
• FP32: 0 | 01111111 | 00000000000000000000000 (sign | 8-bit exponent | 23 fraction bits).

All three encode the number 1.0 exactly, even though their bit layouts differ.

2. Exponent ranges and largest finite values

The unbiased normal exponent range determines which powers of two a format can represent normally.

• FP16: Normal exponents from −14 through +15. Largest finite positive number: 65,504.
• BF16: Normal exponents from −126 through +127. Largest finite positive number: approximately 3.3895 × 10³⁸.
• FP32: Normal exponents from −126 through +127. Largest finite positive number: approximately 3.4028 × 10³⁸.

BF16 and FP32 have the same exponent range, but their precise largest finite numbers differ slightly because their fraction widths differ. Saying their representable ranges are *similar* or they have the *same order-of-magnitude range* is more precise than claiming every extreme endpoint is numerically identical.

Why can this matter? A layer with intermediate values around 100,000 exceeds the largest finite FP16 value and may overflow to inf if actually stored/computed in FP16. The same value is comfortably inside BF16 and FP32's exponent ranges, although BF16 will round it more coarsely.

3. Smallest positive normal and subnormal values

Values near zero can also behave very differently.

• FP16 smallest positive normal: 2⁻¹⁴ ≈ 6.1035 × 10⁻⁵. Smallest positive *subnormal*, when supported: 2⁻²⁴ ≈ 5.9605 × 10⁻⁸.
• BF16 smallest positive normal: 2⁻¹²⁶ ≈ 1.1755 × 10⁻³⁸. Smallest positive subnormal under gradual underflow: 2⁻¹³³ ≈ 9.1835 × 10⁻⁴¹.
• FP32 smallest positive normal: 2⁻¹²⁶ ≈ 1.1755 × 10⁻³⁸. Smallest positive subnormal: 2⁻¹⁴⁹ ≈ 1.4013 × 10⁻⁴⁵.

Normal versus subnormal: Normal numbers include an implicit leading 1 in their significand. Subnormals allow smaller nonzero values by sacrificing relative precision close to zero.

Hardware caveat: The mathematical formats describe subnormals, but some hardware paths or implementations use flush-to-zero, particularly for low-precision subnormal arithmetic. Thus, do not assume every accelerator preserves the theoretical minimum subnormal values. The much wider BF16 normal range is often the more practically relevant advantage.

4. How accurately can values near 1.0 be represented?

A format's machine epsilon is the distance from exactly 1.0 to the next representable number *above* 1.0.

• FP16: 2⁻¹⁰ = 0.0009765625.
• BF16: 2⁻⁷ = 0.0078125.
• FP32: 2⁻²³ ≈ 0.00000011920928955.

That means BF16 values near 1.0 are spaced 8 times farther apart than FP16 values: 0.0078125 / 0.0009765625 = 8.

Important nuance: Epsilon is the spacing *above 1.0*, not a universal absolute error bound for all values or the same quantity as unit roundoff. Relative and absolute rounding errors change with magnitude, and the rounding mode matters.

5. Beginner-friendly numerical example: Three values that expose the trade-off

The following values were independently tested through torch.tensor(value, dtype=...) and conversion back to a Python float on a CPU-backed PyTorch installation. Results assume the usual round-to-nearest conversion behavior and are specific scalar conversion results, not training benchmarks.

Example A — A number only slightly greater than 1: `1.001`

• Input: 1.001.
• FP16 result: 1.0009765625. Its absolute rounding error is |1.001 − 1.0009765625| = 0.0000234375.
• BF16 result: 1.0. Its absolute rounding error is |1.001 − 1.0| = 0.001.
• FP32 result: approximately 1.0010000467, very close to the input.

Reason: BF16's next number above 1.0 is 1.0078125; the input 1.001 is closer to 1.0. FP16's next number above 1.0 is only 0.0009765625 away, so its representation is more precise in this particular example.

Example B — A very small gradient-like value: `0.00000001 = 10⁻⁸`

• FP16 conversion: 0.0, because 10⁻⁸ is too small to round to the smallest positive FP16 subnormal in the chosen conversion.
• BF16 conversion: approximately 1.0011718 × 10⁻⁸, still nonzero.
• FP32 conversion: approximately 9.99999994 × 10⁻⁹, also nonzero.

Reason: 10⁻⁸ is outside FP16's usable nonzero range here, but comfortably inside BF16's normal range. When useful gradient information is lost in a reduced-precision computation, training may stagnate or destabilize.

Example C — A large activation-like value: `100,000`

• FP16 conversion: positive infinity, +inf, because 100,000 exceeds 65,504.
• BF16 conversion: 99,840, a finite rounded approximation.
• FP32 conversion: 100,000, which is exactly representable in FP32.

Reason: BF16 has the exponent range to avoid overflow here but has less detail in the fraction, so the rounded value differs from the original by 160, or 0.16%. FP16's narrower exponent range cannot represent the value at all as a finite number.

Takeaway from all three examples: For values near 1.0, FP16 can be more precise; at very small or large magnitudes, BF16 may retain finite, nonzero information where FP16 cannot.

6. What happens in neural-network training?

FP16 risks

• Underflow: Small gradients or intermediate backward values may round to zero, especially in low-precision arithmetic paths.
• Overflow: Large activations, products, reductions or gradients may become inf and lead to NaN loss or invalid optimizer updates.
• Precision advantage inside range: FP16 has 10 stored fraction bits versus BF16's 7; this can help resolve small differences when values stay within its supported range.

BF16 risks and advantages

• Wide range: BF16 supports FP32-like normal exponents, reducing both underflow and overflow risks for many typical deep-learning values.
• Less fractional detail: Coarse quantization may cause rounding error accumulation or tiny weight updates being lost if weights are directly updated in BF16.
• FP32 still needed selectively: Accumulating some operations in FP32 and maintaining FP32 optimizer state or update precision remains useful. BF16 does not make FP32 universally unnecessary.
• No unconditional stability guarantee: A BF16 model can still produce NaNs, fail to converge, or lose important fine numerical detail.

A useful mental model: BF16 prioritizes dynamic range, FP16 prioritizes significand precision.

7. Why does BF16 often need less gradient scaling?

FP16's limited range creates a real risk that small gradient values produced in FP16 intermediate computations will underflow to zero. A framework may use gradient loss scaling so those intermediates are large enough to retain useful information.

BF16's FP32-like exponent range means many of these gradients remain representable without that extra scaling step. Therefore BF16 training commonly operates without a GradScaler, depending on the framework, model and device.

Important qualifications:

• Gradient scaling primarily addresses underflow, not arbitrary FP16 overflow or every source of instability.
• BF16's coarse mantissa can still cause rounded-away small updates or other errors.
• In ordinary mixed-precision PyTorch, the parameter tensors and their gradient buffers often remain FP32; lower-precision *intermediate* backward arithmetic can nevertheless matter.
• Question 295 will explain the detailed scaling algorithm, inf/NaN detection, unscaling, optimizer ordering and edge cases. Those steps are intentionally not repeated here.

8. Storage and performance: Same bits, different numerical behavior

Both FP16 and BF16 occupy 16 bits = 2 bytes per stored scalar; FP32 occupies 32 bits = 4 bytes.

For 1,000,000 elements stored as one dense tensor:

• FP16 tensor: 1,000,000 × 2 = 2,000,000 bytes = 2 MB.
• BF16 tensor: Also 2 MB.
• FP32 tensor: 1,000,000 × 4 = 4,000,000 bytes = 4 MB.

Do not confuse per-tensor storage with complete training memory. Parameters, activations, parameter gradients, FP32 optimizer states, master weights if present, and temporary buffers determine the actual total. Those wider memory-accounting scenarios were covered in Question 293.

Will BF16 train faster than FP16? Not necessarily.

• Both may use accelerated matrix-multiply or convolution hardware (e.g., Tensor Cores or TPU matrix units), but exact device and kernel support differs.
• BF16 hardware support tends to be available on newer accelerators; some older GPUs accelerate FP16 but do not provide comparable native BF16 arithmetic.
• Performance depends on kernel precision/accumulation, tensor shapes, utilization, data loading, memory bandwidth, launch overhead and operator coverage.
• There is no mathematical rule that says the two formats must run at the same speed just because both use 16 bits.
• Switching from FP32 can benefit either format on appropriate hardware, but every speedup claim should be validated on the intended GPU or CPU.

Numerical hardware note: Some matrix operations accept FP16/BF16 inputs but accumulate products in FP32. This helps accuracy, yet it depends on the selected device and kernel and does not guarantee FP32 precision for every intermediate operation.

9. Practical precision selection

Use the following beginner-friendly decision process.

1. Inspect your hardware: Confirm that the accelerator supports your desired format *natively* and that important kernels benefit. PyTorch provides torch.cuda.is_bf16_supported() for BF16 capability checks; inclusion of emulation and practical performance must be considered separately.
2. Use a trusted FP32 reference: Measure loss curves, evaluation quality, numerical errors and actual runtime with the same data and model.
3. Prefer BF16 as a practical starting candidate on supported modern accelerators when numerical range is a concern. This is often helpful for large transformers or models with varied gradient magnitudes, but must be validated.
4. Consider FP16 when native BF16 support is absent or FP16 performs better, and when your model's values can be kept inside its narrower range with suitable training safeguards.
5. Keep sensitive operations and parameter updates in FP32 where needed, such as selected reductions, softmax or normalization implementations and optimizer state. Let framework autocast policies select safe kernels rather than blindly converting every tensor.
6. Benchmark and compare: Track throughput, peak memory, finite-gradient rates, validation quality and error relative to FP32. A faster format that changes the result unacceptably is not a win.
7. Check model lineage: A checkpoint trained in BF16 may have activations outside FP16's range; forcing it into FP16 can lead to overflow and invalid gradients.

Typical use cases

• Large-language-model and transformer training: BF16 is commonly preferred where supported because of range and ease of numerical management.
• Image classification or detection on older compatible GPUs: FP16 can be an effective accelerated choice if numerical stability is monitored.
• Scientific, sensitive or unusually scaled models: Use careful FP32 reference comparisons; some operations may need FP32 even when most layers use 16-bit arithmetic.
• Inference or fine-tuning: The same range-versus-precision considerations apply, but inference and training have different backward/optimizer memory costs.

10. Small PyTorch demonstration

The following code prints the numerical properties of all three formats and checks the example conversions.

PYTHON
import torch

for dtype in (torch.float16, torch.bfloat16, torch.float32):
    info = torch.finfo(dtype)
    print(dtype, "bits:", info.bits,
          "epsilon:", info.eps,
          "smallest_normal:", info.tiny,
          "largest_finite:", info.max)

    for value in (1.001, 1e-8, 100000.0):
        result = torch.tensor(value, dtype=dtype)
        print(" ", value, "->", float(result))

A CUDA autocast training setup might choose:

PYTHON
# Check that the device and workload support the chosen dtype.
chosen_dtype = torch.bfloat16  # or torch.float16 where appropriate

with torch.autocast(device_type="cuda", dtype=chosen_dtype):
    output = model(inputs)
    loss = loss_fn(output, targets)

# Backward/optimizer update normally run outside autocast.
# FP16 paths may require a GradScaler; BF16 often does not.

What to check after executing this code: Actual format values, real hardware acceleration, FP32 reference accuracy, convergence and numerical warnings—not just a successful cast.

11. Common mistakes and edge cases

• Assuming BF16 always has greater precision: BF16 has greater numerical range but fewer significand bits than FP16.
• Confusing epsilon with smallest positive number: torch.finfo(dtype).eps is the gap above 1, while tiny is the smallest positive *normal*. They are different properties.
• Ignoring subnormal behavior: Theoretical subnormals can be flushed to zero by some hardware. Verify effective behavior rather than relying solely on ideal format tables.
• Assuming BF16's range matches every FP32 finite number exactly: Exponent widths match, but the maximum finite endpoints and rounding granularity differ.
• Believing BF16 never overflows or underflows: It has a large range, not an infinite range.
• Assuming FP16 needs scaling for every model and BF16 never does: Requirements depend on operation dtypes and training dynamics.
• Storing or updating every parameter in 16-bit: Very small updates can be lost, especially with BF16's coarse fractional precision; use the proper optimizer/master-weight design.
• Converting all computations to half precision: Some reductions, normalization, loss functions and optimizer calculations need higher precision.
• Ignoring checkpoint compatibility: A model trained in BF16 may overflow if loaded into an all-FP16 pipeline.
• Confusing theoretical throughput and observed time: Check hardware support, native kernels and actual benchmarks.
• Ignoring convergence differences: Compare FP16/BF16 outputs, loss trajectories, validation metrics, NaN/Inf rates and accuracy against FP32 on representative data.

Curriculum boundary: Question 293 introduced mixed-precision training and overall memory accounting. Question 294 focuses specifically on FP16 versus BF16 bit layouts, numerical range, rounding and hardware choice. Question 295 will separately cover gradient loss scaling mechanisms. This preserves the exact Efficient Training and Model Interpretation prerequisite order.

Key takeaway: FP16 has more numerical detail but a narrower range; BF16 has much wider range but coarser rounding. Both use 2 bytes per element. Prefer the precision that your hardware accelerates, your values can safely represent and your validation tests support.

Research and technical references:

• Kalamkar et al., *A Study of BFLOAT16 for Deep Learning Training* (2019).
• Micikevicius et al., *Mixed Precision Training* (ICLR 2018).
• NVIDIA, *Transformer Engine: Low Precision Training — Introduction*.
• NVIDIA, *CUDA Programming Guide — Floating Point Computation*.
• PyTorch, *Type Info: torch.finfo*.
• PyTorch, *Automatic Mixed Precision*.
• PyTorch, *torch.cuda.is_bf16_supported*.