Chapter 3: Numerical Formats and Transformer Workloads

Floating-point formats and numerical error

Predict and observe overflow, underflow, and rounding along a declared training or inference format path

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

Binary notation
Scientific notation
Addition and multiplication

A training program prints a finite loss for 600 steps and then prints `inf` after one gradient update. The saved gradient tensor contains values near 90,000, but the program stores those values in IEEE 754 half precision (FP16), whose largest finite value is 65,504. NVIDIA Compute Sanitizer reports no illegal memory access, so the failure comes from numerical representation rather than an invalid address.

A machine does not store an arbitrary real number directly. Each tensor element contains a fixed-width binary code with fields for sign, exponent, and significant digits. The chosen format determines the largest finite magnitude, the smallest retained magnitude, and the distance between adjacent representable values. A format conversion can therefore turn 90,000 into infinity or make a small parameter update round back to the unchanged parameter.

The phrase ‘FP16 model’ hides several real locations in the execution path. Model weights can occupy FP16 storage in GPU memory, matrix instructions can multiply FP16 inputs while accumulating partial sums in 32-bit floating point (FP32), a reduction can use FP32, and the output can be rounded back to FP16. The worked investigation records values at each conversion, plots the first overflowing step, and tests loss scaling as a specific intervention rather than treating bit width as an isolated vocabulary item.

Range and precision are separate properties

A binary floating-point value contains a sign, an exponent, and a significand. Increasing exponent bits increases the range of powers of two that the format can express. Increasing significand bits decreases the spacing between representable values at a given magnitude. The exponent and significand bit budgets solve different numerical problems.

FP16 uses one sign bit, five exponent bits, and ten stored fraction bits. The largest finite FP16 value is 65,504. BF16 uses one sign bit, eight exponent bits, and seven stored fraction bits. BF16 reaches approximately the range of FP32 but has coarser spacing. BF16 can therefore avoid an FP16 overflow while rounding nearby in-range values less precisely.

Spacing changes with magnitude

Floating-point numbers are not uniformly spaced. Within each exponent interval, the spacing is fixed, and it doubles when the exponent increases by one. A small update can affect a parameter near zero but disappear when added to a much larger parameter because no representable value lies between the original value and the exact sum.

This explains why range alone is insufficient. BF16 can represent very large and very small magnitudes, but its seven stored fraction bits provide fewer distinct values within each interval than FP16 or FP32. An optimizer update or normalization statistic can round even when it does not overflow or underflow.

Rounding and reduction order

Most arithmetic operations produce an exact mathematical result that must then be rounded to a representable value. Because each intermediate can round, floating-point addition is not associative: (a+b)+c can differ from a+(b+c). A parallel reduction changes the grouping of terms and can therefore change the final bits.

Error depends on the number, order, sign, and scale of terms. Summing many positive values can accumulate rounding in one direction. Adding values of opposite signs can cancel leading digits and expose relative error. Wider accumulation, pairwise reduction trees, or compensated summation can reduce error at a cost in instructions, storage, or throughput.

Separate storage, operand, product, and accumulator formats

Matrix hardware often reads narrow operands, forms products, and accumulates into a wider register format. FP16 or BF16 inputs with FP32 accumulation reduce operand traffic and use high-throughput matrix instructions while protecting partial sums from some narrow-format error.

The output is usually converted again when stored. An epilogue can apply bias or activation while the wider accumulators remain resident, then round once to the output format. Writing an intermediate narrow tensor and reading it for a second kernel introduces traffic and an additional rounding point.

Loss scaling addresses FP16 gradient underflow

Backpropagation can produce gradients too small for normal FP16 representation. Loss scaling multiplies the loss by a factor before backpropagation, which multiplies the gradients by the same factor. The system converts or accumulates these scaled values, then divides by the scale before applying the optimizer update.

A scale that is too small fails to protect tiny gradients. A scale that is too large can overflow other gradients. Dynamic loss scaling detects nonfinite gradients, reduces the scale after overflow, and can increase it after stable intervals. The method addresses representability; it does not guarantee that every operation is numerically safe in FP16.

Stable transforms change the computation without changing its mathematics

Direct softmax computes exp(x_i) divided by the sum of exp(x_j). A large positive input can overflow the exponential. Subtracting the row maximum from every input leaves the mathematical ratios unchanged because the same factor cancels from numerator and denominator, but it keeps the largest exponent at exp(0)=1.

Similar safeguards appear throughout model code: accumulating norms in wider precision, adding epsilon to denominators, rescaling blocks before low-precision conversion, and clipping or scaling values. Include their instructions and data movement in the performance model because numerical stability has a systems cost.

Low-bit quantization is a dataflow

Integer and low-bit formats usually represent a real value through a scale and a finite code. A scale can apply to the entire tensor, one channel, a group of values, or a small block. Smaller groups can follow local ranges more accurately but require more metadata and scale loads.

A lower stored bit count does not guarantee lower elapsed time. The execution path must load packed data, unpack or convert it efficiently, load scale metadata, use supported arithmetic instructions, accumulate appropriately, and avoid materializing a large dequantized tensor. Profile the complete path rather than inferring speed from storage size.

Worked example: Tracing precision through a linear layer

A linear layer stores signed 4-bit weight codes in groups of 128 weights with one FP16 scale per group and no zero-point metadata. The linear layer receives BF16 activations, accumulates matrix products in FP32, and returns BF16 output.

Calculate packed code storage: 128 weights × 4 bits = 512 bits, or 64 bytes. Add a 2-byte FP16 scale. The group uses 66 bytes, which is 0.515625 bytes per weight before alignment or other metadata.
Compare with BF16 storage at 2 bytes per weight. The nominal weight-storage reduction is about 3.88×, not exactly 4×, because scales consume space. Smaller groups would add more scale bytes per weight.
Trace execution. The kernel loads packed codes and group scales, reconstructs or rescales values in registers, and combines them with BF16 activations through a hardware-supported instruction sequence. FP32 accumulators hold partial sums.
Apply bias and any supported activation while accumulators remain on chip. Convert the final value to BF16 once. Materializing the reconstructed weight matrix in HBM would add a large write and read and remove much of the intended traffic reduction.
Test operator error across ordinary groups and groups with large outliers. Compare per-channel and end-to-end model quality because a small number of sensitive channels can dominate the degradation.
Measure HBM traffic, unpack and conversion instructions, scale traffic, and matrix-pipeline activity. Determine whether the new limit is weight bandwidth, dequantization overhead, or arithmetic execution for each important shape.

Conclusion: The layer uses several formats and conversion operations. Report the numerical and performance properties of the complete sequence instead of only the stored-weight format.

Common errors

Equating bit-width reduction with speedup. Hardware support, packing, scale traffic, alignment, and conversion determine elapsed time.
Reporting only the stored or operand format. Product, accumulator, reduction, and output formats can determine both accuracy and performance.
Assuming that a wider exponent always makes a format more accurate. Range and local precision are separate properties.
Checking average tensor error only. Rare outliers and downstream routing or token-selection boundaries can dominate behavior.

Reference summary

A floating-point value stores a sign, a significand, and an exponent. The exponent controls the range of magnitudes that the format can represent. The significand controls how closely it can distinguish nearby values at a given magnitude. Reducing the bit width can reduce storage and enable faster arithmetic, but it also changes which values round, overflow, or underflow.

FP16 uses more fraction bits than BF16 but fewer exponent bits. FP16 represents nearby values more precisely within its range, but FP16 overflows at much smaller magnitudes.
BF16 uses the same exponent width as FP32 and fewer fraction bits. BF16 retains a similar range to FP32 but rounds nearby values more coarsely.
A matrix instruction can multiply narrow operands and accumulate partial sums into a wider format. Operand format and accumulator format must therefore be reported separately.
Loss scaling multiplies gradients before FP16 backpropagation so that small values remain representable, then removes the scale before the optimizer update.
Floating-point addition is not associative. A different reduction tree can change the rounded result even when both algorithms are mathematically equivalent.

Knowledge check

A gradient magnitude reaches 90,000 during training. Explain why FP16 stores infinity while BF16 can store a finite value, then explain why a small update can still round away in BF16.

Show the answer guide

FP16 has a largest finite value of 65,504. Rounding a magnitude of 90,000 to FP16 exceeds that range, so the stored result is positive or negative infinity according to the sign.
BF16 uses the same eight exponent bits as FP32, so its finite range extends to roughly 3.4 × 10^38. A magnitude of 90,000 is therefore within the BF16 exponent range and remains finite.
BF16 retains only seven explicitly stored fraction bits, compared with ten in FP16 and twenty-three in FP32. Its finite range is large, but adjacent representable values are relatively far apart.
Near a large value, an update smaller than approximately half the local spacing rounds back to the original BF16 number. The update can therefore disappear even though neither operand overflows.
Training commonly combines wider accumulation or master weights with loss scaling and range checks because exponent range and update resolution are separate numerical requirements.

Practice

Observe range and spacing directly

With a numerical library that supports FP16 and BF16, convert 90,000 to both formats. Then choose a BF16 value near 1,024 and add progressively smaller increments until the stored BF16 result no longer changes; perform the same additions in FP32 for comparison.

Evidence to keep: Keep the printed FP16 and BF16 values for 90,000, the increment table, the first rounded-away BF16 update, corresponding FP32 results, and a written conclusion separating exponent range from precision.

The format path determines bytes and numerical limits for each operation. The next section expands a transformer layer into those operations and their tensor shapes.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.