Numerical Formats and Transformer Workloads

A transformer diagram can show a box labeled Attention and another box labeled Feed Forward. A running program must turn each box into concrete arrays and operations. For a batch of two prompts with 1,024 tokens each and a hidden width of 4,096 values, one activation array contains 2 × 1,024 × 4,096 numbers. At two bytes per number, that single array occupies about 16.8 megabytes. Graphics processing unit (GPU) programs read the array, use general matrix multiplication (GEMM) to multiply the array by learned weight matrices, create temporary arrays, and keep selected results for later work.

Chapter 3 names such a multidimensional numeric array a tensor only after specifying its axes, element format, physical storage order, producer, consumer, and lifetime. The chapter also shows where floating-point formats enter the real execution path: model weights can be stored in BF16, matrix hardware can multiply BF16 operands while accumulating partial sums in FP32, and the final tensor can be converted back to BF16. Every format choice changes both the representable numbers and the bytes that the machine moves.

The chapter then follows two complete workloads. A training step performs a forward calculation, preserves selected intermediate tensors, computes gradients in reverse order, and updates learned weights. An inference request first processes all prompt tokens, then repeatedly generates one new token while retaining attention history in a key-value (KV) cache. Explicit tensor shapes, counts expressed in floating-point operations (FLOP), and allocation lifetimes explain why the two workloads consume memory and arithmetic resources differently. Later examples name high-bandwidth memory (HBM) as the large off-chip memory attached to the GPU.