Matrix Multiplication (GEMM)

Open a profiler around one layer of the 1-billion-parameter training example, which uses four 2048-token sequences on each of eight GPUs, and several rectangular blocks usually dominate the GPU timeline. The framework may label them aten::mm, aten::linear, cublasLtMatmul, or a fused projection. Each block multiplies one two-dimensional array of numbers by another. During prompt processing, one dimension can contain thousands of token positions. During single-token generation, the same dimension can be only one or a few dozen. The mathematical operation is matrix multiplication in both cases, but the GPU executes the two shapes very differently.

Engineers usually call the operation general matrix multiplication, abbreviated GEMM. A typical GEMM computes C = A × B, sometimes followed by scaling, a bias addition, an activation, or a datatype conversion. The chapter begins with the scalar dot product that produces C[row, column]. The chapter then follows the same input values from GPU high-bandwidth memory into on-chip shared memory and registers, where many output calculations can reuse them.

Nsight Compute makes the progression observable. A first kernel that loads the same row and column repeatedly reports large device-memory traffic and low arithmetic throughput. A tiled kernel reports fewer high-bandwidth-memory transactions per floating-point operation. A pipelined tensor-core kernel can overlap transfers with matrix instructions, but the profile can regress when its shared-memory or register demand leaves too few thread blocks resident. Each optimization therefore has a numeric signature and a resource cost.

The terminology follows the profiler. A central processing unit (CPU) launches each matrix multiplication (GEMM) on a graphics processing unit (GPU). Matrix data usually begins in high-bandwidth memory (HBM). Thread blocks run on streaming multiprocessors (SMs), where the single instruction, multiple threads (SIMT) execution model coordinates lanes. Throughput reports count floating-point operations (FLOPs), while the byte model counts transfers among memory levels.

The chapter derives thread-block tiles, warp tiles, instruction tiles, software pipeline stages, and output epilogues from one concrete matrix shape. The final lesson returns to the profiler timeline and explains why a library selects different kernels for a 4096 × 4096 training multiplication, a group of small mixture-of-experts multiplications, and a decode projection with only a few input rows.