GPU Memory Optimization

A source expression such as y[i] = a * x[i] + y[i] contains one multiply and one addition, but a graphics processing unit (GPU) must also obtain two input values and store one result. For many kernels, those transfers require more time than the arithmetic. The important question is therefore not only how many bytes the program requests. The question is how the hardware groups requests into transactions, where values are reused, and how often a value moves between named storage levels.

This chapter follows one value from off-chip device memory to an on-chip cache near a streaming multiprocessor (SM), into shared memory, through registers, and into a later operator. Data-center accelerators often use high-bandwidth memory (HBM), while many other GPUs use graphics double data rate memory. The same requested-versus-transferred byte analysis applies to both. Coalescing determines whether a warp uses most of each global-memory transaction. Shared-memory bank mapping determines whether on-chip requests proceed in parallel. Register allocation determines how much thread-local state can remain immediately available. Tensor layout determines whether these properties continue across operator boundaries.

The analysis method is consistent throughout the chapter. First identify the logical values required by the algorithm. Then map each lane to an address. Calculate requested bytes, transferred bytes, reuse, and storage capacity. Finally, measure the relevant traffic and elapsed time. This sequence prevents a favorable cache percentage or occupancy number from replacing the actual data-movement calculation.