GPU and Accelerator Architecture

A graphics processing unit (GPU) is not simply a central processing unit (CPU) with more cores. The GPU allocates hardware resources for workloads that contain many similar, independent operations. A CPU invests heavily in reducing the completion time of a small number of instruction streams. A GPU instead keeps many groups of threads available and switches among them when an instruction must wait. Both processors execute instructions and access memory, but the processors tolerate delay in different ways.

The chapter develops the distinction from a concrete memory-load example. The later sections explain how CUDA threads become warps, how warps occupy streaming multiprocessors, how matrix engines accelerate structured operations, and how data moves through the memory hierarchy. The terms are specific enough for NVIDIA profiling tools, but the underlying model also applies to AMD compute units and other tensor accelerators.

The purpose of the chapter is prediction. By the end, you should be able to examine an operator and predict which hardware resources it needs, which conditions can prevent those resources from doing useful work, and which measurements would confirm the prediction. Later chapters will turn this architectural model into CUDA code and profiler analysis.