Performance Measurement and Benchmarking

An engineer runs a text-generation program twice and sees 1,940 output tokens per second in the first run and 2,070 in the second. Neither number is useful until the experiment identifies what entered the program, which work the timer included, whether the graphics processing unit (GPU) had already compiled and cached the operations, and whether both runs produced answers of acceptable quality. A benchmark is the complete controlled experiment that supplies those conditions; a benchmark is not merely the loop that reads a clock.

Chapter 2 builds one benchmark for a 7-billion-parameter decoder service. The lesson specifies the joint distribution of prompt and output lengths, the arrival times of requests, the server and GPU software versions, and the points at which the timer starts and stops. The lesson then shows why a CPU timer can record only the submission of a GPU command even though the GPU continues working for milliseconds after the call returns.

The final measurement step checks the result as well as the time. A faster matrix or attention implementation is not an improvement when numerical changes damage the model's output or a partial GPU tile writes outside its allocation. The chapter therefore joins raw timing samples with deterministic operator tests, model-quality measurements, and a record of the exact environment. Another engineer should be able to reproduce both the observed speed and the allowed result.