Introduction to AI Systems Performance Engineering

A user opens a coding assistant, enters a 2,048-token question, and presses Send. The browser sends an HTTP request to a server. The server's central processing unit (CPU) converts the text into numbered vocabulary entries called tokens, places the request in a waiting queue, and eventually assigns the request to a graphics processing unit (GPU). The GPU runs many small parallel programs over arrays of numbers that represent the text and the model's learned parameters. The service returns one generated token at a time until the answer ends. A delay can arise before the GPU starts, while the GPU moves numbers through memory, between GPU operations, or after the GPU produces an output.

Chapter 0 follows that request before asking you to use performance terminology. The model worker stores its numeric arrays in GPU memory. Engineers call a multidimensional numeric array a tensor. The framework asks the GPU to process those tensors by launching small parallel programs called CUDA kernels.

Measurement software records another concrete object. NVIDIA Nsight Systems can observe selected CPU calls, CUDA commands, data transfers, and GPU kernels while the request runs. The resulting timestamped event record is a profiler trace. A timeline view draws each recorded event as an interval between its start and end times.

The model combines activation tensors with learned-weight tensors through general matrix multiplication (GEMM). During output generation, the worker also retains attention history in a key-value (KV) cache so the next iteration does not recompute every earlier result. Later lessons derive both mechanisms from concrete tensor shapes.

The chapter then introduces the investigation method used throughout the book. An engineer records the result that users need, observes where time and resources are spent, proposes a physical or software cause, changes one relevant condition, and checks the prediction. Diagnostic reports can count arithmetic as floating-point operations (FLOP), but a FLOP rate is useful only after the engineer connects it to the requested result. The exercises produce inspectable artifacts—a request timeline, raw timing samples, a correctness comparison, and a written conclusion—because an optimization claim must be reproducible by someone who did not watch the original experiment.