Performance Models and Metrics

Run one language-model request while NVIDIA Nsight Systems records the activity of the host central processing unit (CPU) and graphics processing unit (GPU). Nsight Systems produces a time-ordered file in which every recorded row has a start time, an end time, and a label such as a Python operation, a CUDA kernel launch, or a memory copy. Engineers call this timestamped event record a profiler trace. A ten-second recording can contain thousands of rows because one generated answer can launch many GPU programs for every output token.

A long event list does not explain the delay by itself. Suppose the trace shows a 1.8-millisecond matrix-multiplication kernel and a 0.4-millisecond normalization kernel. The useful questions are quantitative: How many numerical operations did each GPU program have to perform? How many bytes traveled from the GPU's high-bandwidth memory into on-chip caches? Which recorded intervals lie on the chain of dependencies that determines request latency? How many admitted requests were waiting while the GPU completed 600 requests per second? Each question names an amount, a unit, and a relationship that can make a testable prediction.

The first lesson connects an amount of arithmetic or transferred data to the rate at which named hardware completes that work. The arithmetic unit is a floating-point operation (FLOP). A later example counts the repeated multiply-and-add work in general matrix multiplication (GEMM), written as `C = A × B`.

The second lesson asks how much a shorter GPU interval can reduce the complete request. Amdahl's law supplies the calculation, while a dependency timeline shows whether the shortened interval lies on the request's critical path.

The third lesson connects admitted requests, completed requests per second, and time spent inside named server entry and exit events. The fourth lesson compares arithmetic work with bytes transferred between GPU high-bandwidth memory and the on-chip cache hierarchy. Each model omits details on purpose; later measurements test whether the omitted details matter for the workload under investigation.