Performance Engineering Practice

An engineer changes a normalization operation in the recurring 7-billion-parameter inference service, which runs on one 80 GiB graphics processing unit (GPU). A local CUDA-event benchmark reports a 22-microsecond improvement, but the weekly report contains no tensor shape, raw samples, correctness result, source revision, or end-to-end request measurement. Three months later, nobody can determine whether a compiler update removed the gain or whether the original comparison timed different work.

Chapter 21 turns an optimization attempt into a reproducible experiment. A laboratory record names the question, predicts a mechanism, captures the environment, stores raw timing samples, and links a separate profiler report to the ordinary benchmark. The chapter follows learned weights through general matrix multiplication (GEMM), retained attention data through the key-value (KV) cache, and host preparation on the central processing unit (CPU). High-bandwidth memory (HBM) names the large off-chip memory attached to each GPU whenever the experiment counts device-memory traffic.

The chapter also connects local results to user-visible measures. Time to first token (TTFT) runs from a declared request-entry timestamp to delivery of the first generated token. Time per output token (TPOT) measures the cadence of later generated tokens for an active request. A faster GPU program matters to the inference service only when the experiment shows how the changed interval affects TTFT, TPOT, capacity, cost, or another declared objective.

Every practice section produces artifacts that another engineer can inspect: a versioned experiment record, a cross-layer representation table, a causal report, a bounded capstone proposal, or a version-aware reproduction note. The artifacts preserve failed hypotheses and limitations as carefully as the fastest result because future decisions depend on knowing where the evidence stops.