Chapter 0: Introduction to AI Systems Performance Engineering

Performance optimization process

Write and run an investigation whose predicted observation can be rejected by measurement

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

Ability to name the request stage and timing interval being investigated

A PyTorch inference program applies layer normalization to a tensor with 4,096 rows and 8,192 BF16 values per row. PyTorch asks the GPU to run a parallel program that reads each row, calculates its mean and variance, and writes a normalized row. NVIDIA Nsight Compute, a kernel profiler, reports that this particular GPU program ran for 38 microseconds. Engineers call one parallel program launched on a GPU a GPU kernel; the name describes executable code, not the complete model operation or the hardware itself.

The same Nsight Compute report estimates how many bytes the kernel transferred from high-bandwidth memory (HBM), which is the large device memory attached to the GPU, into the chip's cache hierarchy. A high byte count and low arithmetic rate suggest that data movement might determine the 38-microsecond duration. The profiler report does not prove that explanation. A byte count, an achievable HBM transfer rate, and a controlled code change are required before the engineer can attribute the duration to HBM traffic.

A performance investigation turns the observation into a testable causal claim. The engineer records the exact tensor shape and software environment, calculates the expected bytes, predicts how much time a fused implementation could remove, changes only the intermediate read and write, and repeats both timing and correctness tests. A supported conclusion names the measured shape, machine, mechanism, and remaining uncertainty; a single smaller timing number is not enough.

Establish a representative baseline

Representation and reproducibility are separate requirements. A benchmark is representative when its shapes, data formats, arrival patterns, cache state, quality settings, and system boundaries match the decision you intend to make. A benchmark is reproducible when another engineer can repeat the procedure and obtain a statistically compatible result.

Record the source revision, build configuration, hardware, topology, driver, libraries, clocks or power policy, inputs, warm-up, timing boundaries, and raw samples. Also record state that changes over time. Compilation caches, memory pools, autotuners, GPU temperature, and request queues can all change the result after the first iteration.

Write a quantitative hypothesis

A useful hypothesis identifies a limiting mechanism and predicts observations that follow from it. ‘Memory is slow’ does not specify a memory level, byte count, or expected time. A testable claim is more precise: ‘The kernel transfers at least 96 MB from HBM. At a measured sustainable bandwidth of 2.4 TB/s, those transfers require at least 40 microseconds. The measured kernel time is 44 microseconds, so removing a 32 MB intermediate round trip should reduce time by approximately 13 microseconds until another limit becomes active.’

Write down what would reject the hypothesis. If measured HBM bytes do not fall after the intermediate is removed, the assumed materialization may not have occurred. If bytes fall but time does not, the original transfer may have overlapped other work or another resource may already dominate. A rejected hypothesis improves the model by identifying which assumption was wrong.

Change the proposed mechanism

The intervention must affect the cause named by the hypothesis. To test HBM traffic, change fusion, tiling, caching, or recomputation so that the expected byte count changes. To test launch overhead, keep kernel work stable while changing launch aggregation or graph replay. To test imbalance, change partitioning or work assignment while keeping total work comparable.

Change one causal mechanism at a time when practical. Compiler flags and library settings often alter several mechanisms together. They can still be useful in production, but they provide weaker explanatory evidence. Follow a broad configuration result with controlled experiments that isolate the source of the change.

Verify mechanism, objective, and correctness

Repeat the baseline procedure after the intervention. Compare distributions instead of selecting the best iteration. Check whether the diagnostic quantity changed in the predicted direction and approximately the predicted magnitude. A faster result with unchanged HBM traffic does not support an HBM-traffic explanation, even if the optimization is operationally valuable.

Then evaluate the complete performance objective and correctness requirements. A kernel improvement can disappear after launch, queueing, or communication is included. A precision or reduction change can exceed the accepted error. Report the narrow causal conclusion separately from the broader product conclusion.

Control confounding state

Modern AI systems adapt while they run. Compilers specialize graphs, libraries autotune algorithms, allocators retain blocks, frequencies change with power and temperature, and caches preserve data from earlier inputs. Distributed workers also inherit network and filesystem state. Changing runtime variables can make two nominally identical runs execute different work.

Control a variable only when control does not remove behavior that the question requires. For a steady-state kernel study, complete compilation and warm-up before timing. For a deployment that frequently scales from zero, cold initialization is part of the objective and must be measured separately. The measurement state follows from the intended operating state.

Worked example: Investigating a slow normalization kernel

A residual addition followed by layer normalization takes a median of 38 microseconds for a common BF16 shape. The current implementation writes the residual result to an intermediate tensor and launches a second kernel for normalization.

Define the timed interval as both kernels and their intermediate materialization. Complete compilation and allocator warm-up, then measure thirty intervals with device events. Record shape, strides, formats, clocks, software versions, and all raw samples. The median is 38 microseconds with a narrow distribution.
Count the compulsory reads and writes for the residual input, activation input, intermediate tensor, scale, bias, and output. Use a measured sustainable bandwidth for a comparable access pattern. The resulting model predicts 31 microseconds for the current traffic and 24 microseconds if fusion removes the intermediate write and read.
Check the profile counters against the byte model. If measured HBM traffic is much lower, some data may be served by cache. If it is higher, the model may omit transactions, poor coalescing, or extra passes. Revise the expected saving before changing the code.
Fuse the residual calculation into the normalization kernel. Keep input formats and output contract unchanged. Add correctness cases for odd hidden sizes, non-contiguous legal inputs, large magnitudes, and nearly zero variance.
Repeat the measurements. Suppose the fused result is 25.5 microseconds and HBM traffic falls by the expected amount. Verify the enclosing model's quality and inspect register use to ensure the fusion has not created a new occupancy problem at other shapes.

Conclusion: The observed traffic reduction and elapsed-time change agree with the hypothesis within the accuracy of the model. The result supports fusion for the measured shape family, subject to the completed correctness and shape-sweep results.

Common errors

Changing several mechanisms in one experiment and assigning the complete result to one of them. The combined configuration can be useful, but it does not isolate the cause.
Using the hardware specification's peak rate as if every workload could sustain it. Calibrate important rates with a suitable microbenchmark or document why the peak is applicable.
Profiling before defining the measurement state. A detailed trace of compilation, allocator growth, or cache warm-up may not describe steady operation.
Discarding rejected hypotheses. They document tested assumptions and prevent later investigators from repeating the same incorrect explanation.

Reference summary

A performance investigation begins with a reproducible baseline and a causal hypothesis. A GPU kernel is one program submitted for parallel execution on the accelerator. HBM is the accelerator's off-chip high-bandwidth memory. If HBM traffic limits a kernel, a calculated reduction in transferred HBM bytes should produce a corresponding reduction in elapsed time until another resource becomes limiting. This prediction gives the experiment a condition that can disprove the hypothesis.

Baseline: stabilize the workload and record input shapes, software versions, warm-up, clock policy, measurement boundaries, and variance.
Model: calculate what the proposed limiting resource predicts for time, traffic, utilization, scaling, or queue behavior.
Intervention: change one mechanism that the hypothesis identifies, such as memory traffic, launch count, layout, precision, partitioning, or scheduling.
Verification: repeat the baseline procedure, check the predicted diagnostic change, and test numerical or semantic correctness.
Record: save the setup, raw observations, calculations, failures, and rejected hypotheses so another engineer can repeat the investigation.

Knowledge check

A dashboard reports that GPU activity rose from 60% to 90%. Which request-level measurements and correctness results are still required before claiming that the service improved?

Show the answer guide

Record offered load, admitted requests, completed requests, rejection, cancellation, failure, and retry counts. Higher GPU activity is not useful when the completion or goodput rate does not rise.
Compare the complete request-latency distributions, including TTFT, inter-token gaps or TPOT, and end-to-end latency for the same request classes. A utilization increase can coexist with longer queues and worse tails.
Hold prompt lengths, output lengths, model weights, tokenizer, decoding policy, cache state, and hardware allocation constant so the two measurements describe the same work.
Run the existing operator and model-quality checks, including structured-output or tool-call validity where applicable. A faster path that changes accepted output is not the same service result.
Use scheduler logs and a CPU/GPU timeline to explain the activity increase: larger batches, redundant work, retries, or long requests can all raise activity without improving request outcomes.

Practice

Test a parsing-removal hypothesis

Write a small program that reads the same 10,000-line JSON file 30 times. The baseline parses the file on every iteration; the contender parses once and reuses the resulting objects. Before timing, predict which CPU work disappears. Verify that both paths produce the same record count and checksum.

Evidence to keep: Keep the input checksum, prediction, 30 raw timings per path, output count and checksum test, median comparison, and a causal conclusion that states whether removed parsing explains the measured change.

The investigation process depends on quantitative predictions. Chapter 1 develops the first set of models used to make those predictions.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.

The Rust Programming Language↗
Structural inspiration: progressive chapters, concrete examples, and explicit learning flow.
NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.
PyTorch Profiler documentation↗
Official reference for recording operator activity, CPU and accelerator events, shapes, stacks, and exported traces.
MLPerf Inference Rules↗
MLPerf Training Benchmark↗