Reproducible performance experiments
Produce an experiment record that another engineer can inspect, run, and compare after a software or hardware change
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 2: benchmark contracts and asynchronous timing
- Version control and machine-readable result files
- Access to one programmable CPU, GPU, or accelerator environment
A developer investigates row normalization for a BF16 activation tensor with 4,096 rows and 8,192 values per row. The original PyTorch path launches one GPU program for residual addition and another for root-mean-square normalization. A proposed fused program keeps the intermediate sum on the GPU chip instead of writing the full tensor to high-bandwidth memory (HBM) and reading the tensor back during the second launch.
The developer first records thirty CUDA-event timings without a profiler and saves every sample in a comma-separated-values file. A separate NVIDIA Nsight Compute run records estimated HBM bytes and generated GPU instructions. A Python correctness test compares output values with the original PyTorch path over ordinary inputs, extreme values, and row widths immediately below and above the CUDA tile size. Each artifact answers a different question: elapsed time, physical mechanism, or accepted output.
The laboratory record asks one narrow question before collecting the final data: does fusion remove the predicted intermediate HBM write and read, and does the byte reduction explain the ordinary timing change for the declared shapes? The question can fail in informative ways. Unchanged bytes would reject the assumed materialization; fewer bytes with unchanged time would point to another limit; a numerical failure would disqualify the faster code.
Write an experiment record before running the benchmark
The first field is the decision. State what will change if the result supports or rejects the hypothesis. A benchmark without a decision can continue indefinitely because no result is sufficient. The second field is a narrow question. Define the input shape family, datatype, software path, hardware, operating mode, and metric. The question must be answerable by the experiment you can actually run.
Write the hypothesis as a causal statement. Name the intervention and the expected resource change. For example: fusing residual addition with RMS normalization removes one output write and one later input read, so a bandwidth-bound row should take approximately the remaining-byte fraction of the original time. Record the predicted direction and approximate magnitude before examining the profiler.
List alternative explanations. A timing reduction could also come from fewer launches, a different accumulation order, changed cache state, or compiler specialization. Planning alternative explanations determines which counters, controls, and ablations the experiment needs. The written alternatives also prevent a later timing result from being assigned automatically to the preferred explanation.
Define correctness and the measurement boundary
A performance result is invalid when the contender computes an easier or different operation. Define output shape, datatype, mutation, numerical tolerance, handling of non-finite inputs, and any quality threshold. CUDA's Best Practices Guide recommends comparison with known-good reference outputs and notes that floating-point algorithms often need tolerance-based rather than bitwise comparison.
Define exactly what the timer includes. A kernel microbenchmark may exclude allocation, compilation, data transfer, and dispatch. An end-to-end request timer includes queueing and host work. Both measurements can be useful, but they answer different questions. Synchronize asynchronous accelerator work correctly, and do not compare a CUDA event interval with an unsynchronized host interval as though the boundaries were identical.
Separate setup from the measured region only when production can also amortize that setup. Prepacking constant weights can be excluded if the weights are packed once and reused many times, but the report must state the reuse assumption. Compilation cannot be excluded from a cold-start objective. The benchmark contract should make every such choice visible.
Collect repeated measurements without discarding their shape
Warm-up establishes the intended execution state: compiled code exists, allocator pools are populated, and clocks or caches have reached the state specified by the contract. Warm-up is not permission to discard inconvenient behavior. If production includes cold calls, measure cold and warm modes separately.
Collect repeated samples and preserve them. PyTorch's blocked_autorange increases work per timed block until timer overhead is small, then collects several measurements. Google Benchmark provides warm-up, repetitions, random interleaving, and machine-readable output. Benchmark-harness facilities reduce common timer errors, but they do not choose the correct workload or metric for you.
Inspect the distribution before summarizing it. Multiple modes can indicate clock changes, background work, cache transitions, compilation, or allocator behavior. Report median or another declared statistic with dispersion and sample count. Keep the raw samples beside the report. An aggregate-only console table should not be the only surviving evidence.
Use profiling as a separate controlled experiment
Profiler collection can perturb the program. Nsight Compute may replay kernels to collect incompatible metric sets, save and restore memory, serialize launches, or alter cache state. Its documentation warns that host timers and CUDA events include profiler overhead during such runs. Use ordinary timing runs for elapsed time and profiler runs to explain mechanism unless the tool documents an equivalent low-overhead mode.
Request the smallest metric set that can distinguish the competing explanations. To test removed HBM traffic, collect requested and transferred bytes in a profiler run, then collect elapsed time in a matched ordinary run without the profiler. Join the results by implementation, shape, input state, and environment; do not treat the profiler's perturbed duration as ordinary timing. To test a tensor-core path, inspect native instructions and issue activity. A large counter dump creates more replay and more opportunities for inconsistent state without necessarily adding evidence.
Preserve the profiler command, report file, target kernel selector, replay mode, cache-control setting, and tool version. A screenshot omits too much context. Exported reports and analysis scripts allow another engineer to inspect the same evidence and detect a mistaken interpretation.
Escalate validation only as far as the claim requires
Use a sequence of validation levels. Deterministic unit cases catch implementation errors quickly. Adversarial shapes and values exercise tails, alignment, overflow, underflow, empty inputs, and large cancellation. A microbenchmark isolates one mechanism. A trace replay tests a workload distribution. An end-to-end run tests whether the local change matters to the complete objective.
Do not claim a wider scope than the highest completed level. A microbenchmark can show that a kernel reduces bytes and time for declared shapes. A microbenchmark cannot show service goodput under load. A two-GPU collective test can validate ordering and small-topology behavior. A two-GPU test cannot establish congestion behavior for a large multi-tenant cluster.
Review experiment records over time. Identify models that consistently underpredict, profiler metrics that produced false leads, and environment changes that increased variance. Convert repeated setup into scripts and repeated errors into explicit checks. Keep rejected hypotheses because they prevent the same unsupported explanation from returning later.
Worked example: Building a reusable kernel experiment
A study begins with vector reduction but should support later normalization and attention work.
- Create a shape-and-dtype generator, high-precision reference, device-event timer, warm-up policy, environment manifest, and raw-result format.
- Write the first record before implementation. The question asks whether a hierarchical reduction approaches the streaming-read bound for FP32 input sizes from 2^10 through 2^26, including non-power-of-two tails.
- Write a byte and operation model for reduction and store predicted bounds beside results. Add adversarial scales and non-power-of-two sizes.
- Run correctness tests before timing. Compare with an FP64 CPU reference and define both relative and absolute tolerances for cancellation and near-zero sums.
- When moving to normalization, reuse timing and correctness infrastructure while extending the model to multiple statistics and full-tensor traffic.
- Archive profiler commands and a small analysis notebook that compares prediction with observation across kernels.
- Re-run one saved reduction result after a driver or compiler update. Compare raw distributions and record a new manifest. If the change exceeds baseline variation, isolate the changed driver, compiler pass, or generated instruction sequence when practical; otherwise report a version-associated difference rather than claiming that one software subsystem caused the change.
Conclusion: The reusable benchmark, correctness tests, environment record, and analysis code reduce the setup cost of later experiments.
Common errors
- Using unstructured experiment notes. A standard record format makes results comparable and searchable.
- Automating before understanding the measurement boundary. Infrastructure can reproduce the same mistake at scale.
- Deleting failures. Rejected hypotheses are part of calibration and prevent future repetition.
- Timing while a replay profiler is attached and reporting the result as ordinary execution time. Profiling overhead changes the measured boundary.
- Reporting only the fastest sample. The selected value describes measurement noise rather than the expected operating behavior.
Reference summary
For a fused normalization study, write the predicted removed HBM bytes before running the benchmark. Save ordinary CUDA-event timing samples separately from the NVIDIA Nsight Compute report that estimates HBM traffic. Store the tensor shapes, environment manifest, reference-output comparison, source revision, and plot script beside the conclusion. The complete directory—not the fastest console line—is the experiment record.
- Before every experiment, state the question and predicted mechanism before recording a timestamped CPU/GPU timeline with a profiler such as NVIDIA Nsight Systems.
- Preserve raw samples, failed hypotheses, correctness results, and environmental state.
- Alternate analytical models with hardware validation; investigate the gap between them.
- Revisit an old result on a new shape or architecture to test whether the explanation transfers.
Knowledge check
Which source, environment, raw-data, correctness, and profiler artifacts let a reader inspect the causal claim even when the original GPU is unavailable?
Show the answer guide
- Source artifacts include the exact source revision or archive, build scripts and flags, benchmark and reference implementations, workload generator, and the commands used to run every result.
- The environment manifest records hardware identifiers, topology, operating system, driver, CUDA or accelerator runtime, compiler, framework and library versions, clock or power settings, and relevant environment variables.
- Raw-data artifacts include every ordinary timing sample in a machine-readable file, workload shapes and seeds, sample count, warm-up record, timer endpoints, and the script that produces tables or plots.
- Correctness artifacts include the trusted reference, adversarial cases, tolerance or behavioral contract, complete test output, and model- or task-quality result where the claim extends beyond one operator.
- Profiler artifacts include the tool version, exact command, selected kernel and metric set, cache or replay settings, exported report, and any script used to extract HBM bytes or generated instructions. The report remains inspectable even when the hardware cannot be rerun.
- The written prediction, byte or operation calculation, controlled ablation, conclusion, rejected alternatives, and scope connect the archived observations to the causal claim without requiring access to the original GPU.
Reproduce a reduction after a build change
Benchmark a sum reduction over 2^20 FP32 values using one fixed implementation and input seed. Run 40 ordinary timings, change exactly one compiler optimization level or library build version, rebuild, and repeat while preserving identical input and reference output.
Evidence to keep: Keep both source/build manifests, input checksum, raw timing CSV files, correctness output against a wider reference, generated instruction or compiler report, distribution plot, and a version-bounded conclusion about the observed change.
After you can run reproducible experiments, analyze one performance mechanism across several software and hardware layers.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- The Rust Programming Language↗
Structural inspiration: progressive chapters, concrete examples, and explicit learning flow.
- CUDA C++ Best Practices Guide↗
- Nsight Compute Profiling Guide↗
- MLPerf Inference Rules↗
- torch.compile Programming Model↗
- PagedAttention / vLLM↗