Benchmark requirements

State the decision

A benchmark for kernel development asks whether one implementation reduces device time for a defined shape family while preserving numerical behavior. A benchmark for hardware selection asks how many systems are required to meet a workload and service objective. A benchmark for capacity planning asks where goodput stops increasing under load. The three engineering decisions require different timing boundaries and statistics.

Write the decision in the benchmark record. This determines which costs must be included. Model loading may be irrelevant to a week-long training run, important to a frequently restarted service, and the primary metric for a scale-to-zero deployment.

Describe the workload population

AI work is strongly shape-dependent. For training, record global batch, microbatch, sequence-length policy, padding or packing, parallel configuration, activation checkpointing, optimizer, and data format. For serving, record prompt and requested-output distributions, arrival process, concurrency, batching policy, decoding settings, and cache state.

Preserve relationships between variables. If long prompts usually request short answers, independently sampling prompt and output lengths creates a workload that production never sees. A replay benchmark should retain this correlation or explain the approximation.

Define the system under test

List the hardware and software that can change the mechanism. Hardware model and count are insufficient when memory capacity, topology, power caps, clocks, firmware, driver, runtime, libraries, compiler settings, graph mode, and scheduler configuration affect the result.

Record the source revision and configuration in machine-readable form when possible. The benchmark program can emit a manifest beside its raw samples. This reduces transcription errors and makes later comparisons against a changed environment possible.

Choose cold, warm, and steady states deliberately

The first invocation can include compilation, library initialization, autotuning, memory allocation, page mapping, cache population, and lower device clocks. Later invocations can reuse all of this state. Neither state is inherently more correct. The deployment determines which one matters.

Separate states when they answer separate questions. Report cold-start time for deployment readiness, warm steady-state service for capacity, and cache-hit or cache-miss cases for a prefix-caching system. Do not remove a cost from the timed interval merely because it makes the measurement less stable.

Set measurement boundaries

A GPU-kernel-sequence benchmark can begin immediately before the first kernel launch and end when the device completes the final kernel. An end-to-end inference benchmark can begin when a query enters the service and end when the complete response is delivered. State whether tokenization, network ingress, queueing, transfers, and postprocessing are inside the boundary.

A narrow timing interval helps isolate a mechanism. A wide interval measures experienced performance. Use both when you need to explain why a shorter GPU-kernel sequence does or does not change the complete system.

Report distributions and constraints

A median is often suitable for repeated steady kernel intervals because it resists occasional host interruptions. Interactive services require tail percentiles under a declared offered load. Training time to quality can require repeated runs because initialization and data order affect convergence.

Preserve raw observations and report the quality or correctness limit beside the performance result. MLPerf Inference, for example, defines scenarios with different query generation and latency requirements and requires quality compliance. The broader lesson is that a performance number is meaningful only within its workload and validity rules.

State the sample count behind every percentile. With only 100 independent observations, an empirical p99 is determined by approximately the single slowest observation and is therefore unstable. A production tail estimate usually needs many more observations, a sufficiently long steady interval, and repeated runs. Preserve the empirical distribution instead of reporting a tail percentile with unsupported decimal precision.