Performance report structure

Write the result and scope first

The first paragraph should state the decision, intervention, result, and scope. For example: replacing a two-kernel RMSNorm sequence with one Triton kernel reduced median warmed layer time by 23% for BF16 rows of widths 4096–16384 on an H100, while preserving the declared FP32-reference tolerance. The compiled end-to-end block improved by 6%. The specialized path was slower below width 1024 and is not dispatched there.

The opening paragraph answers what changed and where the result matters. The opening does not explain the complete mechanism yet. Avoid terms such as utilization unless the specific resource and denominator are named. Avoid a single multiplier when the baseline, timing boundary, and shape are absent.

Follow with a short decision statement. State whether to adopt the path, retain it behind a dispatch predicate, reject it, or collect more evidence. The rest of the report should provide enough information to audit this recommendation.

Define workload, baseline, and correctness

List input shapes or their distribution, datatype, layouts, device, software versions, execution mode, concurrency, warm-up, timing method, and target metric. State what setup is excluded and why. For serving, include request distributions and latency constraints. For training, include global batch and parallel configuration.

Select the strongest relevant baseline. A naive educational implementation is useful for explaining a mechanism but cannot establish an improvement over current practice when a mature library is available. If different baselines answer different questions, show both and label them. Use identical semantic and timing boundaries.

Report correctness before performance. Include reference, tolerances, adversarial cases, gradient or quality checks, and unsupported inputs. MLPerf's rules separate performance from accuracy requirements and require system information for replication.

Separate observation, inference, and extrapolation

An observation states a result from a declared method: requested HBM traffic fell from 1.20 GB to 0.68 GB per iteration in the recorded profiler run. An inference connects observations to a cause: the traffic reduction and unchanged instruction path support the conclusion that intermediate removal explains most of the timing change. An extrapolation applies the result elsewhere: a similar benefit is expected on another bandwidth-limited GPU.

Use explicit labels when the distinction could be missed. State measured, calculated, inferred, simulated, or unverified. An inference needs converging evidence. Timing alone establishes correlation with the intervention. A byte model, traffic counters, a controlled unfused variant, and consistent shape scaling provide a stronger causal case.

Do not use counter names as explanations. A lower stall metric can result from several changes and can be affected by replay. Explain how the metric is defined, what competing hypotheses predict, and why the observed direction distinguishes them. Preserve the profiler configuration with the report.

Design figures and tables around questions

Every figure should answer a sentence stated in the text. A shape sweep shows the supported and losing regions. A roofline shows whether a measured point is consistent with a bandwidth or compute limit. An annotated timeline shows which critical-path interval changed. A memory-lifetime diagram shows which overlapping values set peak capacity.

Axes need quantity, unit, and scale. Legends need implementation and configuration. Captions should state the controlled conditions and the conclusion that the figure supports. Show raw points or distributions when aggregation could hide modes. A table can be clearer than a chart for a small number of exact values or eligibility boundaries.

Keep comparisons honest. Use the same axis range, workload, correctness target, and measurement boundary. If values were normalized, preserve absolute values in a table or artifact. If a figure omits outliers, state the rule and show their count. Visual polish does not compensate for an undefined denominator.

Present the counterfactual and alternative explanations

The counterfactual is the expected result without the intervention under the same conditions. The baseline usually represents the counterfactual. Controlled ablations improve the comparison by adding intermediate versions: remove fusion but retain the new tile, retain fusion but change pipeline depth, or disable weight packing while keeping dispatch fixed.

List plausible alternatives and the evidence against them. A kernel speedup attributed to lower traffic might actually come from different launch count, cache state, clocks, or numerical work. The report should show which factors were held constant and which remain unresolved. Rejected hypotheses can be summarized with their decisive test and linked to complete records.

Limitations define where the decision must be reconsidered. Include slower shapes, unsupported architectures, untested scale, compile cost, additional memory, numerical tradeoffs, and operational risks. State rollback or fallback behavior for a production change. Explicit limitations prevent future readers from applying a correct local result outside its evidence.

Attach a reproducibility package

Store the source revision, environment manifest, build instructions, benchmark and correctness commands, input generator, raw measurements, profiler reports, and plot script. Give each figure and table a producing command or data file. Include expected runtime and required hardware.

Test the package from a clean environment or ask a colleague to follow it without verbal guidance. Artifact-evaluation processes such as USENIX separate availability, functionality, and reproducibility because a downloadable archive is not automatically runnable and a runnable artifact does not automatically reproduce every claim.

When hardware or data cannot be shared, publish the smallest permitted reproducer, synthetic input generator, measured parameter table, and explicit access limitation. State which claims another person can verify and which claims depend on unavailable resources.

Reproducibility does not override privacy, security, or license requirements. Remove access tokens, credentials, private hostnames, customer prompts, proprietary model weights, and restricted datasets unless publication is explicitly authorized. Replace protected traces with a documented synthetic generator or aggregate distribution when possible. Record what was removed or transformed and explain which conclusions the substitution can no longer verify.