Performance diagnosis report
Write a report that names the workload, prediction, correctness, hardware mechanism, named-operation timing, application effect, limitations, and reproduction artifacts
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Previous sections on profiler scope, kernel profiles, and native instructions
- Chapter 2 controlled-comparison method
A faster benchmark result is not yet an engineering conclusion. Another engineer needs to know which workload was measured, which physical work changed, whether output remained correct, whether the application improved, and where the optimization does not apply. Without this information, the change cannot be validated or maintained.
A performance report should begin with the decision and quantified result. The supporting sections then establish the workload, baseline, causal mechanism, correctness evidence, affected-kernel timing, end-to-end effect, and limitations. The report order lets a reader understand the conclusion before examining detailed profiler traces.
The report should also preserve failed hypotheses when they are informative. If occupancy tuning did not improve a kernel already at the device-memory bandwidth roof, that result prevents the next engineer from repeating the same experiment. Failed tests belong in a concise investigation record, not mixed into the main conclusion as an unstructured diary.
State scope, baseline, and prediction
Describe the workload distribution, machine, clocks or power state, software versions, load, objective, and correctness constraints. A baseline is an operating point rather than one isolated number. Include latency distribution and throughput together when queueing or batching can trade one for the other.
Name the operation that contributes critical time and state the proposed mechanism before reporting the result. For example: an unfused normalization writes and reads a 200 MB intermediate; fusion should remove those bytes and reduce device-memory service time. This prediction identifies the counters and timeline changes that must accompany a valid result.
Describe the code change in physical terms. Useful categories include bytes removed, launches removed, bank conflicts reduced, active lanes increased, dependency chains shortened, or matrix instructions selected. A source diff alone does not explain why the machine should run faster.
Present four connected forms of evidence
Correctness evidence establishes that the optimized program implements the required operation. Include adversarial shapes, deterministic checks where possible, numerical tolerances, and model-level evaluation when reduced precision or reassociation changes results.
Mechanism evidence shows that the predicted hardware quantity changed. Kernel or subsystem evidence shows that the named operation became faster under controlled timing. End-to-end evidence shows that the original application objective improved at representative load. A complete report contains all four forms of evidence because each excludes a different false conclusion.
For example, fewer device-memory bytes without a kernel speedup can mean that another pipeline became limiting. A faster kernel without an application gain can mean that the kernel contributes little critical time. An application gain without stable correctness is not acceptable. Present the chain so the reader can locate any broken link.
Calculate value and document limits
Translate the measured end-to-end effect into capacity, latency headroom, energy, or cost at the target workload. Use the frequency of the operation and its original critical-time fraction. Do not multiply a microbenchmark speedup directly by fleet size. Amdahl's law provides an initial bound, and the application measurement provides the actual effect.
Let f be the fraction of original critical time spent in the changed normalization kernel and let s be the kernel's speedup. If every other part remains unchanged, the complete-program speedup is 1 / ((1 - f) + f / s). With f = 0.09 and s = 1.12, the bound is 1 / (0.91 + 0.09 / 1.12) = 1.0097, or about 0.97 percent. Invocation count does not determine f because invocations can have different durations or overlap noncritical work.
A latency bound is not yet a capacity result. Queueing, batching, and admission can change when the normalization kernel becomes faster. Repeat the offered-load sweep and find the highest stable load that still meets the same quality and latency objectives. Use that measured goodput change for provisioning. The offered-load sweep prevents a one-percent latency estimate from being reported as a one-percent fleet reduction when reserve, traffic fragmentation, or another bottleneck prevents consolidation.
Document shapes that regress, unsupported datatypes, architecture and compiler sensitivity, additional memory use, operational risks, and fallback behavior. State how the optimization can be disabled or rolled back. Documented limits prevent a valid local result from being applied outside its measured scope.
Archive the commands, binary, profiler reports, raw timing data, and environment description needed to reproduce the result. The narrative should remain understandable without opening every artifact, while the artifacts allow another engineer to verify each claim.
Worked example: Calculating the fleet effect of a 12% kernel improvement
A normalization kernel is 12% faster. The model invokes the kernel 80 times per generated token and serves a large request volume.
- Measure its original fraction of request critical time, not only invocation frequency. Suppose it contributes 9% under the target workload.
- Amdahl's law predicts about a 1% direct latency gain. Verify the end-to-end distribution and whether fusion also reduces memory pressure or enables larger batches.
- Translate the observed capacity gain into fewer provisioned GPU-hours at the same SLO, including headroom and workload mix. Avoid multiplying the microbenchmark percentage by the fleet.
- Document numerical validation, shape coverage, fallback, and the profiler evidence that reduced bytes or instructions caused the normalization-kernel gain.
Conclusion: The local 12 percent gain predicts only about a 1 percent direct latency improvement when the kernel contributes 9 percent of critical time. Deployment value must be calculated from the measured application change, workload frequency, utilization, and required headroom.
Common errors
- Burying the workload and constraints. Readers cannot judge applicability without knowing what the number represents.
- Showing only a faster timing table. A causal report connects the code change, physical mechanism, named kernel or subsystem timing, and complete request or training-step effect.
- Omitting regressions and unsupported shapes. Production decisions need the loss surface and fallback, not only the best case.
Reference summary
State the workload, objective, baseline operating point, limiting operation, predicted mechanism, and code change. Then present correctness evidence, hardware-mechanism evidence, controlled timing for the named kernel or subsystem, and end-to-end application evidence as one connected argument.
Document regressions, unsupported shapes, architecture and toolchain sensitivity, resource costs, fallback, and reproduction artifacts. Preserve concise failed hypotheses when they prevent repeated investigation.
value = workload frequency × saved time × deployment scaleKnowledge check
What evidence distinguishes ‘the kernel became faster’ from ‘the product became more efficient’?
Show the answer guide
- A kernel result establishes a local duration and mechanism only for the declared shapes and environment; a product claim must show the change on the request or training-step critical path.
- The product comparison must preserve correctness or quality, workload distribution, latency or throughput constraints, reliability conditions, and fully loaded resource cost.
- A complete report connects the source change to diagnostic evidence, then reconciles the measured end-to-end interval and cost rather than extrapolating the isolated speedup.
Write a bounded performance report
Choose one measured kernel change and produce a report that separates kernel, step or request, correctness, and resource outcomes. Include one rejected alternative explanation and one limitation.
Evidence to keep: Versioned report, source and environment manifest, raw samples, correctness evidence, profiler artifacts, quantitative prediction, end-to-end reconciliation, and reproducible commands.
Part III applies the single-GPU execution and measurement models to parallel primitives, matrix kernels, attention, and compiler-integrated operators.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.