IIntroduction to AI Systems Performance EngineeringTechnical textbook
Chapter 2: Performance Measurement and Benchmarking
2.3

Correctness validation

Define and test the accepted numerical and model behavior before timing an optimized implementation

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

A new fused softmax CUDA kernel runs 35% faster than a composition of existing PyTorch operations. A test program supplies a tensor containing 32 attention rows of 1,024 values, runs both implementations, and saves the two output arrays. Most corresponding values differ only in the fourth decimal place, but one fully masked row contains `NaN` values in the new output. The timing result cannot be accepted until the implementation's allowed inputs and required masked-row behavior are explicit.

Several mechanisms can change the saved numbers. Parallel GPU threads can add values in a different order, a fused implementation can remove one intermediate rounding step, and a narrower floating-point format can discard low-order bits. Mathematically equivalent expressions over real numbers can therefore produce different finite binary results. A profiler timing cannot distinguish an acceptable rounding difference from an incorrect mask or an out-of-bounds memory access.

A correctness contract states the supported shapes, strides, formats, masks, exceptional values, and numerical limits before selecting the faster implementation. The empirical workflow compares the optimized output with a separately written reference, checks properties such as finite nonnegative probabilities and row sums, sweeps GPU tile boundaries, runs memory-safety tools, and finally measures model-level quality. Each check catches a different failure that a single average error can hide.

Define the reference and input domain

A reference is the implementation or specification against which the optimized result is compared. For a manageable operator, use a simple implementation with FP32 or FP64 intermediates when that representation is more accurate for the tested values. The reference should prioritize clarity and independent construction over speed.

Define which inputs the optimized operation supports. Include shape ranges, strides, alignment, data formats, mask semantics, aliasing, and behavior for NaN or infinity. A fast path that supports only contiguous multiples of a tile size can be valid when the dispatcher checks those conditions and a correct fallback handles everything else.

Choose error measures that match scale

Absolute error measures |x−y| and is useful near zero, where relative error can become unstable. Relative error divides by a reference magnitude and is useful when acceptable error grows with value scale. Many operator tests combine both: |x−y| ≤ atol + rtol×|y|.

The tolerance must follow from the data format and algorithm. A long reduction can accumulate more rounding error than one elementwise multiplication. A normalization with very small variance can amplify small differences. One global tolerance copied from a different operator does not express these conditions.

Test mathematical properties

Elementwise comparison is necessary but not always sufficient. Softmax outputs should be finite and nonnegative, and each valid row should sum approximately to one. Masked positions must obey the exact mask contract. A normalization should satisfy its expected mean or root-mean-square property within the accepted error.

Property checks often identify systematic failures that an aggregate error statistic obscures. A kernel can have a small mean error while producing NaN for one fully masked row. Test such boundary conditions directly.

Use adversarial shapes and values

Specialized implementations change paths at tile boundaries, alignment limits, and size thresholds. Test dimensions immediately below, at, and above those boundaries. Include minimum legal sizes, empty inputs when supported, long reductions, non-contiguous views, and shapes that create partial tiles.

Random normal values do not reliably produce overflow, underflow, cancellation, or nearly equal logits. Construct inputs with extreme magnitudes, tiny differences, repeated values, and mixed signs. Add NaN and infinity when the API defines their handling. Keep these tests deterministic enough to reproduce a failure.

Validate operator, model, and system behavior

Operator validation isolates numerical and memory-safety behavior. Model validation checks whether accumulated differences alter loss, perplexity, convergence, task scores, or generated text under controlled decoding. System validation checks error handling, retries, protocol output, and any determinism requirement.

Operator, model, and system checks cannot replace one another. A kernel can pass operator tolerances and still change a top-k routing decision because the downstream operation is discontinuous. A model-level score can remain stable while one rare operator shape writes out of bounds. Run all levels in proportion to the optimization's risk.

Separate determinism from correctness

A nondeterministic parallel reduction can produce slightly different accepted results across runs because the reduction order changes. Run-to-run variation can be correct when the product permits the measured range. A deterministic implementation can produce the same incorrect result every time.

Test determinism only when reproducibility, debugging, auditing, or application behavior requires it. Record any performance cost of enforcing a fixed order. Keep numerical acceptance and repeatability as separate requirements.

Worked example: Validating a fused softmax path

A new attention kernel fuses score scaling, causal masking, and row-wise softmax. The new kernel uses reduced-precision operands and is substantially faster than the existing composition.

  1. Write a clear FP64 or carefully implemented FP32 reference for manageable shapes. Apply scaling, add the exact mask representation, subtract the row maximum, exponentiate, sum, and normalize in separate steps.
  2. Generate ordinary logits and constructed adverse cases: very large positive and negative values, nearly equal values, one dominant value, long rows, rows with one valid element, and fully masked rows if the API permits them.
  3. Compare outputs using justified absolute and relative tolerances. Also check that outputs are finite and nonnegative, valid rows sum near one, masked positions are exactly or effectively zero according to the contract, and fully masked rows follow the declared behavior.
  4. Sweep sequence lengths below, at, and above tile boundaries. Include supported stride patterns and alignment cases. Run a memory checker and race detector on a reduced test set before accepting timing results.
  5. Validate backward gradients separately because the backward pass uses different reductions and saved values. Compare against automatic differentiation through the reference composition for supported shapes.
  6. Run model-level evaluation at the target precision and decoding settings. If output tokens diverge, evaluate task quality and error cases against the accepted limit. Do not infer model equivalence only from average probability error.

Conclusion: The optimization is acceptable only for the shape and input domain covered by its dispatcher and validation. The report should state numerical differences, property checks, model-level results, and memory-safety evidence beside the speedup.

Common errors

  • Using one global absolute tolerance across values with very different scales. Combine absolute and relative criteria according to the numerical regime.
  • Testing only random normal inputs. Random sampling rarely covers tile boundaries, extreme values, cancellation, and mask edge cases.
  • Evaluating only forward output. Backward computation has separate reductions, saved-state requirements, and potential memory errors.
  • Confusing determinism with correctness. A nondeterministic implementation can remain within the accepted contract, and a repeatable result can still be wrong.

Reference summary

Compiler transformations, fused operations, parallel reductions, approximate functions, and reduced-precision formats can change numerical results. Define acceptable behavior at three levels: the individual operator, the complete model, and the production system. A small elementwise error does not by itself establish that model quality or generated actions remain acceptable.

  • Reference: compare with a trusted implementation, preferably with wider arithmetic for manageable test cases.
  • Input coverage: test ordinary inputs, extreme magnitudes, odd sizes, empty legal inputs, maximum lengths, non-contiguous strides, and supported aliasing.
  • Properties: test invariants such as finite outputs, probability sums, mask behavior, monotonicity, and conservation properties where applicable.
  • Model behavior: measure loss, perplexity, task quality, convergence, or decoding behavior under the same settings used by the target system.
  • Safety: run memory, race, and undefined-behavior checks before accepting a faster low-level implementation.

Knowledge check

Why can one random tensor with a relative tolerance of 0.001 miss errors in fully masked rows, partial GPU tiles, large logits, backward gradients, and model-level token selection?

Show the answer guide
  • A random tensor may contain no fully masked row, while that row needs a separately defined result that avoids division by zero, NaN propagation, or probability assigned to a masked position.
  • Random dimensions often miss sizes immediately below and above a GPU tile boundary. Partial tiles exercise masks, bounds checks, and stores that a divisible size never reaches.
  • Ordinary random magnitudes rarely contain logits large enough to overflow an unstable exponential or expose cancellation. Explicit values such as 1000, 999, and -1000 test the stability mechanism.
  • A relative tolerance of 0.001 is weak or undefined near zero and can accept large absolute errors at large magnitudes. Tests need declared absolute and relative tolerances plus operation-specific invariants.
  • Forward output comparison does not test backward gradients, races, out-of-bounds access, non-contiguous layouts, or aliasing. Backward, memory-safety, and layout paths require gradient checks and safety or layout-specific cases.
  • Small elementwise differences can change an argmax, sampled token, tool call, or model-quality result. Operator tests therefore do not replace model-level behavioral evaluation.
Practice

Build an adversarial softmax table

Compare a direct exponential softmax with a subtract-the-maximum reference on rows `[1,2,3]`, `[1000,999,997]`, a fully masked row, and lengths immediately below and above a chosen block size. State the expected behavior for the fully masked row before running the test.

Evidence to keep: Keep the exact input table, candidate and reference outputs, finite-value and row-sum checks, absolute and relative error calculations, failure output for the unstable large-logit case, and the declared masked-row rule.

Correctness limits depend on the data format and operation structure. Chapter 3 develops these properties for floating-point arithmetic and transformer workloads.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.

  1. CUDA C++ Best Practices Guide↗
  2. NVIDIA Nsight Systems User Guide↗

    Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.

  3. PyTorch Profiler documentation↗

    Official reference for recording operator activity, CPU and accelerator events, shapes, stacks, and exported traces.

  4. MLPerf Inference Rules↗
  5. MLPerf Training Benchmark↗