Chapter 1: Performance Models and Metrics

Units, work, and rates

Estimate arithmetic work and HBM-to-cache bytes for a declared tensor operation before profiling

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

Basic algebra and multiplication

A developer runs one 4,096 × 4,096 matrix multiplication while NVIDIA Nsight Systems records CPU submission and GPU activity. The profiler produces a timestamped event record called a profiler trace. A timeline view draws each recorded event between its start time and end time; the matrix-multiplication interval lasts 1.8 milliseconds. The source operation is `C = A × B`; the GPU program reads two input matrices, performs multiplications and additions, and writes one output matrix. Before inspecting instruction-level details, the developer can calculate two amounts: the number of arithmetic operations required by the matrix shape and the minimum number of bytes that must move from GPU high-bandwidth memory (HBM) into the on-chip cache hierarchy.

The units reveal which machine rate can explain each amount. Dividing 137.4 billion floating-point operations by a measured rate in floating-point operations per second produces seconds. Dividing 100.7 megabytes by a measured HBM transfer rate in bytes per second also produces seconds. Dividing bytes by floating-point operations per second produces no time and therefore cannot explain the 1.8-millisecond observation without another relationship.

AI performance reports mix decimal gigabytes, binary gibibytes, bits, bytes, instructions, and floating-point-operation conventions. The worked investigation keeps every unit visible and compares the calculated lower bounds with the CUDA-event timing and Nsight Compute byte counters. A disagreement identifies a concrete missing assumption, such as repeated input loads, an unsupported matrix instruction, or a rate that the tested shape cannot sustain.

Amounts, rates, and time

An amount of work and a work rate are different quantities. A matrix operation can require 137.4 GFLOP. A device can sustain 100 TFLOP/s for that operation. Dividing work by rate gives 0.001374 seconds, or 1.374 milliseconds. Keep the powers of ten visible: giga is 10^9 and tera is 10^12 in these rate conventions.

Apply the same relationship to data movement. If an implementation transfers 100.7 MB from HBM into the GPU cache hierarchy and sustains 2.5 TB/s, the transfer requires at least about 40 microseconds. The 40-microsecond value is a lower bound because the calculation assumes that the measured bandwidth is continuously available and that no additional bytes travel between HBM and the cache hierarchy.

Logical work and physical work

Logical work follows from the algorithm. For C[M,N] = A[M,K] × B[K,N], each of the M×N outputs forms a dot product of length K. Under the common convention, each of K multiplications and K additions contributes approximately two floating-point operations, which gives 2MKN FLOP.

Physical work follows from the implementation. A matrix kernel can pad dimensions, calculate values that a mask later discards, recompute an intermediate, or execute conversion instructions. The kernel can also load the same logical value many times. Two implementations can therefore produce the same output and have the same logical operation count while placing very different demands on instruction pipelines and memory.

Compulsory traffic and actual traffic

A compulsory-traffic estimate counts the transfers required by an ideal implementation across one named source-and-destination pair. For the first estimate, count bytes transferred between GPU high-bandwidth memory (HBM) and the on-chip cache hierarchy. A simple GEMM must read A and B from HBM and eventually write C back to HBM. If each matrix is 4096×4096 in FP16, those two reads and one write total about 100.7 MB. The estimate assumes perfect on-chip reuse after each input is fetched from HBM.

Actual traffic can be higher. A value can be evicted and fetched again, a partial output can be written and reread, or poorly aligned requests can transfer unused sectors. Hardware counters can estimate actual traffic. Compare those counters with the compulsory floor to determine whether the implementation is realizing the assumed reuse.

Name both ends of every data transfer

Data moves through several levels. A value can travel from HBM to L2, from L2 to a core-local cache or shared memory, and from there into registers. Each level has a different capacity and rate. The phrase ‘the kernel moves 100 MB’ is incomplete unless it identifies the measured source and destination.

The same kernel can reuse each value well enough to minimize HBM-to-cache traffic and still request shared-memory data faster than the on-chip shared-memory hardware can supply it. Begin with HBM-to-cache transfers when the tensors reside in HBM and Nsight Compute reports high device-memory throughput. Add the interface between shared memory and registers when HBM traffic is low but shared-memory counters or warp-stall evidence indicate an on-chip transfer limit. Later chapters repeat the calculation for each named interface.

Combine bounds according to overlap

If arithmetic and HBM transfer cannot overlap, add their lower-bound times. If they overlap completely, the larger time bounds the CUDA kernel's execution. Most real kernels lie between these cases because pipelines overlap portions of load, execution, and store work. The simple maximum still provides a useful first approximation.

Do not automatically add communication time to computation time in a distributed step. First draw the dependency timeline. Communication that runs concurrently with independent computation contributes only its exposed portion to the critical path. Units verify the arithmetic; dependencies determine how individual time terms combine.

Worked example: A lower bound for square GEMM

Calculate a first-order lower bound for C = A × B, where A, B, and C are 4096×4096 matrices stored in FP16. Products accumulate in FP32, and the final output is converted to FP16. Ignore bias and other epilogue operations.

Calculate logical work: 2 × 4096 × 4096 × 4096 = 137,438,953,472 FLOP, or about 137.4 GFLOP. No time appears yet because this is an amount of work.
Calculate one matrix size: 4096² elements × 2 bytes per FP16 element = 33,554,432 bytes. Reading A and B once and writing C once gives 100,663,296 bytes, or about 100.7 MB in decimal units.
Use a sustained compute rate of 100 TFLOP/s for the relevant matrix instruction path. The compute time bound is 137.4×10^9 FLOP divided by 100×10^12 FLOP/s, which is about 1.37 milliseconds.
Use sustained HBM bandwidth of 2.5 TB/s. The traffic time bound is 100.7×10^6 bytes divided by 2.5×10^12 bytes/s, which is about 40 microseconds.
Under a complete-overlap approximation, the larger bound is 1.37 milliseconds. A well-tiled implementation should therefore reach a compute-related limit before a compulsory-HBM-traffic limit. This statement does not yet guarantee that the implementation uses the required matrix instructions or achieves the assumed input reuse.
Compare actual HBM traffic with 100.7 MB. A naive kernel that repeatedly reloads input values can transfer far more data and become HBM-limited. Tiling changes this physical traffic without changing the 137.4 GFLOP of logical matrix work.

Conclusion: The model predicts the limiting side for an implementation that approaches compulsory HBM traffic and sustains the assumed compute rate. Measurements must verify both assumptions before the classification is accepted.

Common errors

Dropping units during intermediate steps. This makes a missing conversion between milliseconds and seconds or bytes and bits difficult to detect.
Using a specified peak rate as the sustained rate for an unsupported instruction mix or inconvenient shape. A data-sheet ceiling is not an observation.
Counting each tensor once when the implementation rereads it. Compulsory traffic and actual traffic answer different questions.
Comparing operation counts that use different fused-multiply-add conventions. State the counting convention beside the result.

Reference summary

Dimensional analysis preserves the meaning of a calculation. A quantity such as 200 has no performance meaning until it carries a unit. Two hundred GFLOP is an amount of arithmetic work. Two hundred GFLOP/s is a rate. Dividing the first by the second produces seconds. If the units do not cancel to the quantity that you intend to calculate, the model is incomplete or inconsistent.

operation time ≥ work / sustained rate

Apply the relation separately to floating-point operations completed by the selected GPU instruction path, bytes transferred from HBM into the GPU cache hierarchy, and bytes transferred over a named network link.

Distinguish logical work from physical work. The expression C[M,N] = A[M,K] × B[K,N] requires approximately 2MKN floating-point operations under the usual counting convention. The implementation determines how often it loads each value, whether it pads dimensions, and whether it recomputes intermediate values. Logical work is fixed by the algorithm; physical instructions and traffic depend on the implementation.

text

M = N = K = 4096
logical work = 2 × M × N × K
             ≈ 137.4 GFLOP
compulsory HBM bytes for FP16 A, B, and C
             = (M×K + K×N + M×N) × 2 bytes
             ≈ 100.7 MB

The byte count is an optimistic floor. The calculation assumes that A and B each travel from HBM into the on-chip cache hierarchy once and that C travels back to HBM once.

Knowledge check

A CUDA kernel performs 200 billion floating-point operations and Nsight Compute estimates 80 GB transferred from HBM into the cache hierarchy. What is the HBM arithmetic intensity, including units, and which measured machine rate is still needed to predict transfer time?

Show the answer guide

Arithmetic intensity is arithmetic work divided by traffic: 200 billion FLOP / 80 billion bytes = 2.5 FLOP per HBM byte.
The byte boundary must remain explicit: the 80 GB estimate describes transfers from HBM into the GPU cache hierarchy, not all cache or register traffic.
Predicting transfer time also requires the sustained HBM-to-cache bandwidth for the same GPU, access pattern, and operating state, expressed in GB/s or TB/s rather than the vendor peak alone.
With a measured bandwidth B GB/s, the transfer-only lower bound is 80 GB / B GB/s. Compute time and other dependencies can make observed kernel time larger.

Practice

Model a vector-add transfer

For C = A + B with 2^24 FP32 elements, calculate logical additions, compulsory input and output bytes, and arithmetic intensity. Run at least 30 timed iterations with a local array library on one declared CPU or GPU, using preallocated arrays and correct completion timing.

Evidence to keep: Keep the unit-carrying work and byte calculation, raw timing samples, median time, achieved GB/s, output checksum or reference test, and a note comparing achieved bandwidth with the declared machine-memory rate.

Work and rate models estimate the time of one named GPU program, memory transfer, or communication operation. The next section calculates how much shortening that recorded operation can affect the complete request or training step.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.

Amdahl, Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities↗
Little, A Proof for the Queuing Formula: L = λW↗
Roofline: An Insightful Visual Performance Model↗
NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.
Nsight Compute Profiling Guide↗
MLPerf Training Benchmark↗