Roofline performance model
Classify a kernel as bandwidth- or compute-limited
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
NVIDIA Nsight Compute reports that a normalization CUDA kernel completed 240 billion floating-point operations and transferred 20 gigabytes from GPU high-bandwidth memory (HBM) into the on-chip cache hierarchy during the measured repetitions. Dividing the operation count by the transferred bytes gives 12 floating-point operations per HBM byte. The report also shows that the kernel reached about 2.5 terabytes per second of HBM traffic but only 30 trillion floating-point operations per second. A separate microbenchmark using the same scalar and vector arithmetic instructions reaches a higher arithmetic rate, so the comparison does not incorrectly use the GPU's matrix-multiplication ceiling for a normalization kernel.
The observation has a physical explanation. GPU arithmetic instructions cannot use a tensor value until memory hardware delivers the value to an on-chip location accessible by the executing threads. A program that performs only 12 operations for every byte fetched can exhaust the measured HBM transfer rate before the arithmetic pipelines receive enough operands to reach their own rate limit.
The roofline model compares those two named limits. Arithmetic intensity is operations divided by bytes transferred across a specified source-and-destination pair; this lesson initially uses transfers from HBM into the chip's cache hierarchy. Multiplying HBM bytes per second by operations per byte gives the maximum arithmetic rate supported by that traffic. The resulting plot predicts whether a traffic-reduction experiment or an arithmetic-execution experiment is the better first test, while profiler counters verify whether the prediction describes the actual kernel.
Derive arithmetic intensity
Arithmetic intensity is logical or executed arithmetic work divided by bytes transferred across a named boundary. If a kernel performs 240 GFLOP and transfers 20 GB from HBM, its HBM arithmetic intensity is 12 FLOP/byte. State whether the numerator uses algorithmic work or executed instructions and whether the denominator is a compulsory estimate or measured traffic.
Intensity changes when the implementation changes physical traffic. Fusion can keep an intermediate on chip instead of writing it to HBM. Tiling can reuse a loaded value for several operations. Recomputation can replace some stored bytes with additional arithmetic. Each change moves the kernel's point horizontally in an HBM roofline model.
Construct the two ceilings
The compute ceiling is the relevant sustained or peak arithmetic rate. The bandwidth ceiling at intensity I is bandwidth × I. The product has units bytes/s × FLOP/byte = FLOP/s. The attainable rate cannot exceed either ceiling, so the roofline bound is their minimum.
A logarithmic axis gives equal visual distance to equal multiplication factors rather than equal additions. Values 1, 10, and 100 therefore occupy evenly spaced positions because each step multiplies by ten. Roofline plots use logarithmic axes so workloads whose rates differ by orders of magnitude remain visible on one chart. The bandwidth ceiling rises diagonally because doubling operations per byte doubles the arithmetic rate supported by the same transfer rate. The compute ceiling is horizontal because more reuse cannot exceed the selected arithmetic capacity. Their intersection is the ridge point.
Calculate and interpret the ridge point
Divide the compute ceiling by bandwidth to obtain the ridge point. A device with 120 TFLOP/s and 3 TB/s has a ridge point of 40 FLOP/byte. Below 40 FLOP/byte, the HBM bandwidth product is lower than the compute ceiling. Above 40, the compute ceiling is lower in this first-order model.
The classification belongs to the chosen machine, data format, implementation, and source-and-destination pair for measured bytes. A lower-precision matrix path can raise the compute ceiling without changing HBM bandwidth, moving the ridge point to the right. The same algorithm can therefore be compute-limited in FP32 and bandwidth-limited in a much faster low-precision path unless reuse also improves.
Use hierarchical rooflines
HBM is only one boundary. A tiled GEMM can read each input block once from HBM and reuse it many times on chip, giving high HBM intensity. The same implementation repeatedly reads operands from shared memory into registers and can approach a shared-memory bandwidth limit.
A hierarchical roofline repeats the calculation for HBM, L2, shared memory, and selected instruction paths. Each level uses traffic measured across that boundary and its corresponding rate. Begin with one level to establish the method, then add levels when the measured point remains far below the first roof.
Recognize what the roofline cannot diagnose
A point far below both basic ceilings indicates unused theoretical capacity, but it does not identify the cause. The kernel can have insufficient parallelism, long dependency chains, divergent control flow, bank conflicts, inefficient instructions, small launch size, or a limit at another memory level.
A point near the HBM roof supports a traffic-focused investigation. Check actual transactions and useful bytes before changing the algorithm. Poor access efficiency can saturate transferred bandwidth while delivering fewer requested bytes, and reducing unneeded transactions can then improve useful performance without changing the algorithmic tensor sizes.
attainable ≤ min(peak, bandwidth × intensity)Move arithmetic intensity across the ridge. The plotted point follows the sloped bandwidth roof, then stops at the flat compute roof.
Worked example: Classifying two kernels on the same GPU
A GPU sustains 120 TFLOP/s for the relevant arithmetic and 3 TB/s from HBM under comparable conditions. Compare a normalization kernel with measured HBM intensity of 12 FLOP/byte and a tiled GEMM at 90 FLOP/byte.
- Calculate the ridge point: 120 TFLOP/s divided by 3 TB/s = 40 FLOP/byte. The units reduce correctly because tera appears in both numerator and denominator.
- For normalization, calculate the HBM roof: 3 TB/s × 12 FLOP/byte = 36 TFLOP/s. The 36-TFLOP/s traffic roof is lower than 120 TFLOP/s, so the model classifies the kernel on the HBM-bandwidth side.
- Suppose normalization achieves 30 TFLOP/s. Divide by intensity to obtain 2.5 TB/s of achieved HBM bandwidth. The measured 2.5-TB/s rate is close enough to the assumed 3-TB/s ceiling to justify investigating traffic reduction and transaction efficiency first.
- For GEMM, the bandwidth product is 3 TB/s × 90 FLOP/byte = 270 TFLOP/s. The 120-TFLOP/s compute ceiling is lower, so the model classifies the implementation on the compute side.
- Suppose GEMM achieves only 55 TFLOP/s. HBM bandwidth is not the first explanation under this model. Check whether dimensions qualify for the intended matrix instructions, whether tiles provide enough parallel work, and whether dependencies, register pressure, or another memory level limits issue.
- Recalculate both points when datatype or implementation changes. A fused normalization can transfer fewer HBM bytes, while a lower-precision GEMM can raise the compute ceiling. Either change can move the ridge point or the kernel's intensity.
Conclusion: The model identifies the first resource investigation for each kernel. Detailed counters and code analysis are still required to determine the specific implementation change.
Common errors
- Using compulsory bytes as if they were measured traffic. An implementation with repeated or inefficient transfers can occupy a different region.
- Mixing decimal and binary prefixes or bytes and bits. Unit inconsistency changes the ridge point and the plotted position.
- Using a compute ceiling for a datatype or instruction path that the kernel does not execute. The ceiling must correspond to the actual operation class.
- Treating all points below a roof as one problem. The roof quantifies headroom; lower-level measurements identify the mechanism.
Reference summary
attainable compute rate ≤ min(compute ceiling, bandwidth × arithmetic intensity)The roofline model combines two time bounds. Arithmetic work divided by the selected GPU compute rate gives one lower bound. Bytes transferred between HBM and the cache hierarchy divided by sustained HBM bandwidth gives another. A CUDA kernel below the ridge point performs too little arithmetic per HBM byte to reach the compute ceiling. Tiling, fusion, and reuse can move the kernel rightward by preventing repeated HBM transfers.
A plotted point to the right of the ridge point reaches the selected compute ceiling before the HBM ceiling in this first-order model. The measured kernel can still run far below that ceiling because its shapes do not use the intended instructions, its dependency chain limits issue, or another storage level supplies data too slowly. The roofline selects the first resource investigation; lower-level evidence identifies the exact cause.
Knowledge check
A GPU sustains 120 trillion floating-point operations per second on the tested path and 3 TB/s between HBM and the cache hierarchy. Calculate the ridge point with units. Which side contains a CUDA kernel measured at 20 floating-point operations per HBM byte?
Show the answer guide
- The ridge point is peak or sustained compute rate divided by sustained HBM bandwidth: 120 TFLOP/s / 3 TB/s = 40 FLOP per HBM byte.
- A kernel at 20 FLOP per HBM byte lies below and to the memory-limited side of the 40-FLOP/byte ridge point.
- Its bandwidth roof is 20 FLOP/byte × 3 TB/s = 60 TFLOP/s, below the 120-TFLOP/s compute roof, so the simplified model predicts an HBM-bandwidth limit.
- The conclusion applies only if the FLOP count matches the executed instruction path and the byte count measures the declared HBM-to-cache boundary under the tested workload.
Place two kernels on one roofline
Use a machine roof of 120 TFLOP/s and 3 TB/s. Calculate arithmetic intensity and predicted performance for a 20-FLOP/byte kernel and an 80-FLOP/byte kernel, then calculate how each prediction changes if sustained bandwidth falls to 2 TB/s while compute remains fixed.
Evidence to keep: Keep the ridge-point calculations for both bandwidth values, a two-line roofline plot with all four kernel points, a table of predicted TFLOP/s, and a conclusion naming the active roof in every case.
The four models produce testable limits. Chapter 2 explains how to build measurements that can validate or reject them.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.
- Amdahl, Validity of the Single Processor Approach to Achieving Large Scale Computing Capabilities↗
- Little, A Proof for the Queuing Formula: L = λW↗
- Roofline: An Insightful Visual Performance Model↗
- NVIDIA Nsight Systems User Guide↗
Official reference for capturing and reading CPU, CUDA API, GPU workload, and communication timelines.
- Nsight Compute Profiling Guide↗
- MLPerf Training Benchmark↗