Reading a kernel profile
Form a kernel-limitation hypothesis from physical work and test the predicted traffic, pipeline, readiness, and timing changes
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
A kernel profile contains many counters, and the interface often emphasizes large percentages or warnings. Reading the most prominent number first can lead to an unsupported optimization. A stall metric can describe why a sampled warp was unable to issue, but it does not state that removing that stall would reduce kernel time. Low occupancy can coexist with full use of the limiting pipeline.
Begin with the work the kernel completed. Confirm the input shape and launch instance, measure its duration separately, and estimate operations and bytes. Completed operations, transferred bytes, and elapsed seconds place the kernel against relevant compute and memory ceilings. Only after the likely limiting resource is identified should scheduler and source metrics be used to explain why the kernel approaches or falls below that ceiling.
The result is a causal chain rather than a single counter: this invocation moved a stated number of bytes, the access pattern caused a measured transaction count, the kernel approached an attainable bandwidth, and removing an intermediate reduced both traffic and time. Several consistent observations are stronger than one unusually high percentage.
Confirm the invocation and calculate physical work
A kernel name can represent many shapes. Verify that the profiled launch is frequent and important for the target workload rather than a rare initialization or tail case. Record grid dimensions, block dimensions, dynamic shared memory, registers, and input sizes. If the application launches the same symbol hundreds of times, distinguish instances by duration and shape.
Estimate useful arithmetic and required bytes from the algorithm. Then compare requested and transferred bytes from the profiler. A large difference suggests overfetch, redundant traffic, spills, or other effects absent from the logical model. Include integer address operations, conversions, special functions, and control instructions when they consume important pipelines even though a floating-point operation (FLOP) count omits them.
Use dedicated CUDA-event timing or another low-perturbation method for final duration. Nsight Compute may replay a kernel to collect counter sets, and collection can alter cache state or elapsed time. Profiler duration is valuable for context, but heavily instrumented time should not be the final speed claim.
Compare achieved work with attainable ceilings
A roofline compares achieved arithmetic rate with arithmetic intensity, the applicable arithmetic ceiling, and a named data-transfer ceiling. For device-memory analysis, use bytes moved between HBM and the on-chip L2 cache per second under comparable conditions rather than an unqualified marketing maximum.
A point near the sloped roof set by bytes transferred between high-bandwidth memory (HBM) and the on-chip L2 cache supports a device-memory-bandwidth limitation. A point below every visible roof does not mean that the roofline failed. The kernel may have insufficient parallelism, dependency chains, scattered transactions, on-chip bottlenecks, or instruction overhead. Extend the model to the relevant cache or pipeline and inspect readiness.
The roofline classifies the current operating point. The classification does not prove that a source edit will help. A bandwidth-limited kernel can improve through fewer HBM bytes, more useful work per HBM byte, or fusion; increasing arithmetic throughput alone will not raise the HBM-to-L2 transfer roof. Predict which measured quantity the proposed change will alter.
Use scheduler and stall data to explain unused capacity
Distinguish resident, active, eligible, and issued warps. Resident warps have allocated state. Active warps have not completed. Eligible warps have a next instruction whose operands and execution conditions permit issue. A scheduler can have many active warps and still find none eligible because they wait on similar dependencies.
Memory-dependency samples can result from long latency, too few outstanding requests, poor locality, or a saturated path. Execution-dependency samples can result from a serial accumulator chain. Barrier samples can indicate that some warps finish a phase earlier than others. The stall label identifies a state; source correlation, instruction mix, and other counters identify the cause.
NVIDIA's profiling guide specifically advises focusing on stalls when schedulers fail to issue. A large stall category can coexist with strong issue and throughput because sampling observes warp states while other warps make progress. Compare before and after with the same collection setup, and require the predicted traffic, issue, or dependency change to accompany the timing improvement.
Worked example: A low-occupancy kernel at the bandwidth roof
A streaming transformation reports 37% achieved occupancy, which looks poor, but its measured HBM bandwidth is 94% of a sustainable copy roof.
- Count its low arithmetic intensity and verify requested-to-transferred efficiency. The kernel performs little work per byte and transactions are already efficient.
- Because HBM-to-L2 traffic is near the sustainable copy benchmark in bytes per second, more resident warps are unlikely to increase completed transfer bytes per second. Additional warps may only queue more requests at the GPU memory controllers.
- Estimate traffic-saving alternatives: fuse the consumer, avoid an intermediate write, compress values, or change the algorithm. A block-size change aimed solely at occupancy cannot move the HBM ceiling.
- After fusion, time the larger region and verify reduced total bytes. Expect the next bottleneck to move, perhaps to arithmetic or another unfused tensor.
Conclusion: Low occupancy was real but not limiting. The roofline and traffic evidence prevented a plausible, low-value tuning project.
Common errors
- Optimizing the highest stall percentage directly. Stalls describe why a warp could not issue, not what system change yields value.
- Comparing metrics from different shapes or clock states as though all else were equal. The benchmark contract still governs profiler work.
- Using achieved occupancy as a universal health score. Enough ready warps to saturate the limiting path is enough.
Reference summary
Confirm the exact kernel invocation by its tensor shape, grid, block, duration, operation count, and requested and transferred bytes. Compare achieved floating-point operations per second with the applicable arithmetic ceiling. Compare bytes transferred between GPU high-bandwidth memory and the on-chip L2 cache per second with a sustainable bandwidth measured by a representative copy benchmark.
If the kernel is far below the relevant roof, distinguish resident, active, eligible, and issued warps. Then use source-correlated scheduler and instruction metrics to explain memory waits, dependency chains, barriers, imbalance, or other limits.
- Compare achieved values with a measured or tool-provided roof, not marketing peak alone.
- Check instruction mix: integer address work, conversions, special functions, and control instructions consume issue slots too.
- Use baselines to compare the same metric before and after a controlled change.
- Remember that metric collection may replay kernels and perturb timing; use a dedicated timing measurement for final speed.
Knowledge check
A kernel has low achieved occupancy but reaches its measured device-memory bandwidth roof. Should occupancy be your first optimization target? Explain.
Show the answer guide
- No. Reaching a measured HBM-to-cache bandwidth roof supports a device-memory-traffic limit for the current implementation even when achieved occupancy is low.
- Raising occupancy does not lift that bandwidth roof and can worsen performance if it reduces reuse or increases spills and traffic.
- The first experiment should reduce requested or transferred HBM bytes or improve transaction efficiency, while checking that elapsed time changes with the traffic prediction.
Classify one kernel with counters
Profile one CUDA kernel at a declared shape. Calculate its HBM arithmetic intensity and compare achieved HBM bandwidth and arithmetic rate with measured or justified ceilings before choosing one controlled optimization.
Evidence to keep: Saved profiler report, operation and byte calculation, roofline position, occupancy and eligible-warp data, predicted intervention, raw before/after timing, and correctness results.
Use compiler output, PTX, and native SASS when kernel metrics cannot explain instruction selection, memory traffic, or resource use.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.