Occupancy and latency hiding
Calculate the active residency limit and use profiler evidence to decide whether more resident warps would reduce elapsed time
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 4 register, shared-memory, warp, and streaming-multiprocessor model
- Previous section's distinction between block shape and grid size
A normalization CUDA kernel processes an activation tensor with shape [2,048 tokens, 4,096 features] during prompt processing for the recurring 7-billion-parameter inference model. Two compiled variants differ only in tile size. Nsight Compute reports 50 percent achieved occupancy for the faster variant and 100 percent for the slower variant, so occupancy cannot be interpreted as a performance grade.
Nsight Compute obtains compiled resource counts and samples hardware behavior while the selected kernel launch runs. The report lists threads per block, registers per thread, shared memory per block, theoretical occupancy, achieved occupancy, eligible warps, memory traffic, and elapsed cycles. The learner should also time the same invocation with CUDA events because collection can replay or perturb a kernel.
When a block is assigned to a streaming multiprocessor (SM), the SM reserves registers for its threads and shared memory for the block. Register and shared-memory allocations remain reserved until the block completes. The SM also has architectural limits on resident blocks, warps, and threads. The first limit reached determines how many blocks can reside at one time.
Occupancy is the number of resident warps divided by the architecture's maximum resident warps. The percentage answers how much warp state is present. Occupancy does not report how many warps have ready operands, which execution pipeline limits the kernel, or how much unnecessary work the kernel performs. Registers and shared memory can purchase reuse and independent instructions, so removing those resources to raise occupancy can make the complete kernel slower.
The reason to care about occupancy is latency hiding. When one warp waits, another resident and eligible warp may issue. The reason not to maximize it blindly is that registers and shared memory are also used for reuse and instruction-level parallelism. Reducing those resources can increase traffic or create spills even while it increases the occupancy percentage.
Calculate resident blocks from each resource
For each candidate block, calculate a register limit, a shared-memory limit, a thread or warp limit, and a maximum-block limit. The minimum is the predicted number of resident blocks. For example, if registers permit four blocks, shared memory permits three, and the thread limit permits two, then only two blocks can reside. The unused shared memory in this example cannot compensate for the thread limit.
Allocation is quantized. Registers may be assigned in architecture-specific units, and blocks cannot be divided across SMs. NVIDIA documents cases in which a change from 64 to 65 registers per thread removes an entire resident block. Shared-memory stage counts can cause similar thresholds. An allocation threshold explains why a small source edit can cause a large, discontinuous occupancy change.
Theoretical occupancy uses declared and compiled resource requirements. Achieved occupancy is measured while the kernel runs and can be lower because of short execution, tail waves, or other effects. Neither value reports whether resident warps are eligible to issue.
Latency can be covered across warps or within a warp
Thread-level parallelism covers latency by keeping more warps resident. Instruction-level parallelism covers latency by giving one warp several independent operations. Four independent accumulators can allow arithmetic to proceed while an earlier accumulator result is pending. Prefetching a future address can overlap memory work with current computation.
Instruction-level parallelism usually requires more live values and therefore more registers. Additional registers can reduce occupancy while improving issue efficiency. The opposite configuration can have many resident warps whose instructions all form long dependency chains. Warps following the same dependency pattern may become ineligible at the same time. The scheduler needs ready work, not a percentage by itself.
Use occupancy as an experiment variable
Compile several block or tile configurations. Record threads per block, registers per thread, shared memory per block, spills, predicted resident blocks, achieved occupancy, eligible warps, and elapsed time. Change one major resource at a time when possible. This table shows which sacrifice purchased each occupancy increase.
If the kernel already approaches the measured number of bytes transferred between high-bandwidth memory (HBM) and the on-chip L2 cache per second, more occupancy is unlikely to increase that completed-byte rate. If schedulers frequently have no eligible warp and memory latency is exposed, another resident block may help. If forcing more residency creates local-memory spills, the added traffic can make the kernel slower. The interpretation always returns to the active resource and the kernel's physical work.
Worked example: Comparing 50% and 100% occupancy
Use a hypothetical SM with 65,536 registers and a maximum of 64 resident warps. A block has 256 threads, or 8 warps. Compare a small tile that uses 32 registers per thread with a larger tile that uses 64. Assume registers are the active residency limit and ignore allocation rounding for this first calculation.
- The 32-register block requests 256 × 32 = 8,192 registers. The register file can hold eight such blocks, which contain 8 × 8 = 64 warps. Under the stated assumptions, register-limited occupancy is 100 percent.
- The 64-register block requests 16,384 registers. Four blocks and 32 warps fit, so register-limited occupancy is 50 percent. On a real target, round the register allocation as documented and apply the shared-memory, thread, warp, and block-count limits before accepting these counts.
- The small tile reloads operands more often and performs extra address calculations. Model its device-memory and shared-memory traffic per output. The large tile can provide more reuse and independent accumulators even though it has fewer resident warps.
- Check spills. If the compiler keeps the 64-register path on chip, its traffic advantage remains; forcing it down to 48 registers may create local-memory loads that erase the gain.
- Measure both configurations across shape families. The large tile can be faster for large aligned inputs. The small tile can provide more blocks and better tail handling for narrow inputs.
Conclusion: Occupancy is one mechanism for maintaining ready work. Reuse and instruction-level parallelism are others, and the fastest kernel balances them for its shape.
Common errors
- Treating 100% as a score. Residency above the amount needed to cover latency does not improve the limiting resource.
- Applying a register cap without checking spills and instruction count. The compiler must store displaced live values somewhere.
- Reading theoretical occupancy as achieved scheduling quality. Dependencies and imbalance can leave resident warps ineligible.
Reference summary
Occupancy is the number of resident warps divided by the architectural maximum. Resident blocks are limited independently by registers, shared memory, threads or warps, and the maximum block count. The smallest limit determines residency.
More resident warps can help when schedulers lack eligible work. Additional residency does not help when the limiting execution or memory pipeline is already saturated. Forcing lower register or shared-memory use can also reduce tile reuse, remove instruction-level parallelism, or create spills.
resident blocks = min(register limit, shared-memory limit, thread limit, block limit)Knowledge check
Describe a kernel that becomes slower when occupancy rises from 50% to 100%.
Show the answer guide
- A reuse-heavy tiled kernel can already have enough resident warps to cover latency at 50% occupancy while using many registers or shared-memory bytes per tile.
- Forcing 100% occupancy can shrink the tile, remove register-held reuse, add instructions, or cause spills, increasing memory traffic or reducing instruction-level parallelism.
- The decision requires elapsed time, eligible-warp or stall evidence, resource use, spills, and traffic; occupancy alone is a residency ratio rather than a performance objective.
Sweep occupancy without treating it as a score
Vary block size, tile size, or a justified register limit for one kernel to produce at least three theoretical occupancy values. Predict which setting should win before timing.
Evidence to keep: Resource calculations, compiler reports, theoretical and achieved occupancy, eligible-warp or stall measurements, traffic, spills, and raw timing distributions.
Resident threads can cooperate only under explicit ordering rules. Barriers, atomics, and memory semantics define when shared state is safe to observe.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.