Registers, spilling, and instruction-level parallelism
Explain a register-allocation change using live values, instruction-level parallelism, residency, spill instructions, and measured local-memory traffic
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 5 occupancy calculation
- Basic ability to inspect compiler resource output
- Understanding that one instruction can depend on a previous result
A thread repeatedly uses values such as loop indices, addresses, temporary arithmetic results, and output accumulators. The compiler normally places frequently used thread-local values in registers. An instruction can name those registers directly, so no explicit memory load is required for each use.
Registers are not an unlimited private cache. Each SM has a finite register file shared by its resident threads. If one thread requires more registers, a block requires more of the SM's register capacity. At a threshold, fewer blocks can reside. If a value cannot remain in a register, the compiler may place it in local memory, which is private in address space but served through the cache and device-memory hierarchy.
The competing effects create a non-obvious optimization problem. Loop unrolling can remove branch instructions and expose independent arithmetic, but unrolling can also keep more values live at the same time. A larger output tile can reuse input data, but a larger tile needs more accumulators. Source code alone does not reveal the final allocation, so compiler reports and generated instructions are part of the analysis.
Live values determine register demand
A value is live from the point where it is defined until its final use. Two source variables whose live ranges do not overlap can reuse one physical register. One source array can require many registers, or it can be placed in local memory if the compiler cannot convert its elements into individually addressed values. Counting variable names is therefore not a valid register estimate.
Unrolling duplicates the loop body. Unrolling can expose several independent iterations, but values from those iterations may remain live together. A matrix tile has a similar effect: every output element assigned to one thread or lane group needs an accumulator throughout the K loop. The compiler balances these live values with instruction scheduling and target limits.
Compile with resource reporting enabled and preserve the exact target architecture. Register allocation can change with compiler version, optimization flags, inlining, and architecture. A source change that appears unrelated to the hot loop can alter inlining or live ranges and move the kernel across a residency threshold.
Registers support instruction-level parallelism
Instruction-level parallelism means that one warp has multiple independent instructions ready. Suppose one accumulator depends on the result of the previous multiply-add. The accumulator dependency limits how quickly instructions for that chain can issue. Four independent accumulators can interleave operations while earlier results complete.
Prefetched addresses and operands provide another form of independent work. A kernel can calculate or load values for the next iteration while computing the current one. Future addresses and operands must remain live until use, so the overlap consumes registers. Lower occupancy can be an effective trade when the resulting instruction-level parallelism reduces the number of resident warps needed to cover latency.
Reducing register count can therefore make a kernel slower even when occupancy rises. The compiler may serialize operations that previously overlapped, reduce tile reuse, or insert spills. Measure eligible warps, dependency stalls, instruction count, and traffic along with occupancy.
Identify and correct spills
A spill stores a live value outside the register file and reloads it later. CUDA calls the address space local memory because each thread has a private view. Physical requests still pass through caches and can reach off-chip device memory. A spill inside a frequently executed loop adds memory instructions for every thread and iteration.
Compiler output reports spill loads, spill stores, and local storage, but a nonzero amount does not state where the operations execute. Use source correlation and SASS to locate them. Profiler local-memory transactions show whether they create significant traffic. A spill in a cold error path is different from a spill in the inner matrix loop.
Possible corrections include partial rather than full unrolling, shorter scopes that end live ranges, replacing dynamically indexed private arrays, splitting phases, or recomputing a cheap value. Each correction has another cost. A phase split can add storage or synchronization; recomputation adds instructions; fewer accumulators can expose latency. Select the version with the best complete execution time and correct output, not the smallest register number.
Worked example: Diagnosing an unroll regression
Manually unrolling an eight-iteration loop reduces branch instructions but makes a kernel 18% slower.
- Compare compiler reports. Register use rose from 48 to 92 per thread, and a small amount of local storage appeared.
- Inspect SASS and source correlation. Several spill loads occur inside the repeated arithmetic body, not only at setup.
- Recalculate residency. The register increase also removed one resident block per SM, leaving fewer warps to cover memory dependencies.
- Try partial unrolling and reorganize temporary scopes. Measure instruction count, spills, eligible warps, and time across configurations instead of forcing a register cap immediately.
Conclusion: The full unroll improved one mechanism and damaged two others. Partial unrolling may retain branch savings and ILP without the spill and residency cliff.
Common errors
- Counting source variables to predict registers. Optimization, scalarization, and live-range reuse break any one-to-one mapping.
- Forcing fewer registers and assuming the compiler found a cleverer schedule. The compiler may simply emit local-memory traffic.
- Avoiding all recomputation. Recreating a cheap value can be better than keeping it live through a long region or spilling it.
Reference summary
Registers hold thread-local values that are live during execution. Unrolling, larger output tiles, and software pipelines can increase independent work and reuse, but they also keep more values live. The resulting allocation can reduce resident blocks.
When values spill, the compiler inserts loads and stores in the local address space. Local-address-space loads and stores use the GPU device-memory hierarchy. Locate spills in generated code and measure their traffic; a compiler summary does not show whether the spill occurs in a hot loop.
- Inspect compiler register and spill reports, then confirm local-memory traffic in the profiler.
- Reduce live ranges, avoid unnecessary arrays indexed dynamically, and reconsider aggressive unrolling.
- Balance instruction-level parallelism against occupancy; both are latency-hiding mechanisms.
- Read SASS when source-level intuition and observed instruction/traffic counts disagree.
Knowledge check
How can manually unrolling a loop both increase arithmetic throughput and decrease total kernel performance?
Show the answer guide
- Unrolling can expose independent arithmetic and remove loop-control instructions, allowing more useful instructions to issue while earlier operations wait.
- Unrolling also lengthens live ranges and can increase registers per thread, reduce resident warps, enlarge instruction code, or cause spills to local memory.
- Total performance depends on the balance among instruction count, dependency stalls, residency, spill traffic, and code-cache effects, not on arithmetic throughput alone.
Measure an unroll tradeoff
Compile one fixed-work CUDA loop with unroll factors 1, 2, 4, 8, and 16. Check generated code so the factor actually changes the loop and preserve identical outputs.
Evidence to keep: Disassembly or compiler evidence, registers per thread, spill traffic, achieved residency, instruction counts where available, correctness results, and timing distributions.
Registers are the fastest storage level within a kernel. The next section analyzes layouts across operators because one operator's preferred layout can add conversion or access cost to another operator.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.