SMs, warps, and SIMT
Map a CUDA grid to blocks, 32-thread warps, and streaming multiprocessors and explain the visible effect of branch divergence
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Previous section's distinction between latency and throughput
- Ability to read a per-thread array-index expression
Begin with a kernel that adds one to each array element. In the source code, each thread calculates an index, reads one value, and writes one value. The threads appear independent. On an NVIDIA GPU, however, the streaming multiprocessor creates and schedules them in groups of 32 called warps. One warp instruction operates on the active lanes in that group.
NVIDIA documentation calls the relationship between logical threads and warp instruction issue single instruction, multiple threads (SIMT). Both views are necessary. The logical-thread view makes indexing and ownership easy to express. The warp view explains instruction issue, branch divergence, coalesced memory access, and warp-level communication. Code can be correct in the logical view and inefficient in the warp view. Code that assumes undocumented warp behavior can also be incorrect even when the code often passes tests.
A thread block adds another level. Every block is assigned to one streaming multiprocessor for its lifetime. Threads in the block can exchange data through shared memory and can use block-scoped barriers. Threads in different blocks are intentionally independent unless the program uses a wider mechanism. This restriction allows the GPU to schedule blocks in any order.
From a grid to resident warps
A launch creates a grid of blocks. The runtime distributes those blocks across streaming multiprocessors as resources become available. A block never migrates to another SM while it runs. Its threads are divided into warps, with a partially filled final warp when the block size is not a multiple of 32. Inactive positions in that final warp still belong to the hardware scheduling unit, so block shapes should account for them.
An SM contains warp schedulers, a register file, shared-memory and cache capacity, and multiple execution pipelines. When a block becomes resident, its registers and shared memory are reserved. The scheduler can select a warp only when that warp's next instruction has ready operands and the required pipeline can accept the instruction. A resident warp is therefore not always an eligible warp, and an eligible warp is not always issued in a given cycle.
The resident-to-eligible sequence explains a common profiler result. A kernel can report high occupancy because many warps are resident, yet report few eligible warps because all of them wait on similar memory dependencies. Residency creates opportunities for latency hiding; residency does not guarantee independent work.
Newer CUDA devices can also schedule thread blocks as a cluster and can expose distributed shared memory across those blocks. A block still executes on one SM, but cluster membership adds placement and cooperation rules beyond ordinary block scope. Use the ordinary independent-block model unless the launch and kernel explicitly use cluster features.
Branch divergence within a warp
Consider if (x[i] > 0) for consecutive elements. If every lane sees a positive value, the warp executes the true path with all lanes active. If some values are positive and others are not, the warp must execute the instructions required by both paths while enabling only the lanes assigned to each path. The result remains correct, but some execution slots do no useful work during each path.
Divergence cost depends on the work in the paths, not merely on the presence of an if statement. A branch whose condition is uniform for a complete warp does not split lane activity. A divergent branch with two instructions may be inexpensive. A divergent branch in which one lane performs a long loop can be costly because the other lanes remain inactive while that path completes. Data ordering and variable trip counts therefore matter.
Do not assume that branch removal is always an improvement. Predication can execute instructions for every lane and discard some results. Predication is useful for short paths, but predication can waste more work than a branch when the guarded operation is expensive. Measure active-lane behavior and total instructions for the target data distribution.
Independent thread scheduling and explicit cooperation
GPUs with Independent Thread Scheduling maintain more execution state per lane than earlier lockstep designs. The additional state permits finer scheduling around divergence. Independent Thread Scheduling does not mean that lanes issue completely independent instructions like CPU cores, and the feature does not make implicit communication between lanes safe.
When lanes exchange values with shuffle operations or communicate through memory, specify the participating mask and use the synchronization operation required by the CUDA programming model. A mask describes the lanes that must participate. Passing a full-warp mask when some named lanes cannot reach the operation can violate the contract. The safe method is to derive the participation set before divergence or to use a cooperative group that represents it explicitly.
Worked example: Reasoning about a divergent filter
Each lane examines one token. Tokens that pass a predicate perform 20 arithmetic instructions; failures perform 2. In one warp, only four lanes pass.
- If the branch is divergent, the warp executes the long path with four lanes active and the short path with the other lanes active. Issue time resembles both paths, not a weighted average of per-lane work.
- Measure whether passing tokens are clustered. Reordering data so warps contain mostly pass or mostly fail cases can turn a divergent predicate into a nearly uniform one, at the cost of compaction or preprocessing.
- Compare predication. For a very short path, executing a few instructions with inactive results may be cheaper than explicit branch control. For 20 expensive instructions, predicating every lane wastes too much work.
- Check the next stage. Compaction may improve this kernel but add traffic and synchronization; include its end-to-end cost and whether the compacted order is acceptable.
Conclusion: The relevant quantities are the path lengths, the distribution of passing tokens across warps, and the cost of reorganizing the input. A branch is not automatically a problem, and branch-free code is not automatically faster.
Common errors
- Thinking of a warp as 32 independently issuing CPU threads. Lanes have separate state but share issue and often execute the same instruction together.
- Removing every branch. Predication or duplicated work can be worse than a mostly uniform conditional.
- Using warp-synchronous tricks without masks or synchronization. Scheduling flexibility makes unstated ordering assumptions fragile.
Reference summary
A CUDA launch creates a grid of thread blocks. Each block is assigned to one streaming multiprocessor for its lifetime. NVIDIA hardware divides the block's threads into warps of 32 lanes and schedules instructions for those warps.
Each lane has its own registers, index values, and control state. The lanes still share warp instruction issue. If lanes follow different branches, the warp executes the required paths with the appropriate lanes active. The cost depends on the length of each path and the distribution of lanes.
- A streaming multiprocessor (SM) owns schedulers, register storage, shared memory/cache resources, and execution pipelines.
- A block is assigned to one SM for its lifetime; its threads can cooperate through shared memory and block synchronization.
- Warp-level communication must use the participation mask and synchronization required by the programming model.
Knowledge check
What happens when half a warp takes the if branch and half takes the else branch? What cost depends on the amount of work in each path?
Show the answer guide
- The warp must execute the instructions required by both branch paths when both paths contain active lanes; lanes not assigned to the current path are inactive for that instruction.
- The cost depends on the amount of work and memory activity on each path, the active-lane pattern, and whether the paths can reconverge quickly.
- Independent Thread Scheduling does not turn the lanes into unrelated CPU cores, so cross-lane communication still needs the documented participation mask and synchronization.
Observe warp divergence
Benchmark a CUDA kernel whose branch condition sends alternating lanes down two paths, then compare it with all lanes taking one path and with a warp-uniform branch. Keep the total element count and memory accesses comparable.
Evidence to keep: Kernel source, block shape, branch conditions, raw timings, and profiler branch or active-thread evidence that explains the measured difference.
Warps can issue general arithmetic and memory instructions. Many AI workloads also use matrix instructions in which a warp or warp group cooperates on a tile. The next section explains the additional shape, layout, and data-delivery requirements of those instructions.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.