CUDA Execution and Synchronization
Map parallel work and dependencies to CUDA
Map logical work to CUDA threads, calculate resource-limited residency, establish synchronization invariants, and express dependencies across asynchronous operations.
A CUDA kernel is a C++ function that runs once for each logical thread in a launch on an NVIDIA graphics processing unit (GPU). The simple description hides several decisions. The indexing formula assigns data to threads. The block shape determines which threads can share memory and synchronize. Per-thread registers and per-block shared memory limit the work that can reside on one streaming multiprocessor (SM). Streams and events order kernels and transfers outside the kernel.
The chapter develops those decisions in the order required for correct reasoning. The first section uses a grid-stride loop whose output can be checked by inspection. Later sections separate launch size from residency, atomicity from memory ordering, and stream order from actual hardware overlap. Each section uses a dependency or resource calculation before discussing optimization.
Do not treat these topics as independent CUDA features. An indexing change can alter coalescing. A larger tile can increase register use and reduce residency. A new stream can expose a missing dependency. The goal is to maintain one model that connects program structure, hardware scheduling, correctness, and elapsed time.
Prerequisites
- Chapter 4 thread, warp, streaming-multiprocessor, and storage-hierarchy model
- Ability to read C++ functions, pointers, arrays, loops, and integer index expressions
Learning objectives
- Prove that a CUDA index mapping covers every output exactly once
- Calculate resource-limited block residency and interpret occupancy measurements
- Express the narrowest correct dependency with barriers, atomics, streams, and events
Study this chapter
- 5.1Kernels, grids, and indexingBegin lesson →
Write and test a bounds-safe grid-stride kernel whose ownership proof covers zero, tail, and very large sizes
- 5.2Occupancy and latency hidingBegin lesson →
Calculate the active residency limit and use profiler evidence to decide whether more resident warps would reduce elapsed time
- 5.3Barriers, atomics, and memory orderBegin lesson →
Choose a barrier, atomic operation, fence, or separate kernel from the participant set, shared data, and required visibility order
- 5.4Streams, events, and overlapBegin lesson →
Draw a copy-and-kernel dependency graph and implement only its required cross-stream event waits
Extended exercises
Complete these projects after the lesson-level practice tasks. Each project combines several mechanisms from the chapter.
Grid-stride primitives
Implement vector addition and SAXPY, then test n = 0, 1, block size minus one, block size plus one, and a large non-multiple.
Evidence: Exact output checks and CUDA-event duration for every declared input size.
Resource tradeoff
Sweep at least four block or tile sizes while keeping the mathematical operation and input shape fixed.
Evidence: A table of registers per thread, shared memory per block, predicted and achieved occupancy, eligible warps, spills, and separately timed kernel duration.
Overlapped pipeline
Move pinned-host-memory chunks through copy-in, CUDA computation, and copy-out stages using streams and event dependencies.
Evidence: An Nsight Systems timestamped timeline annotated with each dependency plus throughput and first-item latency against a serialized baseline.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.