Software pipelines and asynchronous copies

Construct a staged producer-consumer pipeline and enforce readiness and reuse dependencies

Before you begin

Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.

Tiled GEMM
Streams versus in-kernel asynchrony
Barriers

A tiled matrix-multiplication kernel processes the K dimension in 32-element panels. Nsight Compute shows matrix instructions executing, followed by an interval in which the warp issues no next matrix instruction. The Source Counters report maps many sampled issue stalls to the wait immediately before the kernel consumes the next shared-memory panel; the memory report and instruction listing show the corresponding global-to-shared loads still in the dependency chain. The mapped stalls and outstanding loads support the hypothesis that panel arrival is exposing memory latency, but the counters alone do not prove the cause. A two-stage version tests the hypothesis: if it overlaps the next panel load with current matrix work and reduces the mapped stalls and elapsed time without changing the bytes or arithmetic, the measured response supports the dependency explanation.

A revised kernel reserves two shared-memory regions. CUDA threads compute from panel 0 while an asynchronous copy begins moving panel 1 from GPU high-bandwidth memory into the other region. The profiler should show less producer-wait time and a higher matrix-instruction issue rate when the overlap succeeds. Engineers call the repeating load-and-compute arrangement a software pipeline because the kernel keeps several K panels at different stages of progress.

A tiled GEMM can have excellent reuse and still stop between every panel: load a tile, wait, compute it, then load the next. Software pipelining rearranges time. While matrix instructions consume one stage, asynchronous operations prepare a future stage. If supply and compute remain balanced, the loop approaches the slower steady-state stage instead of their sum.

A software pipeline uses a finite number of storage stages. Additional stages allow loads to start earlier, but each stage uses shared memory and prefetched fragments use registers. Pipeline startup and completion also add overhead. Short K dimensions can complete before additional stages recover their resource cost.

The lesson separates correctness from performance. Correctness requires that the consumer never read an incomplete stage and that the producer never overwrite a stage still in use. Performance asks how far ahead the producer must run to keep the arithmetic pipeline supplied without consuming too much shared memory.

Why alternating load and compute leaves idle time

In a single-buffer kernel, the K loop has two ordered phases. All threads load the next A and B panels and wait until the data is visible. They compute from those panels and wait until every consumer has finished. Only then can they overwrite the buffer. If the load takes L cycles and the arithmetic takes C cycles, one iteration approaches L+C cycles because the phases do not overlap.

With two buffers, the block can compute from buffer 0 while the producer starts filling buffer 1. In an ideal steady state, iteration time approaches max(L,C), not L+C. The timing benefit requires an asynchronous movement mechanism or enough independent instructions to overlap the work. Merely allocating two arrays does not create overlap.

Prologue, steady state, and epilogue

The prologue initiates enough loads to make the first consumer stage ready. The steady-state loop waits for the current stage, computes from it, and schedules a later stage. Stage indices advance through a circular buffer. The epilogue stops issuing new loads and consumes the valid stages that remain.

Separating these phases prevents boundary errors. The prologue must not read beyond K when the problem contains fewer K tiles than pipeline stages. The steady-state loop must associate each readiness event with the correct buffer. The epilogue must not consume a buffer merely because its ring index exists; the corresponding asynchronous operation must have completed.

Producer-consumer synchronization

Each stage has two relevant transitions. After the producer finishes writing the stage, consumers may read it. After every consumer finishes, the producer may reuse the storage. A readiness barrier enforces the first transition. A reuse barrier or the equivalent pipeline protocol enforces the second. Missing either transition can produce intermittent wrong values because warp scheduling and memory latency vary.

CUDA asynchronous-copy facilities and newer transfer engines differ by architecture, alignment, and synchronization API. Treat these as implementations of the same state machine. For every stage, identify who produces it, who consumes it, what event marks production complete, and what event makes reuse safe. Then map those events to the documented primitive on the target architecture.

Selecting the number of stages

Pipeline depth determines how early a load can begin. If compute per K tile is long enough, one future tile may cover most memory latency. If compute is short, several future tiles may need to remain in flight. Deeper staging also creates more independent memory requests, which can improve latency tolerance until the memory system or request queues saturate.

Every additional stage allocates another shared-memory panel and may keep more register fragments live. Added storage can reduce resident blocks per streaming multiprocessor. A kernel that removes its internal wait stalls but cuts occupancy in half can lose overall throughput. Compare stage counts with shared-memory allocation, register count, resident blocks, producer-wait metrics, and end-to-end time across K sizes.

Worked example: Choosing between two and four stages

A GEMM K iteration spends about 220 cycles of matrix computation per tile, while an uncached panel load has roughly 500 cycles of effective latency.

With two stages, one future tile can be in flight while one computes. Roughly 220 cycles of the latency are covered, leaving producer wait unless multiple requests and lower cache latency help.
A simple uncovered-latency estimate is max(0, 500−220) = 280 cycles. Treat this only as a starting estimate because loads overlap internally and effective latency depends on cache behavior and concurrency.
Four stages can place several loads ahead and cover more of the 500-cycle delay, but doubles shared storage relative to two stages.
Recalculate resident blocks. If four stages reduce residency from two blocks to one, the SM loses another source of latency hiding and may expose different bubbles.
Benchmark both over K sizes. Deep staging should help long steady loops more than short K, where prologue and epilogue occupy a large fraction.
Inspect the profile for memory-dependency waits and matrix-pipeline issue rate. A timing difference without these measurements does not show whether staging fixed the expected cause.

Conclusion: Stage count is not a fixed architecture recipe. The selected count balances within-block lookahead against storage capacity and whole-streaming-multiprocessor concurrency.

Common errors

Increasing stages until producer waits disappear while ignoring residency loss. Another form of latency hiding may have been sacrificed.
Forgetting pipeline startup and drain. Short problems can become slower despite a better steady state.
Reusing a shared buffer before all consumers finish. Asynchronous production requires equally explicit lifetime control.
Assuming an asynchronous API call implies completed data movement. The call initiates work; the documented wait operation establishes visibility to consumers.

Reference summary

A simple tiled kernel alternates load and compute phases. Multi-stage pipelines reserve several shared-memory buffers: while matrix engines compute from one stage, asynchronous copy machinery fills a later stage. Correctness requires tracking when a stage is safe to overwrite and when its data is visible to consumers.

Prologue: fill enough stages before the steady state.
Steady state: wait for the current stage, compute, initiate a future stage, and advance circular indices.
Epilogue: drain remaining stages without reading beyond K.
Stage count trades latency coverage against shared-memory capacity and occupancy.
Architecture-specific mechanisms such as asynchronous global-to-shared copy or tensor memory acceleration change implementation, not the dependency model.

Knowledge check

What dependency must be enforced before the producer overwrites a circular shared-memory stage?

Show the answer guide

A circular pipeline reuses each shared-memory stage. Before a producer overwrites a stage, every consumer that reads the previous tile from that stage must have completed those reads.
The required dependency is a consume-before-reuse or read-after-use ordering edge. The reuse dependency is distinct from the producer-to-consumer edge that makes newly loaded data visible before computation begins.
A block barrier can enforce the dependency in a synchronous pipeline. An asynchronous pipeline uses stage-specific barriers, transaction counters, or wait operations so the producer does not reuse a stage while any consumer still owns it.
The protection must include completion and visibility at the correct scope; merely issuing the consumer instructions or the next asynchronous copy does not establish safe reuse.

Practice

Demonstrate a circular-stage overwrite hazard

Build a two-stage tiled copy-and-compute kernel for a 4096-by-4096 array in which each stage is loaded, consumed by a deterministic reduction, and then reused. First implement correct stage-specific waits. Then create an experimental version that removes only the consume-before-reuse wait and run both versions repeatedly with randomized input and an artificial delay in one consumer warp.

Evidence to keep: Submit both code paths, a deterministic reference test, failure counts across at least 1,000 stress iterations, and a pipeline timeline or annotated trace showing load, consumer read, wait, and stage reuse. Explain the exact ordering edge whose removal permits stale or overwritten data.

Pipelining feeds arithmetic efficiently. Matrix instructions determine the precise fragment shapes and lane mappings that perform the arithmetic.

References

Sources and further reading

The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.