Overlap, exposed communication, and bubbles
Measure readiness, completion, first consumption, exposed tail, concurrent-kernel slowdown, and peak memory for one overlap change
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
A serialized segment contains a 25-millisecond matrix multiplication followed by a 20-millisecond gradient all-reduce, for 45 milliseconds. After the independent all-reduce moves earlier, overlap lengthens the matrix multiplication by 4 milliseconds, from 25 to 29 milliseconds, and the collective finishes 4 milliseconds after the matrix multiplication. The overlapped segment therefore takes 29 + 4 = 33 milliseconds and improves by 45 − 33 = 12 milliseconds, not by the full 20-millisecond collective duration. The arithmetic shows why visual concurrency does not make communication cost disappear when both operations compete for GPU and memory resources.
Capture three controlled runs with identical shapes and clock conditions: compute alone, communication alone, and their scheduled overlap. PyTorch Profiler or Nsight Systems provides event start and end times; CUDA events can provide lower-overhead duration checks for declared stream intervals. GPU memory-bandwidth and scheduler measurements can show whether the communication kernel reduces the resources available to the matrix kernel.
Communication is exposed when a dependent consumer must wait and no independent useful operation fills the interval. In the example, a 4-millisecond collective tail after compute is directly exposed, while concurrent transfer can still have an indirect cost by lengthening compute. A bubble is an interval in which a pipeline stage has no ready useful operation because of fill, drain, imbalance, or another dependency. Resource contention is different: both operations are present but make slower progress.
Measure readiness time, completion time, first-consumer time, compute slowdown, communication slowdown, and peak memory. Earlier prefetch or smaller gradient buckets can enlarge the overlap window, but the same changes can retain full parameters longer, increase startup count, or collide with another collective. Complete step time decides whether the new schedule helps.
Define readiness and consumption
Communication cannot begin before its input exists and its stream dependencies are satisfied. The transfer must finish before the first dependent consumer proceeds. The interval between readiness and consumption is the available window for overlap.
Changing bucket membership can move readiness earlier. Prefetch can issue an all-gather before natural demand. Scheduling another independent kernel before consumption can extend the window. Each change can also alter memory lifetime or computation order.
Measure exposed communication
Suppose a 20-millisecond all-reduce begins while 25 milliseconds of independent backward compute remains and finishes before that compute. In a no-interference model, it contributes no direct tail. If 4 milliseconds remain after compute, those 4 milliseconds are exposed.
A counterfactual without communication is not always directly measurable because the result is required. Use controlled variants, dependency analysis, and isolated durations to estimate it. Reconcile the estimate with complete step time rather than summing all collective durations.
Measure interference
Overlap can slow both operations. NCCL kernels can use SMs and HBM bandwidth. Model kernels and communication can share memory, PCIe, NVLink, NIC, or fabric resources. A 20-millisecond transfer under overlap can become 26 milliseconds while a 25-millisecond GEMM becomes 30.
Measure compute alone, communication alone, and concurrent execution with the same shapes and clocks. The best schedule minimizes the critical path including mutual slowdown, not the amount of colored overlap in a trace.
Trade overlap against memory
FSDP prefetch obtains full parameter groups earlier. The transfer can move under current compute, but the gathered parameters remain live until use and release. Deeper prefetch can hold several groups simultaneously.
Gradient bucketing has a related trade. A smaller bucket launches earlier with less wait for readiness but pays more startup and can contend for longer. Include peak allocation and startup count in every overlap experiment.
Classify pipeline idle intervals
Structural fill and drain occur because early or late pipeline slots do not have a microbatch ready. Stage imbalance creates recurring idle time near a slower stage. A delayed point-to-point transfer creates dependency wait. A host launch gap or missed prefetch is a scheduling defect.
Label each interval from the rank timeline. Increasing microbatch count can reduce structural fill/drain but does not repair a slower stage or host gap. Repartitioning stages does not remove an unavoidable dependency unless the ownership or schedule changes.
Consider simultaneous parallel axes
A large-model step can overlap tensor collectives, FSDP all-gathers, data reductions, pipeline sends, and expert all-to-all. The communication operations can use different communicators but share network-interface ports and fabric cuts.
Build a link and device-resource schedule. Moving one collective earlier can collide with another and create a new tail. Measure at the intended mesh scale when the hypothesis concerns shared-fabric contention.
Use a dependency interval calculation
For each communication operation, record input-ready time r, completion time c, and the time u when the dependent consumer would otherwise be ready in a no-communication or no-wait comparison. For this one dependency, direct exposed delay is max(0, c−u), and the consumer can begin at max(c,u). Estimate u from a controlled variant or from the independent predecessor path; the observed consumer start alone cannot reveal the counterfactual.
Interference changes u. For example, suppose communication finishes at 26 milliseconds and compute would finish at 25 milliseconds alone, but concurrent execution slows compute until 30 milliseconds. The communication has no direct tail in the concurrent schedule because c < 30, yet overlap adds five milliseconds through compute interference. Use the resulting consumer-ready time once; do not add the 26-millisecond communication bar and the five-millisecond slowdown.
The calculation is local to a dependency and does not automatically equal whole-step improvement. Removing one tail can expose a later collective, pipeline stage, or host gap. Recompute the complete critical path after each schedule change.
Check correctness of reordered work
Earlier communication requires correct stream and event dependencies. A reduction must not read an incomplete gradient, and its result must be complete before the optimizer consumes it. Buffer reuse must wait for both communication and dependent computation.
Capture race-detection evidence where available and run numerical equivalence tests across many steps. Rare stream-order bugs can produce plausible losses before causing corruption or a hang.
Worked example: Fixing the last gradient bucket
A 210-millisecond backward pass overlaps most gradient reductions. After the final compute kernel, one 7-millisecond all-reduce remains fully exposed.
- List parameters in the final bucket and their ready times. Most become ready between 194 and 202 milliseconds, but one early-layer gradient arrives at 209 milliseconds and holds the 96 MiB bucket.
- Split the early-ready 80 MiB into one bucket and leave 16 MiB with the last gradient. The 80 MiB transfer can begin at 202 milliseconds while 8 milliseconds of compute remains. The smaller final transfer pays a separate startup.
- Use the alpha–beta curve to predict both transfers. Before the change, the measured segment is 210 milliseconds of backward work plus a 7-millisecond tail, or 217 milliseconds. With no interference, the new schedule predicts 210 + 2 = 212 milliseconds, a 5-millisecond improvement.
- Measure remaining backward kernels. They lengthen by 1 millisecond during the early collective because both use HBM bandwidth. The after interval is therefore 211 + 2 = 213 milliseconds, so the measured improvement is 217 − 213 = 4 milliseconds. The before and after intervals reconcile without adding the overlapped transfer twice.
- Test an alternative parameter order and a 48 MiB split. Compare distributions, not one step, and verify reduction scaling and optimizer inputs.
- Select the schedule with the lowest backward-plus-exposed-tail time and acceptable memory. Document that its value depends on current gradient readiness and compute kernels.
Conclusion: A configuration with lower total-bandwidth efficiency can still reduce step time if it removes communication from the critical path.
Common errors
- Equating concurrent bars with zero cost. Shared-resource interference can lengthen compute and communication.
- Adding every communication duration as overhead. Overlapped intervals are not independent contributions to wall time.
- Moving communication earlier without checking input readiness, output consumption, and memory lifetime.
- Increasing microbatches to reduce every pipeline idle interval without distinguishing structural bubbles from imbalance and scheduling gaps.
Reference summary
Total communication time and exposed communication time are different quantities. Communication is exposed when a dependent operation cannot proceed and no independent useful work fills the interval. Communication that overlaps independent compute can leave the critical path, but the two operations can still slow each other through shared HBM bandwidth, copy engines, SM resources, PCIe, or network links.
- Readiness: a gradient bucket or parameter all-gather cannot begin until its input or schedule dependency is satisfied.
- Consumption: communication must finish before the first dependent kernel or optimizer update consumes the result.
- Memory: earlier FSDP prefetch extends the lifetime of full parameters and can create a higher peak.
- Interference: NCCL kernels and compute can compete for device execution and memory resources even when their trace bars overlap.
- Fabric contention: simultaneous data-, tensor-, and expert-parallel collectives can share NICs and network cuts.
- Pipeline idle time: separate required fill/drain slots from stage imbalance, delayed transfers, and scheduler gaps.
Knowledge check
Why can moving an all-reduce earlier increase overlap yet slow the GEMM it overlaps?
Show the answer guide
- Moving the all-reduce earlier can make its event bar overlap an independent GEMM, but NCCL kernels can consume streaming-multiprocessor slots and HBM bandwidth used by that GEMM.
- The GEMM can therefore lengthen even when the collective's direct tail shrinks; the net result is the new compute duration plus any remaining dependent tail, counted once.
- A controlled comparison needs compute alone, communication alone, concurrent execution, complete step time, readiness, first consumption, memory peak, and correctness for identical shapes and clocks.
Measure overlap and interference
Run one representative compute kernel and collective alone and in the intended concurrent schedule. Record the before and after critical-path equations before deciding whether visible overlap helps.
Evidence to keep: Event timelines, isolated and concurrent duration distributions, readiness and consumer times, HBM or scheduler evidence, peak memory, explicit arithmetic that reconciles both total intervals, and correctness checks.
The distributed diagnosis process combines objectives, quantitative models, aligned traces, and controlled changes to identify the cause of a scaling regression.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.