Cross-rank trace alignment
Align ranks by logical step and collective identifiers, separate arrival skew from remaining transfer, and locate the first material divergence
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 13 definitions of rank, communicator, and collective order
- Chapter 7 explanation of how a profiler records timestamped events
- Ability to follow producer-consumer dependencies backward
An eight-rank training step takes 220 milliseconds. On ranks 0 through 6, a PyTorch Profiler timeline shows a 28-millisecond `nccl:all_reduce` event. Rank 7 shows only 9 milliseconds for the same collective, but its preceding attention backward kernel ends 19 milliseconds later than the corresponding kernels on the other ranks. Reading rank 0 alone would incorrectly assign the entire 28 milliseconds to network transfer.
A profiler trace is a timestamped event record captured while a declared workload runs. PyTorch Profiler can record CPU operators and, through GPU tracing support, CUDA kernels and collective calls. Nsight Systems can record CUDA API calls, GPU workloads, and stream rows. Each event has a local start and end time plus identifiers such as process, thread, CUDA stream, and operation name. Profiling overhead and unsynchronized host clocks are part of the measurement limits.
Align the eight records by training step and collective sequence number rather than assuming that host timestamps share a perfect clock. Mark when each rank's gradient buffer becomes ready and when the first dependent consumer can proceed. The spread between earliest and latest readiness is arrival skew. Transfer after the latest required arrival is a better approximation of the remaining collective tail, although chunked algorithms can already have moved some data.
The rank whose delayed dependency determines group progress is the critical rank for that synchronization. Walk backward on rank 7 through GPU kernels, host gaps, input preparation, earlier collectives, and device telemetry until the first material divergence appears. Follow dependencies forward again to verify that later peer waits are consequences rather than separate causes.
Align by logical events
Host clocks across nodes can differ and drift. Use training step, microbatch, module, and collective sequence identifiers to establish logical correspondence. When clock synchronization is accurate enough, use it for a global visual timeline; retain logical IDs as the correctness anchor.
Record readiness, enqueue, device start when available, and completion. A host launch timestamp alone does not show when prior stream dependencies released the operation. The trace needs enough events to reconstruct causality.
Split a collective interval
For one collective sequence, list each rank's input-ready and completion times. Earliest-to-latest ready spread is arrival skew. The interval after the last required rank arrives contains the remaining coordinated transfer and local work, subject to chunking and asynchronous progress.
An early rank's collective bar can equal skew plus service. A late rank can display only service. Comparing one early rank with a standalone alpha–beta model incorrectly attributes upstream waiting to the network.
Do not interpret post-last-arrival time as the complete communication duration without checking the algorithm. A ring or chunked implementation can move some chunks among ready neighbors before the final rank reaches the operation. Per-channel progress events, transport counters, or a controlled replay with synchronized readiness can estimate full service more accurately.
Walk backward from the last arrival
On the last rank, inspect the immediately preceding kernel, event wait, allocation, input operation, and earlier collective. Compare the same logical operations on peer ranks. Find the first event whose completion distribution diverges.
Continue backward until the divergence has a mechanism. A long expert kernel can follow a larger token assignment. A GPU gap can follow a host thread deschedule. A delayed parameter all-gather can follow a congested NIC path. Stop at the first unsupported inference and design a measurement for it.
Classify fixed and rotating stragglers
A persistent physical rank or node suggests topology placement, hardware health, CPU affinity, thermal limits, or local storage. A straggler that follows one tensor-parallel coordinate suggests group mapping. A straggler that rotates with batches suggests data shape or routing distribution.
Record physical host, GPU, NIC, mesh coordinates, and workload properties for every tail event. Correlation does not by itself prove cause, but it determines which controlled swap or replay can distinguish placement from workload.
Analyze maxima and distributions
Synchronous step time follows maxima at rendezvous. Average per-rank compute can remain stable while the maximum grows with rank count because more samples are drawn from the tail. Collect rank distributions per step and step distributions over time.
Correlate step time with maximum sequence cost, expert tokens, input readiness, temperature, error counters, and network queue telemetry. Use several steady-state windows so one unusual trace does not define the diagnosis.
Account for propagated delay
A late rank delays its collective group. The delayed result can then make a pipeline stage late, which delays another group. Many ranks can display waits even though one earlier divergence caused them.
Follow dependency edges forward from the first divergence to verify that observed later waits are consequences. This prevents several copies of the same missing time from being counted as independent bottlenecks.
Handle incomplete clocks and traces
A complete global clock is useful but not required for every conclusion. Collective sequence and step identifiers establish partial order. Stream events establish device dependencies within a rank. Message send and receive pairs constrain order across ranks. Combine these relations into a logical timeline when absolute timestamps are unreliable.
Trace collection can itself drop events or perturb host scheduling. Record buffer overflows, sampling policy, and enabled domains. Compare the sum of visible intervals with measured step time. If 40 milliseconds has no recorded cause, retain it as missing evidence instead of assigning it to the nearest bar.
Confirm the cause with a controlled exchange
After finding a correlation, exchange one variable while preserving others. Move the same workload shard to another healthy rank, keep the rank but change the shard, or remap a process group onto another physical path. A delay that follows the input supports a workload cause; one that follows the hardware location supports a placement cause.
Repeat enough steps to compare tail distributions. The exchange can still alter cache warmth or route sharing, so document those changes and use a second test when the result is ambiguous.
Worked example: Diagnosing delay before a collective
A gradient all-reduce appears to take 18 milliseconds on seven ranks. The alpha–beta model and isolated benchmark predict 5 milliseconds for its message size. The eighth rank displays a 5-millisecond collective.
- Align by step 2,140 and collective sequence 863. Ranks 0–6 make the gradient ready near global time 100 milliseconds. Rank 7 becomes ready at 113 milliseconds.
- All ranks finish near 118 milliseconds. Early ranks show approximately 18 milliseconds from enqueue or entry to completion. Rank 7 shows about 5 milliseconds after its input is ready.
- Walk backward on rank 7. Its preceding expert grouped GEMM takes 13 milliseconds longer than peers. Earlier input and communication events are aligned.
- Compare router outputs. Rank 7 received 1.8× the median expert tokens and executed two unusually full experts. Replay that routing count on another physical rank; the delay follows the workload.
- Change expert placement or routing-balance policy while preserving the model's capacity and quality contract. Alternatively redistribute the grouped expert work if supported.
- Re-run across many steps. Verify that collective arrival spread contracts, post-last-arrival service remains near 5 milliseconds, maximum expert time falls, and no new overflow or quality change appears.
Conclusion: Synchronization amplified a compute imbalance into visible wait across the group. The collective exposed the straggler but did not create it.
Common errors
- Comparing duration bars without aligning readiness. Rank-local bars mix waiting and service differently.
- Inspecting only an average rank. Rendezvous progress follows the last required arrival.
- Stopping at the slow preceding operation without explaining why it was slow. Continue to shape, placement, input, or resource evidence.
- Using one trace as the distribution. Rare per-rank delays become common at scale and require repeated observations.
Reference summary
A collective interval on one rank can contain both transfer and waiting for peers. Align the timestamped profiler records by step and collective sequence, compare when every rank becomes ready, and identify the last arrival. The interval after the last required arrival estimates the remaining collective tail more accurately than the complete event bar on an early rank. The tail is not necessarily the complete transfer time because a chunked collective can make progress before the last global arrival.
- Align ranks with logical step, microbatch, and collective identifiers when host clocks are not sufficiently synchronized.
- Measure the spread between earliest and latest collective readiness separately from transfer after readiness.
- Walk backward from the latest rank through GPU kernels, host gaps, input work, allocations, and earlier communication.
- Track whether the same physical rank, topology coordinate, or workload class remains slow across steps.
- Correlate rank maxima with sequence lengths, expert tokens, temperature, error counters, CPU scheduling, and network telemetry.
- Collect enough steady-state steps to distinguish structural imbalance from occasional system noise.
Knowledge check
All ranks spend 20 ms inside all-reduce, but only one rank’s preceding GEMM takes 18 ms longer. What is the likely first investigation?
Show the answer guide
- The likely first investigation is the preceding GEMM on the one late rank because its 18-millisecond delay matches the peer waiting interval and determines the last collective arrival.
- Align the ranks by step and collective sequence, confirm gradient-ready times, and inspect the late rank's GEMM shape, input, placement, clock, and kernel counters before assigning the delay to the network.
- The all-reduce bars on early ranks include arrival skew; the post-last-arrival tail and a synchronized-readiness replay provide better evidence for transfer service.
Recover one cross-rank causal path
Capture at least ten synchronized logical steps on all ranks and choose one collective with visible duration spread. Align readiness and walk backward from the last arrival to the first material divergence.
Evidence to keep: Step and collective IDs, aligned rank table, readiness spread, post-last-arrival tail, preceding operation evidence, critical-rank identity across samples, and a forward dependency explanation of peer waits.
Aligned causality lets us separate total communication from the fraction that remains exposed after intended overlap with compute.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.