NCCL execution and diagnosis
Separate rank-arrival skew, incompatible collective order, stream dependencies, and slow transfer using profiler events, NCCL logs, and health status
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Previous section's collective contracts and schedules
- Chapter 5 CUDA stream dependencies
- Ability to identify GPU, host, and network-interface endpoints
An eight-rank PyTorch job stops after backward step 418. Seven processes show `nccl:all_reduce` as their latest profiler event. Rank 7 shows a data-validation error and never records that event. The seven visible collective calls wait because the communicator cannot finish an eight-participant operation with one participant absent; network bandwidth is not the first explanation.
The NVIDIA Collective Communications Library, pronounced ‘Nickel’ and abbreviated NCCL, provides GPU collective operations such as all-reduce, all-gather, reduce-scatter, and broadcast. A framework submits NCCL work from each rank to a CUDA stream. The host function can return after submission while the GPU and network continue working, so a host log timestamp does not establish device completion.
Collect evidence from all ranks before changing transport settings. PyTorch Profiler records the order and duration of supported communication calls. Nsight Systems records host CUDA calls and device activity on named streams. `NCCL_DEBUG=INFO` with selected debug subsystems records communicator initialization, discovered topology, transport, and network-interface choices. NCCL reliability, availability, and serviceability queries can identify missing processes or communicator operation-count mismatches in supported releases.
A diagnosis compares collective type, sequence number, element count, datatype, communicator, and stream dependencies across ranks. Compatible calls that begin together but transfer slowly support a route or contention investigation. One late or absent rank supports an application, input, device-error, or process-state investigation. The distinction prevents a downstream waiting location from being mistaken for the initiating fault.
Understand the execution boundary
NCCL implements collective and point-to-point primitives for a communicator. NCCL does not schedule the model's control flow or decide which collective each rank should call. The application and framework must create compatible communicators and issue compatible operations in a compatible order.
For CUDA tensors, NCCL work is enqueued on a CUDA stream. A host API return can mean that work was submitted, not completed. Input readiness depends on earlier stream events, and output consumption must wait through the correct stream dependency or work handle. An unexpected dependency can delay the start without changing network service.
Establish a clean baseline
Run official or accepted communication tests across a message-size sweep. Begin with one GPU, peer pairs, and one node. Add a second node and then the intended rank count. Record NCCL, CUDA, driver, firmware, network stack, topology, and every NCCL environment setting.
Inspect discovered topology, peer access, NIC selection, and transport initialization. If primitive tests fail or underperform at the smallest affected scope, investigate hardware, routing, permissions, GPUDirect support, or system configuration before adding the model graph.
Verify collective order and metadata
A collective completes only when the expected ranks participate. Compare step and operation sequence across ranks. Record communicator identity, collective type, count, datatype, reduction operation, root when applicable, and stream. One data-dependent branch that skips or substitutes an operation can block every peer.
A failure can occur before the collective. One rank can raise an exception, enter an infinite host loop, encounter an asynchronous device error, or wait for input. Peers then appear stuck in NCCL because that is their next dependency. Find the first rank and event that diverged.
Separate arrival skew from transfer
Align rank timelines at the collective sequence. Measure when each rank's input becomes ready and when NCCL work begins. The spread between earliest and latest readiness is application skew. Transfer after the last required arrival more closely reflects communication service.
This separation also applies to a slow but completing collective. A rank can enter 13 milliseconds late after a long expert kernel, while the network transfers the payload in the expected 5 milliseconds. Tuning the transport cannot remove the preceding compute imbalance.
Use logs to test topology and selection
NCCL selects transport, algorithm, protocol, and channels from topology, message, operation, and configuration. Debug subsystems can expose initialization, graph, and network decisions. Compare selection changes with message-size curve transitions and topology expectations.
Set one diagnostic override at a time. Forcing an algorithm or interface can test whether automatic selection caused the difference. If it helps, determine the topology or selection condition that led to the default before retaining a production override. Random collections of environment variables make the result difficult to explain and can harm other message regimes.
Use RAS for communicator health
Current NCCL RAS runs an out-of-band health network among process-local RAS threads and can report job and communicator status. A query can identify missing or unresponsive processes and incomplete communicators. Machine-readable JSON is available in recent NCCL releases.
RAS narrows the location of the problem. An unresponsive rank can still be blocked in host code, executing a very long kernel, affected by a device error, or unable to use a network path. Combine RAS with stacks, CUDA errors, logs, and topology telemetry before deciding the cause or recovery action.
Diagnose performance in increasing scope
For low bandwidth, compare pairwise route tests, one-node collectives, two-node collectives, and full-group collectives. Look for the first scope at which the curve changes. For latency, inspect small messages, collective count, launch gaps, and synchronization. For large messages, inspect selected route, channels, shared cuts, and contention.
Then reproduce under application concurrency. Model compute kernels can consume HBM and SM resources while NCCL kernels run. Several process groups can share NICs. A healthy isolated collective can slow in the complete schedule without any hardware fault.
Worked example: Diagnosing an apparent all-reduce hang
In an eight-rank job, ranks 0–6 enqueue gradient all-reduce sequence 418 and wait. Rank 7 does not show that collective after a local data-validation branch.
- Align logs by training step and communicator sequence. Ranks 0–6 expect all-reduce 418 with identical count and datatype. Rank 7's last completed collective is 417.
- Query communicator health and inspect rank 7's process and GPU state. The process is responsive and no transport error appears, which weakens a network-failure explanation.
- Walk backward on rank 7. A validation check detected one malformed sample and returned from the step before the gradient synchronization. The condition was local, so peers did not take the same branch.
- Replace the local early return with a coordinated decision. Reduce an error flag across the group, then make every rank follow the same abort or skip path. Ensure side effects and data position remain consistent.
- Add debug assertions for collective sequence and tensor metadata at selected boundaries. Test an injected rank exception and verify that the launcher terminates or reforms the worker group instead of leaving peers blocked.
- Re-run the healthy communication baseline to ensure the code correction did not conceal any independent topology or network problem.
Conclusion: The collective kernel is where the symptom accumulates, not where the causal divergence began. Cross-rank sequence evidence locates the first disagreement.
Common errors
- Reading the longest rank-local collective bar as network transfer time. The displayed interval can include arrival skew and stream waits.
- Changing many NCCL environment variables at once. The experiment no longer identifies which selection or path mattered.
- Debugging one rank without the communicator sequence from its peers. Collective correctness and progress are relational.
- Treating a RAS status as a complete root-cause report. RAS identifies communicator and process health; other evidence must explain the process state.
Reference summary
NCCL discovers topology and selects transports, algorithms, protocols, and channels for the requested collective. A framework call normally enqueues NCCL work on a CUDA stream. The operation begins only when its input dependencies are satisfied, and its result is safe to consume only after the required stream ordering or work wait completes.
NCCL_DEBUG=INFO NCCL_DEBUG_SUBSYS=INIT,GRAPH,NET torchrun ...
# Validate topology and bandwidth separately with CUDA/NCCL tests.- Verify that every rank reaches the same collective sequence with compatible counts, datatypes, reduction operations, and process groups.
- Inspect GPU assignment, CPU affinity, peer access, NIC selection, topology discovery, and transport initialization before forcing algorithm settings.
- Compare single-rank, peer-to-peer, single-node, and multi-node message sweeps so each added layer has a known baseline.
- Align rank timelines to separate transfer after the last arrival from waiting caused by upstream compute or input skew.
- Use NCCL logs to test topology and selection hypotheses. Remove diagnostic overrides after the experiment unless evidence supports a production setting.
- Use NCCL RAS status to locate incomplete communicators and unresponsive processes, then collect system and application evidence for the cause.
Knowledge check
A collective begins late on one rank. Why might preceding work, instead of network transfer, cause the NCCL wait interval?
Show the answer guide
- A NCCL event can begin late because its input gradient, stream dependency, or application control path becomes ready late on that rank.
- Peers that arrive earlier can spend time waiting inside their collective events, so a long event bar does not equal network transfer time.
- Align collective sequence and input-ready events across all ranks, then walk backward from the last arrival before changing network settings; only comparable arrivals followed by slow progress support a transfer investigation.
Separate arrival skew from collective service
Capture the same collective sequence on every rank for at least ten steps. Record input-ready, collective launch, and first-consumer times, then identify the last arriving rank in each step.
Evidence to keep: Cross-rank table or aligned trace, readiness spread, post-last-arrival tail, preceding operation on the critical rank, NCCL selection logs, and one supported or rejected network hypothesis.
With communication understood, we can decide how model state and computation should be partitioned across ranks.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.