Network Topology and Collective Communication

Consider one backward pass for a transformer trained on eight graphics processing units (GPUs). One central processing unit (CPU) process controls each GPU, while GPU kernels execute the backward operations that compute a local 256 MiB gradient tensor in the GPU's high-bandwidth memory (HBM). The optimizer cannot update the weights until the eight device-resident gradient copies have been combined. PyTorch therefore asks a communication library to sum the tensors across the participating GPU processes and to return the same sum to each process.

A learner can observe the communication in two places. A PyTorch Profiler capture records a timestamped event for the collective call on each process. NVIDIA Nsight Systems records the CUDA calls made by the process and the communication kernels that execute on each GPU stream. Such a timestamped record is called a trace. A horizontal bar in the trace represents an operation with a recorded start time and end time; the bar does not, by itself, explain which physical route carried the bytes or why the operation waited.

The 256 MiB payload can cross a direct GPU link, an on-board GPU switch, Peripheral Component Interconnect Express (PCIe) switches, a network interface card, and data-center fabric switches. Remote direct memory access (RDMA) lets a capable network device access registered GPU memory without staging the payload through an ordinary CPU buffer. The route depends on GPU placement, network-interface affinity, and the topology detected by the collective library. Competing transfers can share one link even when the application assigns different rank numbers to their endpoints.

This chapter connects the application request to that physical route. NCCL executes communication kernels on the GPU's streaming multiprocessors (SMs), so communication can compete with model kernels for GPU execution resources as well as memory bandwidth. The sections derive message startup and transfer time, follow the steps of common collective algorithms, and explain how the NVIDIA Collective Communications Library (NCCL) selects and executes a path. Each model ends with a measurement that can show whether the predicted route, byte count, or waiting time appears in the running job.