Distributed Performance Analysis

Consider a training job that uses 64 devices, where each device is a graphics processing unit (GPU). The job's median step time increased from 410 milliseconds to 485 milliseconds after a software update. One central processing unit (CPU) process controls each GPU. A cluster telemetry service samples NVIDIA Data Center GPU Manager activity for every GPU once per second and publishes a dashboard average of 81 percent. The sampled activity reports that GPU work was present during many sampling intervals, but the cross-device average does not identify the extra 75 milliseconds. One delayed process can make 63 healthy peers wait, so the process that displays the longest wait is not necessarily the process that caused the delay.

The investigation begins by collecting the same steady-state steps on every rank with PyTorch Profiler or Nsight Systems. Each profiler writes timestamped events for CPU operations, CUDA kernels, and supported communication calls. The resulting trace is a timeline record produced during that profiling run. Engineers add step, microbatch, layer, and collective identifiers so events from different process clocks can be matched even when their absolute timestamps are imperfectly synchronized.

A concrete comparison might show ranks 0 through 62 reaching gradient all-reduce number 417 at 360 milliseconds, while rank 63 arrives at 433 milliseconds after an unusually long general matrix multiplication (GEMM). The wide collective bars on the early ranks mostly record waiting for rank 63. The earlier matrix operation on rank 63 is therefore the first unexplained divergence, while the collective wait is a downstream symptom. An NVIDIA Nsight Compute profile of that GEMM can estimate streaming multiprocessor (SM) activity and bytes transferred between high-bandwidth memory (HBM) and the on-chip cache hierarchy. The named Nsight Compute measurements test compute-underuse and HBM-traffic explanations without treating the dashboard average as a cause.

This chapter defines efficiency ratios from explicit numerators and denominators, including estimated model floating-point operations (FLOPs). The chapter also explains how distributed traces are produced and aligned, distinguishes useful overlap from visual overlap, and reduces large-job regressions to controlled experiments. Every calculation must reconcile with measured step time; any remaining interval is labeled unclassified rather than assigned to a convenient cause.