Distributed diagnosis process
Reduce a large-scale regression through one-rank, one-node, two-node, and full-scale tests while preserving the mechanism that distinguishes the hypotheses
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Previous sections' efficiency calculations, rank alignment, and overlap experiment
- Ability to preserve workload and machine conditions across comparisons
A 128-GPU training configuration reaches only 1.45 times the throughput of a healthy 64-GPU run, although the global batch and model are unchanged. ‘Poor network scaling’ is not yet a testable explanation. The additional ranks also change local microbatch size, matrix shapes, collective groups, host load, topology cuts, and sensitivity to one slow worker.
Preserve a reproducible slow step before reducing scope. Record model and data settings, global useful work, software versions, rank placement, topology, device clocks, and per-rank step distributions. Capture synchronized logical events for several steady-state steps and collect message-size curves on the intended process groups. The evidence identifies whether missing time first appears in local compute, arrival skew, collective transfer, pipeline idle time, or host orchestration.
Run the same local shapes on one rank, then exercise the eight GPUs in one server, then use two servers, and finally restore the full rank count only when the distinguishing behavior needs it. A one-rank regression supports a kernel or host investigation. A new difference at one server implicates local fabric or rank placement. A difference that begins across two servers supports investigation of network interfaces, fabric routes, or cross-node collective selection.
A distributed diagnosis process is this ordered reduction of hypotheses. Correlations with host, rack, network rail, tensor coordinate, data shard, message size, or time since startup guide the next controlled swap. Every experiment states the observation that would support the hypothesis and the observation that would reject it.
Verify the regression and numerator
Reconstruct a healthy comparison with the same model, valid work, global batch, sequence distribution, precision, checkpoint policy, and quality target. Compare manifests for source, configuration, libraries, driver, firmware, topology mapping, power policy, and data pipeline.
Confirm that throughput loss is not a numerator change. More padding, longer sequences, higher expert capacity, extra recomputation, or stricter validation can increase executed work while the headline model name remains unchanged.
Localize the missing wall time
Reconcile step time into local compute, exposed communication, arrival skew, pipeline idle intervals, host/input gaps, memory management, checkpoint interference, and recovery. Compare categories with the healthy baseline.
Identify which ranks and operations determine the changed critical path. A cluster-wide average can remain stable while one stage or rank tail grows enough to reduce every step.
Correlate with structure
Test whether anomalies follow host, rack, rail, network interface, GPU, mesh coordinate, process group, input shard, message size, or time since initialization. Correlation dimensions suggest different causes and reduction experiments.
Swap workload shards between ranks to separate data from hardware. Swap rank placement while preserving group shape to test topology. Repeat one message size outside the application to test the primitive path.
Reduce to the smallest sufficient experiment
Use one rank for local compute and input behavior, one node for GPU fabric and local collectives, two nodes for NIC and fabric paths, one layer for tensor shapes, or one collective group for communication. Preserve the mechanism suspected of causing the difference.
A reduced test that does not reproduce the regression still provides information. The negative result can eliminate local compute, one topology level, or one message regime. Record the narrowed scope rather than immediately returning to a full run.
Write a falsifiable perturbation
Change one mechanism with a predicted effect. If arrival skew causes the collective wait, balancing sequence work should reduce entry spread while post-arrival transfer remains stable. If a rail is congested, remapping the affected group should move or remove the slow message-size curve.
Define the rejection condition before the run. A change that improves performance through an unexpected mechanism can still be useful, but it does not support the original explanation.
Scale the verified change safely
Apply the change to a small group, then a representative slice, and finally the complete scale required by the hypothesis. At each step, check workload equivalence, correctness, memory, tail behavior, and the predicted diagnostic metric.
Retain rollback and avoid broad full-scale flag sweeps. A diagnostic configuration can reduce observability, exceed memory, or destabilize communication. The cost of a failed large run belongs to the investigation plan.
Validate over time
Intermittent network congestion, thermal behavior, allocator growth, and rare workload tails may not appear in a short trace. Run long enough to include the recurrence interval relevant to the original regression.
Compare distributions before and after. Preserve raw traces and manifests. If the mean improves but tail stalls worsen, the result may not improve useful calendar throughput.
Close the accounting
After the change, rebuild the wall-time decomposition and the useful-work numerator. The category predicted by the hypothesis should shrink by approximately the observed step improvement, allowing for newly exposed work and measurement error. If step time changes but the predicted metric does not, the proposed mechanism remains unsupported.
Record the accepted explanation, rejected alternatives, test scope, configuration, and remaining uncertainty. Include the smallest reproducer, expected metric movement, actual distributions, and rollback condition. Future regressions can then compare against a reproducible baseline instead of repeating the same broad investigation. Revalidate after library, firmware, topology, model-shape, or scheduler changes because each can alter the mechanism and invalidate the accepted parameter ranges.
Worked example: Narrowing a utilization regression
After a deployment, median step time rises from 1.8 to 2.1 seconds on 512 GPUs. Model configuration and global batch names are nominally unchanged, and early traces show longer collective bars.
- Compare manifests. The model, optimizer, and nominal global batch match, but a new sequence-packing policy changes how lengths are assigned to ranks. Average valid tokens per rank remain close.
- Reconcile wall time. Post-last-arrival collective service is unchanged, but all-reduce entry spread rises by 280 milliseconds. The slow rank varies with each batch.
- Compare per-rank inputs. The maximum sum of squared sequence lengths grows substantially under the new policy even though token counts remain balanced. Long attention kernels precede the delayed collective entry.
- Replay old and new packed batches on an eight-GPU node. The new distribution reproduces compute skew without inter-node communication, eliminating the fabric as the primary cause.
- Change the packer to balance an attention-cost estimate as well as tokens. Predict lower maximum attention time and lower collective entry spread with unchanged post-arrival transfer.
- Validate on one node, two nodes, and the original scale. Confirm valid-token throughput, loss normalization, padding, sequence coverage, maximum rank compute, and step-time distribution.
Conclusion: The regression looked like collective overhead because waiting accumulated there. The causal change was workload partitioning and quadratic attention imbalance.
Common errors
- Starting with full-scale configuration sweeps. They are expensive, noisy, and do not isolate a cause.
- Using average rank load to dismiss imbalance. Synchronous progress follows maxima at rendezvous.
- Changing infrastructure before verifying workload equivalence. Padding, shape, and recomputation changes can appear as systems loss.
- Accepting a reduced reproduction that removes the suspected mechanism. The small test must preserve the distinguishing path, shape, or schedule.
Reference summary
A distributed diagnosis reduces scope while preserving the behavior that distinguishes the hypotheses. Verify the objective and workload, compare a healthy baseline, localize the missing time, identify critical ranks and topology coordinates, reproduce the suspected mechanism on the smallest sufficient group, and change one causal variable.
- Confirm that model, global work, quality settings, padding, and data distribution match the baseline.
- Reproduce on one rank and one node to establish compute, input, and intra-node communication behavior.
- Validate pairwise and collective paths independently at representative message sizes.
- Calculate expected local compute, message bytes, collective rounds, overlap windows, and pipeline idle time.
- Capture steady-state timelines across enough ranks and steps to identify the critical path and last-arriving rank.
- Perturb one mechanism such as placement, message size, bucket order, microbatch, routing balance, or parallel degree.
- Scale the validated change gradually and retain rollback, correctness, and long-duration stability checks.
Knowledge check
Why should you run a single-node baseline before investigating poor multi-node scaling?
Show the answer guide
- A single-node baseline determines whether the same local shapes, input path, host work, and GPU kernels are already slow without inter-node networking.
- If the regression first appears when the second node is added, the distinguishing variables narrow to GPU-to-NIC placement, fabric routes, cross-node collective selection, and multi-node arrival effects.
- Scope reduction preserves a causal discriminator and makes controlled swaps cheaper; a full-scale average can hide the first rank, group, or topology coordinate that diverges.
Reduce one scaling regression
Reproduce one fixed-work slowdown at the smallest available scopes: one rank, one node, two nodes, and the required larger group. Preserve software, shapes, and placement policy wherever the test permits.
Evidence to keep: Environment manifest, scope-by-scope step distributions, local shapes, communication curves, wall-time decomposition, first scope where the regression appears, and one controlled change with a predicted observation.
Some large-scale loss is not a tuning regression but the ordinary consequence of failures, checkpoint pauses, and recovery. Reliability must enter the throughput model.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.