Latency–bandwidth communication models
Use a message-size sweep to estimate startup seconds and seconds per payload byte, then identify where the two-term model stops fitting
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Previous section's physical route drawing
- Ability to fit or evaluate a straight-line expression with seconds and bytes
A backward pass produces 200 gradient buffers of 64 KiB each. Sending the buffers as 200 separate all-reduces takes about 5 milliseconds on the target eight-GPU server. Combining the same 12.5 MiB of payload into one ready buffer takes less than 1 millisecond in an isolated communication benchmark. The comparison suggests that repeated startup work matters, but it does not prove the cause because the backward run and isolated run have different readiness, overlap, and contention. A message-size sweep under one declared operation, route, rank group, and load is needed to estimate the fixed term.
Measure the effect with a message-size sweep rather than with one transfer. A communication benchmark sends the same operation repeatedly at 1 KiB, 2 KiB, 4 KiB, and progressively larger sizes, synchronizes at the declared completion boundary, and reports payload bytes divided by elapsed time. A PyTorch Profiler capture can also record a timestamped readiness trace for the gradient buffers. The small-message points appear nearly flat in elapsed microseconds; large-message points grow approximately in proportion to bytes. A new bend can indicate a protocol, algorithm, or transport change.
The latency-bandwidth model writes the measured time as α + nβ. The Greek letter alpha names the fitted intercept in seconds for the selected operation, measurement boundary, route, rank group, concurrency, and message-size region. Alpha can combine dispatch, queue, launch, protocol, and propagation effects; alpha is not a direct measurement of link latency or proof of a particular route. The letter beta names fitted seconds per payload byte, so 1/β is the sustained payload bandwidth in bytes per second. The variable n is the payload size that the benchmark declares, not the number of wire bytes after packet headers or retries.
The model deliberately omits queueing, chunk pipelines, shared-link contention, local reduction, and overlap with computation. Engineers use the two-term line to predict a first regime and then inspect residuals—the measured time minus predicted time. Systematic residuals are observable evidence that the communication path needs a richer, usually piecewise, explanation.
Separate startup from transmission
For a payload of n bytes, T(n) ≈ alpha + n beta. Alpha is the fixed cost observed at the selected boundary. The fitted startup term can include host dispatch, device kernel launch, queue setup, protocol work, and propagation. Beta has units seconds per byte and is the reciprocal of sustained payload bandwidth.
At n = 0, the fitted intercept approximates alpha. At large n, the slope approaches beta when one stable transport and protocol remain active. Real libraries can switch protocols or algorithms, so a single straight line may apply only within one message-size region.
Find the transition size
Startup and transmission contribute equally when alpha = n beta, so n = alpha/beta. If alpha is 20 microseconds and sustained bandwidth is 20 GB/s, beta is 1/(20×10^9) seconds per byte. The transition size is about 400,000 bytes. Much smaller messages are startup-sensitive; much larger messages are bandwidth-sensitive.
This calculation predicts the direction of an optimization. Halving alpha almost halves a 1 KiB transfer dominated by startup. Doubling bandwidth has little effect on that transfer. For a 1 GiB message, the transmission term dominates and bandwidth matters far more.
Calculate a serialized-round estimate
A collective algorithm performs several communication rounds. If k rounds are modeled as fully serialized and each sends m bytes, the serialized-round estimate is k(alpha + m beta). The expression is not an upper bound: pipelining can make the measured collective faster, while contention, local reduction, or extra synchronization can make it slower. The estimate describes one declared no-pipeline schedule used for comparison.
Count the dependency depth separately from total transfers. A tree can use approximately logarithmic depth, while a ring uses P−1 steps per phase. For large pipelined buffers, steady link use can outweigh the ring's greater round count. For small messages, repeated alpha costs can favor the shorter depth.
Add local reduction and memory work
A reduce operation does more than transmit bytes. The receiving GPU reads an incoming chunk, reads or accesses its local chunk, performs the reduction, and writes the result. This consumes device memory bandwidth and execution resources. Add a gamma term for local work when it is significant: T ≈ alpha + n beta + n gamma.
Communication kernels can also share HBM bandwidth or SM resources with model kernels. The isolated communication curve can therefore underpredict duration during attempted overlap. Measure both operations alone and together to quantify interference.
Fit a piecewise model from measurements
Run a geometric message-size sweep from bytes to the largest representative tensor. Repeat enough times to obtain distributions. Plot time against bytes and bandwidth against bytes. Look for fixed-cost plateaus, rising throughput, protocol transitions, and large-message saturation.
Fit alpha and beta only within a region whose transport and algorithm remain comparable. Keep separate fits for intra-node, cross-node, and contended conditions. Use the residual between predicted and measured time to test for chunk overhead, topology sharing, queueing, or a selection change.
Connect aggregation to readiness
Combining several small payloads pays alpha fewer times and often reaches a better bandwidth regime. The combined buffer cannot launch until its final member is ready. Aggregation therefore saves startup but can delay communication and reduce overlap.
For gradient buckets, compare saved alpha with the amount of backward compute that remains after each bucket becomes ready. The best bucket is a schedule result, not simply the size that maximizes isolated collective bandwidth.
State the model boundary
Alpha and beta summarize observations at a selected boundary; the fitted terms do not identify which software stage, queue, link, or protocol action produced either value. Queueing, retransmission, route changes, credit limits, packetization, channel count, and protocol transitions can bend the curve. Use the residual pattern and lower-level counters to decide which richer model or experiment is required.
Do not extrapolate far beyond the fitted rank count, message range, topology, or concurrency. A parameter obtained from one peer pair cannot predict an all-to-all that saturates a shared fabric cut.
T ≈ 2(P−1)α + 2(P−1)n/(P·BW)Use this as an expectation to test, not a substitute for the application's physical topology.
Worked example: When to combine small collectives
A backward pass launches 200 serialized 64 KiB all-reduces. A message sweep estimates 20 microseconds of startup and 5 microseconds of size-dependent cost at 64 KiB.
- Estimate the current serialized communication: 200 × (20 + 5) microseconds = 5 milliseconds before other dependency or contention effects.
- Calculate total payload: 200 × 64 KiB = 12.5 MiB. If one 12.5 MiB all-reduce sustains a 0.7-millisecond transmission and pays one 20-microsecond startup, the isolated estimate falls to about 0.72 milliseconds.
- Inspect readiness. The 200 gradient tensors become ready over 9 milliseconds of backward compute. One complete bucket cannot begin until the last gradient is ready, so nearly all communication becomes a post-backward tail.
- Test 1, 4, 16, and 64 MiB buckets. For each bucket, calculate startup count, readiness time, predicted transfer, and compute remaining until backward ends.
- Measure compute kernels alone and with communication. Smaller early buckets can contend with backward and lengthen it even while reducing the final communication tail.
- Select the size that minimizes complete step time and preserves optimizer semantics. Retain the readiness trace because a model architecture or compilation change can alter the result.
Conclusion: The alpha–beta model explains why aggregation helps, while the computation graph explains why unlimited aggregation can hurt.
Common errors
- Using one bandwidth value for every message size. Startup and protocol selection make completed payload bytes per second depend on message size.
- Fitting one line across an algorithm or transport transition. Use a piecewise model when the execution path changes.
- Summing communication bars without dependency analysis. Overlapped intervals are not independently removable from the critical path.
- Treating fitted alpha and beta as hardware constants. They depend on route, library, concurrency, operation, and measurement boundary.
Reference summary
T(message) ≈ α + nβWhen n is small, the startup term dominates and additional link bandwidth changes little. When n is large, nβ dominates and sustained payload bandwidth controls the result. A collective repeats or pipelines communication across several dependent rounds, so its cost model includes the round count, bytes per round, local reduction work, and topology.
- Fit α and β from a message-size sweep on the intended route. One measurement cannot separate startup from bandwidth.
- Aggregate small ready messages when saved startup time exceeds the overlap lost while waiting to fill the larger buffer.
- For large messages, use sustained payload bandwidth from a representative test rather than the raw signaling rate.
- Model hierarchy separately when local and global steps use different links and rates.
- Measure median and tail behavior under representative sharing because queueing and congestion make α and β vary over time.
Knowledge check
Which optimization helps more for 1 KB all-reduces: doubling link bandwidth or halving startup latency? What about 1 GB?
Show the answer guide
- For a 1 KiB message, nβ is usually small relative to the fitted startup α, so halving α has the larger predicted effect while doubling bandwidth changes little.
- For a 1 GiB message in a stable large-message regime, nβ usually dominates, so doubling sustained payload bandwidth has the larger predicted effect.
- Both conclusions are conditional on the operation, route, rank group, concurrency, message region, and fitted residuals; α and β are not universal hardware constants.
Fit a piecewise startup-plus-byte model
Benchmark one collective or peer transfer from 1 KiB through at least 1 GiB on one declared process group and route. Fit α and β only within regions that retain the same execution regime.
Evidence to keep: Raw samples, time and payload-bandwidth plots, fit ranges and coefficients with units, residuals, transition size, and one observed point where the two-term model stops fitting.
Collective algorithms arrange many alpha–beta exchanges into patterns that move and combine data across all ranks.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.