Distributed efficiency metrics
Reconstruct model FLOP utilization and scaling efficiency from declared work, elapsed seconds, hardware count, datatype peak, and shape assumptions
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Chapter 1 floating-point-operation and useful-work accounting
- Ability to distinguish fixed-global-work from fixed-per-rank-work experiments
- Chapter 3 training-step phases
A 64-GPU training run completes 2.0 million valid tokens per second. A report also claims 52 percent model floating-point-operation utilization. Before the percentage can guide a decision, a reader needs the model-arithmetic convention, step duration, GPU count, applicable datatype peak, and treatment of padding, recomputation, mixture-of-experts routing, and optimizer work.
Collect the numerator from declared model shapes and valid-token counts, then collect wall-clock step time from synchronized steady-state measurements. Model floating-point-operation utilization, commonly abbreviated MFU, divides estimated model-required floating-point operations per step by step seconds and the relevant aggregate hardware peak. A separate hardware counter can estimate executed arithmetic, including recomputation and unnecessary padded work, so the two percentages can move in opposite directions without a contradiction.
Record valid tokens per second beside every MFU value. For scaling, compare the 64-GPU run with the smallest valid baseline while holding global useful work fixed. A fixed-work comparison measures strong scaling. A comparison that keeps local work fixed and increases the global batch measures weak scaling. Local tensor shapes must accompany either result because adding ranks can make each matrix multiplication smaller and less efficient.
Efficiency is therefore not one health score. GPU-active time reports whether GPU work was present. Scaling efficiency reports speedup relative to resource growth. Goodput reports correct useful work that satisfies the declared constraints. Calendar efficiency includes checkpoints and failures. Each rate states its completed quantity, elapsed-time boundary, and qualification for useful completion.
Define the useful-work numerator
Model FLOP estimates begin from the executed architecture and valid tokens or samples. State the convention for multiply-add, attention, embeddings, backward, activation recomputation, and mixture-of-experts routing. If padding executes but does not advance training on real data, separate executed padding from useful model work.
Tokens per second is easier to audit and should accompany a model arithmetic estimate. When sequence lengths vary, report valid, padded, and dropped tokens. When quality or convergence changes, use progress to a fixed target rather than assuming that each token has equivalent value.
Choose a compatible peak denominator
Select the peak for the actual hardware count, datatype, sparsity convention, and instruction path. A tensor-core peak is not applicable to operations that execute on other pipelines. Power limits, unavailable devices, or heterogeneous hardware also change the relevant denominator.
Advertised peak provides a ceiling. A measured per-device reference can provide a more practical comparison, but it must use compatible shapes and work. Report which denominator is used instead of presenting one percentage without a recoverable calculation.
Separate MFU from executed hardware utilization
MFU estimates the rate of model-required arithmetic divided by total relevant peak. Hardware FLOP utilization estimates executed arithmetic, which can include recomputation, padding, redundant work, or communication kernels. Executed utilization can rise while useful model work remains flat or falls.
A high duty cycle also does not guarantee useful arithmetic. GPUs can remain active running small inefficient kernels or communication. Use these metrics as views of different quantities, not interchangeable names for efficiency.
Calculate strong-scaling efficiency
For fixed global work, speedup at P devices is T1/TP relative to a baseline T1, and ideal speedup is P. Strong-scaling efficiency is (T1/TP)/P, or throughput_P / (P × throughput_1). The parentheses state that aggregate scaled throughput is compared with P copies of baseline throughput. Use a strong baseline that executes the same model and global work efficiently.
A one-device baseline may not fit. Use the smallest valid group and define incremental efficiency relative to it. Keep local and global shapes visible because the baseline and scaled configuration can use different kernel regimes.
Distinguish weak scaling
Weak scaling keeps per-device work approximately fixed while global work increases with P. Stable step time can then produce linear global throughput. This answers whether the system can process proportionally more work, not whether it completes the original global batch faster.
Larger models can also create larger local GEMMs and higher MFU, as reported in Megatron-LM scaling work. Efficiency can improve because the workload becomes more hardware-efficient, not because communication overhead vanished. State the changed problem.
Reconcile the step with wall time
Build categories that explain the critical path: required compute, extra executed compute, exposed communication, pipeline fill/drain, stage or rank imbalance, host and input gaps, checkpoint interference, and unclassified time. Overlapped operations can appear in more than one resource view but must not be added twice to elapsed time.
Within compute time, compare achieved rates with shape-specific roofs. Within communication, separate arrival skew from service and hidden from exposed time. The decomposition identifies where to investigate; the utilization ratio alone cannot.
Report uncertainty and measurement scope
Arithmetic counts can be analytical estimates, profiler-derived execution counts, or framework conventions. Peak rates can be advertised ceilings or measured references. Time samples can cover a short steady window or complete calendar operation. Label each source and provide the formula and interval so another reader can reconstruct the ratio.
Report distributions when workload shapes or incidents vary. A median step metric can hide rare stalls that dominate completion time, while one maximum can overstate normal behavior. Include the number of devices and samples, warm-up exclusions, synchronization boundary, and confidence or observed spread.
Worked example: Interpreting a 31% utilization claim
A 20,000-GPU training run reports 31% model FLOP utilization. A review asks whether the remaining 69 percentage points are network overhead.
- Reconstruct the numerator from architecture, valid tokens, and forward/backward convention. The published value excludes activation recomputation and padding, so those executed operations can lower MFU without creating idle time.
- Reconstruct the denominator from actual accelerator count and the dense BF16 peak used by the convention. Verify power mode, unavailable ranks, and whether all modeled arithmetic can use that path.
- Calculate useful token throughput beside MFU. Compare with valid-token rate before the deployment so a packing or padding change cannot hide in the arithmetic estimate.
- Align representative traces. Reconcile the step into 43% model compute, 9% recomputation and padding, 14% exposed communication, 18% pipeline and rank imbalance, 7% host/input gaps, 4% checkpoint interference averaged over time, and 5% unclassified.
- Within the 43% model-compute interval, inspect local GEMM shapes and elementwise kernels. Achieved compute remains below peak even when no communication is exposed, so the 69-point difference cannot be labeled network time.
- Investigate the 14% exposed communication through arrival and transfer analysis, and the 18% imbalance through stage and per-rank maxima. Retain the unclassified 5% as an explicit measurement gap.
Conclusion: The percentage begins the investigation; it cannot identify a network bottleneck without a reconciled time and work decomposition.
Common errors
- Subtracting MFU from 100% and naming the remainder communication. The ratio does not partition elapsed time.
- Using an incompatible peak for the executed datatype or instruction path.
- Changing global batch and calling the result strong-scaling efficiency. Useful global work must remain fixed.
- Counting padded, dropped, or repeated tokens as useful without disclosure. Executed work and training progress use different numerators.
Reference summary
MFU = model-required FLOP per step / (step time × total relevant peak FLOP/s)strong-scaling efficiency = (T₁/Tₚ)/P = throughputₚ / (P × throughput₁)Model FLOP utilization compares an estimate of required model arithmetic with a hardware peak. The numerator normally excludes padding, recomputation, communication kernels, optimizer work, and other executed operations unless the convention states otherwise. Hardware FLOP measurements can include those operations. Report the numerator convention, datatype-specific peak, hardware count, and workload beside the ratio.
- Report tokens or samples per second with MFU so the reader can recover the useful-work rate.
- Distinguish model-required arithmetic from executed arithmetic that includes padding, recomputation, or redundant work.
- Use strong-scaling efficiency only when global useful work is fixed. Use weak-scaling language when per-rank work stays fixed and global work increases.
- Record local tensor shapes because a larger run can change matrix efficiency as well as communication.
- Use time to a fixed quality target when a change affects numerical behavior, batch semantics, or convergence.
- Reconcile wall time into critical compute, exposed communication, bubbles, host/input delay, checkpointing, and unclassified intervals.
Knowledge check
How can hardware FLOP utilization rise while MFU falls?
Show the answer guide
- Hardware FLOP utilization can rise because the GPUs execute more recomputation, padded work, redundant operations, or communication arithmetic even when useful model-required work does not increase.
- MFU can fall at the same time when valid model FLOP per second declines relative to aggregate relevant peak, for example because extra executed work lengthens the step.
- The report must state both numerator conventions, datatype peak, device count, valid token or sample rate, and wall-clock boundary; neither percentage alone partitions elapsed time.
Reconstruct distributed efficiency metrics
For one fixed-global-work run and baseline, calculate valid tokens per second, MFU, executed hardware FLOP utilization where available, speedup, and strong-scaling efficiency with explicit parentheses.
Evidence to keep: Model work derivation, valid and padded token counts, datatype peak and device count, raw step samples, formulas with units, local shapes, and a reconciled wall-time table.
Efficiency metrics quantify the performance difference. Time-aligned traces identify the rank and dependency that cause the difference on the critical path.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.