Failure cost model
Calculate checkpoint overhead, expected repeated work, and recovery waste across candidate useful-training intervals between checkpoint pauses and measured incident ranges
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Ability to calculate a mean interval between complete-job interruptions
- Measured healthy-step, exposed-checkpoint, detection, and restore durations
A 256-GPU job completes one optimizer step every 1.5 seconds and begins a verified checkpoint after every 60 minutes of useful training. The save pauses useful training for 90 seconds. After a worker failure 47 minutes into the next useful-training interval, detection and restore consume 18 minutes and the job repeats those 47 minutes of steps. Healthy-step speed alone omits every one of those lost calendar intervals.
Build an incident table from scheduler records, application progress markers, and checkpoint metadata. For each interruption, record the last completed step, the age of the newest verified checkpoint, first fault indication, detection time, replacement time, restore time, and first valid step after recovery. Job-level intervals between interruptions are the relevant samples because any required rank or shared dependency can stop synchronous progress.
Let C be exposed checkpoint seconds. Let I be scheduled seconds of useful training from the end of one checkpoint pause to the start of the next checkpoint pause; the separate pause C is not part of I. Let M be mean job-level seconds between interruptions, D be detection seconds, and R be restart and restore seconds. Under a simple periodic model with C small relative to I, checkpoint overhead is approximately C/I, repeated useful work contributes approximately I/(2M), and fixed detection plus recovery contributes approximately (D+R)/M to expected fractional waste.
The model provides a starting checkpoint cadence, not a promise. Correlated failures, storage congestion, asynchronous save interference, invalid checkpoints, and nonuniform failure timing violate its assumptions. Evaluate nearby intervals with measured distributions and compare correct completed tokens per allocated accelerator-hour over a representative operating period.
Use job-level interruption data
A synchronous job can stop when one required GPU, host process, network-interface path, storage service, scheduler allocation, or shared dependency fails. Reliability of any one device does not directly equal the time between complete-job interruptions. Measure completed job intervals and classify the initiating event.
Independent GPU, process, host, and network-path failures make interruption probability grow with the number of required resources. Shared switches, power, software rollout, or storage can create correlated failures that violate an independence model. Keep isolated and correlated categories separate and use empirical job data when enough history exists.
Calculate checkpoint overhead
Let C be exposed checkpoint time and let I be useful-training time from the end of one checkpoint pause to the start of the next. One cycle consumes I + C wall-clock time, so the exact pause fraction for that cycle is C/(I+C). The simpler C/I term is the first-order approximation used in this lesson when C is small relative to I. If asynchronous saving slows each step instead of pausing completely, convert that slowdown and staging pause into an equivalent exposed C before using the same definition of I.
Checkpoint duration is a distribution. Shared storage load can make tail saves much slower. Measure at the intended rank count and include staging buffers, device-to-host transfer, metadata, and completion publication.
Calculate lost work
Under periodic checkpoints and a failure time uniformly distributed across the I seconds of useful training, expected useful progress since the last verified checkpoint is I/2. Progress after that checkpoint must be repeated. The approximation changes when failures also occur during the checkpoint pause, failures correlate with checkpointing, hazard varies with time, or some progress can be recovered from local state.
Use the age of the last verified checkpoint, not the last attempted save. A partial or corrupt checkpoint cannot reduce repeated work. Incident records should report actual rolled-back steps or tokens.
Add detection and restart
A hang can consume allocated resources until a watchdog declares missing progress. Detection delay D is separate from restart and restore time R. Restart can include scheduling a replacement, starting processes, reconstructing communicators, reading shards, compiling graphs, warming kernels, and restoring input state.
Expected fractional loss from these fixed per-failure costs is approximately (D+R)/M for mean job interruption interval M. A faster checkpoint does not reduce a 25-minute detection delay or a 15-minute resource queue.
Derive the simple interval
Ignoring fixed recovery cost because it does not depend on I, approximate interval-dependent waste as C/I + I/(2M), where I retains the definition of useful-training seconds between checkpoint pauses. Differentiate with respect to I and set the derivative to zero: −C/I² + 1/(2M) = 0. The resulting useful-training interval is sqrt(2CM).
The square-root result is a starting estimate. The estimate assumes periodic synchronous checkpoints, memoryless interruption timing, constant C, and complete rollback to the prior checkpoint. Evaluate nearby intervals with measured distributions and storage contention.
Measure calendar useful progress
Healthy tokens per second applies only while the job advances. Calendar useful rate divides correct retained progress by total allocation time, including checkpoint interference, failures, detection, replacement, restore, warm-up, and repeated work.
A configuration with slightly slower healthy steps can win if it saves faster, fails less often, detects faults sooner, or recovers on a smaller failure domain. Use time to a training-quality target when resume semantics or batch order can change convergence.
Test sensitivity and tail risk
Checkpoint interval depends on estimates that can change with cluster size and system load. Calculate several M, C, D, and R scenarios. Include correlated incident cases separately because one rack or storage event can invalidate many local checkpoints or delay every restart.
Completion deadlines can depend more on tail recovery than mean waste. Report the probability or observed rate of finishing within the intended window, not only the average calendar efficiency.
Check the approximation against event data
For each interruption, record time since the last verified checkpoint, actual repeated work, detection delay, allocation delay, restore time, warm-up, and first valid step. Compare observed weekly or job-level loss with the formula's prediction. Large residuals indicate nonuniform failure timing, overlapping costs, invalid saves, or omitted incident classes.
Use simulation when save durations, interruption hazards, or resource delays have important distributions. The square-root interval remains a reference point, but the operational policy should be selected from measured completion and cost outcomes.
Worked example: Comparing useful-training intervals between checkpoint pauses
A job has a measured mean interruption interval of six hours. A checkpoint exposes a four-minute pause. Candidate I values denote useful-training minutes between the end of one pause and the start of the next. Detection averages five minutes, and resource replacement plus restore averages ten minutes.
- Convert to one unit: C = 4 minutes, M = 360 minutes, D+R = 15 minutes. The simple optimum is sqrt(2×4×360), about 54 useful-training minutes between checkpoint pauses.
- For I = 60 useful-training minutes, the first-order checkpoint term is C/I = 4/60 = 6.67%. Expected rollback is I/(2M) = 60/(2×360) = 8.33%. Detection and restore add 15/360 = 4.17%. Approximate total waste is 19.17%.
- For I = 120 useful-training minutes, the first-order checkpoint term is 3.33%, rollback is 16.67%, and fixed recovery remains 4.17%. Approximate total is 24.17%.
- For I = 45 useful-training minutes, the first-order checkpoint term is 8.89%, rollback is 6.25%, and recovery is 4.17%, totaling about 19.31%. The one-hour useful-training interval is close to the simple optimum under these assumptions.
- Now evaluate asynchronous saving. Suppose exposed pause falls to one minute but training runs 4% slower for ten minutes because of device-to-host and storage traffic. Convert both effects to equivalent C before recalculating.
- Compare the model with incident logs: checkpoint age at failure, actual repeated steps, detection tail, restore duration, and correlated events. Select an operational interval and monitor whether its assumptions remain valid.
Conclusion: The best interval is an economic balance, not the most frequent checkpoint the storage system can sustain.
Common errors
- Using one device's mean time between failures as job-level M. The number of required GPUs, hosts, network paths, and shared services changes complete-job interruption frequency.
- Using I/2 expected rollback without defining I as useful-training time between checkpoint pauses, stating the uniform-failure assumption, or verifying checkpoint validity.
- Calling asynchronous saving free. Staging buffers, copy traffic, CPU work, and storage can slow healthy execution.
- Optimizing healthy-step time without detection and recovery. Measure retained useful progress over calendar allocation.
Reference summary
Healthy-step throughput is only one part of long-duration progress. Checkpoint pauses, detection delay, resource replacement, restore, warm-up, and work repeated after the last completed checkpoint all consume allocated time. Measure failures at the job level because a synchronous job can stop when any required rank or shared service fails.
expected fractional waste ≈ C/I + I/(2M) + (D+R)/M- Estimate job-level interruptions per calendar hour from incident data. Separate isolated GPU, process, host, and network-path faults from correlated rack, power, storage, and software events.
- Measure exposed checkpoint time and any concurrent slowdown under representative storage load.
- Include detection, replacement allocation, process initialization, restore, graph compilation, and input replay in recovery time.
- Track the age of the last verified checkpoint at failure and the amount of work actually repeated.
- Optimize useful training progress per calendar day or probability of completion by a deadline, not only healthy-step time.
Knowledge check
Why does a 2× faster checkpoint not necessarily justify checkpointing 2× as often?
Show the answer guide
- A twofold smaller C lowers the checkpoint-pause term, but checkpointing twice as often also changes the useful-training interval I, startup count, storage load, and asynchronous interference.
- Detection and restart terms do not shrink merely because checkpoint writing is faster, and correlated incidents or invalid checkpoints can dominate actual loss.
- Evaluate candidate I values with the consistently defined useful-training interval, measured C distributions, job-level M, D, R, actual rollback, accelerator-hours or dollars, and deadline tails.
Build a checkpoint-interval sensitivity table
Evaluate I = 10, 20, 40, 60, and 120 useful-training minutes for at least three job-level interruption estimates. Include one asynchronous-save scenario by converting its pause and slowdown to an equivalent C.
Evidence to keep: Declared definitions and units for C, I, M, D, and R; exact and first-order pause terms; rollback and recovery terms; accelerator-hour effects; and a selected interval with assumptions and rejection conditions.
Checkpoint economics determines when to save. Distributed checkpoint design determines whether state can be written, moved, and restored without recreating a full-model bottleneck.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.