Reliability resource budget
Compare reliability designs using correct completed work per charged calendar allocation and explicit prevention, detection, recovery, and integrity objectives
Before you begin
Follow a linked prerequisite when you cannot explain it from memory. Background skills without links are stated as abilities rather than hidden terminology.
- Previous sections' interruption, checkpoint, and recovery measurements
- Ability to state a maximum repeated-work or recovery-time objective
A seven-day training allocation has an explicit completion deadline. One proposed configuration uses every byte of device memory and every available worker for healthy steps. A second configuration reserves 3 GiB per GPU for checkpoint staging, keeps one compatible spare server available, and runs health probes. The second configuration has lower peak steady-state throughput but can recover faster and avoid failed checkpoint saves.
Measure the costs instead of classifying reliability as free overhead. Device-memory snapshots show checkpoint-buffer peaks. Network and storage telemetry show save traffic. Launcher records show spare allocation and replacement time. Restore drills show the distribution from failure detection to the first valid post-restore step. Incident records show repeated steps or tokens.
A reliability budget allocates compute, memory, storage, network capacity, and operator attention to declared objectives. Example objectives include no more than 30 minutes of repeated training, a 95th-percentile recovery time below 20 minutes, and a stated probability of completing inside the allocation window. A short exploratory run and a multi-week production run can rationally choose different budgets.
Evaluate both configurations using correct completed work divided by charged accelerator-hours or total dollars over the calendar window. Include active accelerators during healthy execution, checkpoint interference, detection, replacement, restore, warm-up, repeated work, and failed retries, plus idle spare capacity charged to the workload. Include integrity as a completion condition; a fast continuation from inconsistent optimizer state is not useful progress.
State four reliability objectives
A repeated-work objective is analogous to a recovery-point objective: after an interruption, how many minutes or steps may need to run again? A recovery-time objective bounds detection, replacement, restore, and warm-up. A completion objective gives the required probability of reaching a quality target before a deadline. An integrity objective specifies the checks needed to accept restored and final state.
Use distributions and percentiles where tails matter. An average ten-minute restore does not support a 30-minute recovery target if one in ten restores takes two hours. Define whether the objective applies to one incident, an entire job, or a class of jobs. Record exceptions for exploratory or preemptible work rather than weakening the production objective implicitly.
Allocate across the incident lifecycle
Prevention includes input validation, burn-in, topology screening, and software checks. Detection includes progress markers, device and network errors, and communicator health. Containment stops invalid members and prevents partial output from being published. Recovery creates membership, restores state, and validates resumed execution. Learning retains evidence and changes tests, placement, or mechanisms after incidents.
Spending only on frequent checkpoints leaves long detection and replacement delays unchanged. Spending only on spare capacity does not protect against invalid checkpoint state. For each objective, identify which stage dominates its observed miss. The next investment should reduce that stage or the variance that makes its tail unacceptable.
Assign each mechanism a measurable cost
For checkpointing, record exposed pause, background slowdown, staging memory, storage volume, and restore time. For health detection, record probe overhead, warning frequency, false terminations, and evidence volume. For spare capacity, record reserved accelerator-hours and the distribution of replacement delay avoided. For redundancy, include synchronized communication and the failure correlations that remain.
Expected loss, mechanism cost, and recovery time permit comparison in a common unit such as expected accelerator-hours, probability of deadline completion, or cost per retained training token. Engineering time can also be included when comparing a complex mechanism with a simpler operational change. The estimates need ranges because incident frequency and correlated failures are uncertain.
Model fault domains and capacity after failure
A fault domain is the set of work affected by one event. A fault domain can be a process, GPU, host, rack, network plane, storage service, software release, or shared scheduler. Replicas or checkpoints placed inside the same domain do not protect against its failure. List which domains each mechanism covers and which remain common.
Training recovery may require the original world size, allow a different mesh, or continue at lower capacity. Each choice changes queue delay, resharding requirements, step time, and convergence semantics. Service recovery must consider whether remaining replicas can meet latency under redistributed load. Headroom is useful only if it exists in the needed domain and can be activated inside the recovery target.
Calculate effective calendar rate
Let U be retained correct work and A be charged accelerator-hours over the selected calendar window. The effective resource rate is U/A. Calculate A by summing each active or reserved accelerator's charged hours across healthy execution, checkpoint interference, detected and undetected stalls, replacement, restore, warm-up, repeated work, failed retries, and idle spare reservation. When device prices differ, convert each allocation to dollars and report U divided by total dollars instead. The categories must reconcile with the billable resource total, not only with wall-clock minutes.
A change that slows healthy steps by 1% can improve effective rate if it prevents enough failures or reduces recovery. Conversely, a 3% kernel improvement can have little calendar value when the job repeatedly waits hours for replacement. Report both healthy rate and effective rate so that changes to steady execution and incident behavior remain visible.
Compare expected loss and completion tails
Expected loss is useful for recurring capacity planning: incident rate multiplied by detection, rollback, and recovery cost, plus planned mechanism overhead. Completion planning also needs event combinations and tails. Two independent-looking failures may occur in one job, while one correlated event can remove several nodes and the latest checkpoint store.
Evaluate scenarios with measured or bounded incidents per calendar hour: isolated GPU or process loss, repeated fault on replacement, rack or fabric event, invalid checkpoint, and storage delay. For each scenario, calculate the work repeated, resources unavailable, and deadline effect. The model is a decision aid, not proof that future failures follow the same distribution.
Retain evidence and revise the budget
For every incident, preserve timestamps for first fault indication, first detection, termination, replacement, restore start and completion, first valid step, and return to normal completed-steps-per-second rate. Preserve checkpoint age, failed process or hardware identity, retries, and whether diagnostics identified a cause. Aggregate mean and tail values without discarding rare high-cost events.
Review objectives and costs when scale, topology, checkpoint size, model architecture, or operational policy changes. Larger jobs generally encounter device, process, and shared-infrastructure events more often and can enlarge the number of ranks affected by one failure. A budget calibrated on 64 devices should not be copied unchanged to thousands of devices.
Worked example: Valuing faster detection
A 512-GPU training job fails twice per week. Each hang takes 25 minutes to detect, while restart and restore take 12 minutes. For the illustrative cost conversion, each allocated GPU-hour costs $2.50.
- Detection consumes 2 × 25 × 512 / 60 = 426.67 accelerator-hours per week, or about $1,066.67 at $2.50 per accelerator-hour. Restart and restore consume another 2 × 12 × 512 / 60 = 204.8 accelerator-hours, or $512.00. Rollback and planned checkpoint cost must be added separately.
- A phase- and shape-aware watchdog reduces detection from 25 to 3 minutes. If incident frequency stays at two per week, the saved allocation is 2 × 22 × 512 / 60 = 375.47 accelerator-hours per week, worth about $938.67 at the illustrative rate.
- Suppose the watchdog falsely terminates one healthy job every eight weeks. Each false termination costs 12 minutes of restart and restore plus 20 minutes of repeated work. The expected weekly cost is (32/8) × 512 / 60 = 34.13 accelerator-hours, or about $85.33. Before probe overhead, the expected net saving is 375.47 − 34.13 = 341.34 accelerator-hours, or about $853.34 per week.
- Check the tail objective. A warning at two minutes collects stacks and communicator state; termination at three minutes occurs only when both useful-step and lower-level progress bounds are exceeded. Fault injection verifies detection, evidence retention, whole-group termination, and successful restore.
- After deployment, measure incident and false-termination rates, p50 and p99 detection, restore duration, repeated work, and first-step validation. Recalculate if the workload begins using longer sequences or checkpoint restore time changes.
Conclusion: The detection change is justified by expected calendar savings only if its false-termination cost, probe overhead, recovery correctness, and tail behavior remain inside the stated objectives.
Common errors
- Treating reliability as binary correctness. Recovery time and repeated work are continuous performance costs.
- Maximizing redundancy without an economic model. Spare capacity should map to an availability or completion objective.
- Measuring only healthy windows. Effective throughput must include checkpoints, incidents, and recovery.
- Using a device failure rate as the complete job model. Software, shared fabric, storage, membership, and correlated events create other interruption modes.
- Reducing detection time by discarding all incident evidence. Faster termination can increase future loss when repeated causes cannot be identified.
Reference summary
Reliability mechanisms consume compute, memory, storage, network capacity, and operator time. Checkpoint buffers can raise the device-memory peak. Spare nodes can remain idle. Health probes and logging use host and network resources. Checkpoint, spare-capacity, and monitoring costs are justified when they reduce expected lost progress or help the system meet a recovery objective.
effective useful rate = correct completed work / charged accelerator-hoursDefine separate budgets for prevention, detection, containment, recovery, and incident learning. Connect each budget to a measurable objective such as maximum repeated work, p95 recovery time, probability of finishing within a window, or service capacity after one failure domain is lost.
Knowledge check
Give two ways that operating at maximum memory occupancy can reduce effective long-run throughput.
Show the answer guide
- Maximum memory occupancy can leave no space for checkpoint staging or recovery buffers, causing saves or restores to fail even when healthy steps fit.
- Removing memory and compute headroom can also prevent graceful failover, force long replacement queues, increase fragmentation or allocation stalls, and make tail incidents consume more charged accelerator-hours.
- Compare correct retained work per charged accelerator-hour or dollar, including spares, checkpoints, detection, recovery, repeated work, and false termination, rather than selecting peak healthy-step throughput.
Price a reliability mechanism
Compare a baseline and one reliability change such as faster detection, spare capacity, or checkpoint staging for a declared accelerator count and price. Include at least one false-positive or correlated-failure scenario.
Evidence to keep: Incident assumptions, accelerator-hour and dollar equations, healthy and effective useful rates, recovery-tail objective, probe or reserve overhead, sensitivity range, and a decision with a condition that would reverse it.
The distributed system can now train and recover. Part V turns to continuous inference, where requests, caches, schedulers, and service objectives reshape the same hardware constraints.
Sources and further reading
The lesson explanations synthesize primary papers and official references. Follow the originals for proofs, exact APIs, architecture details, and version-sensitive behavior.