Fault Tolerance and Checkpointing

Imagine a seven-day training job that uses 512 devices, where each device is a graphics processing unit (GPU). One central processing unit (CPU) process controls each GPU. A healthy optimizer step takes 1.2 seconds, but one worker stops responding after 19 hours. The remaining workers wait in a collective for 11 minutes before a watchdog ends the job. Replacement allocation, process startup, and checkpoint restore take another 17 minutes, and the job repeats 43 minutes of training completed after its last usable checkpoint.

An incident timeline makes every interval observable. The launcher records process exit and replacement times. Application progress markers record the last completed step. NCCL reliability, availability, and serviceability status can identify an unresponsive process or inconsistent communicator state. Checkpoint metadata identifies the latest version whose shards were completely written and verified. The next valid optimizer step marks the end of recovery.

Healthy-step throughput excludes all of those intervals. Calendar progress includes checkpoint interference, failure detection, replacement, restore, warm-up, and repeated work. The job-level interruption rate must also be measured from completed jobs or incidents because one shared switch, storage service, or software rollout can interrupt many otherwise healthy devices at once.

This chapter turns the incident record into a checkpoint interval model, a portable state design, a progress-detection policy, and a resource budget. The final quantity is correct completed training work per allocated accelerator-hour over a representative operating period.