Hangs, health, and elastic restart

Define progress at several layers

Process liveness answers whether the operating system still sees the process. In this design, a worker-side watchdog thread emits a timestamped host heartbeat to the external job supervisor every five seconds; heartbeat freshness answers whether that CPU thread can run and communicate with the supervisor. A CUDA event polled by the watchdog records whether a declared GPU stream has retired new work. A framework collective-sequence record answers whether ranks launch and complete compatible communication. An application completed-step counter answers whether the job retains useful progress. Each producer, record, and boundary is different, so each signal excludes some causes but not all causes.

Make progress markers monotonic and attach timestamps and operation identities. Useful fields include current step and microbatch, last launched and completed collective sequence, collective type and element count, current input shape, last completed device event, checkpoint phase, hostname, device identifier, and topology location. A single message such as rank alive contains too little information for causal comparison.

Use operation-aware time bounds

A timeout is a claim about how long valid work can take. One fixed value is usually poor. A large checkpoint write, process-group initialization, expert all-to-all, and small gradient all-reduce have different distributions. Long sequences or compilation can also extend a step without indicating failure.

Record healthy duration distributions by phase, shape, message size, and scale. Set a warning bound before a termination bound. The warning can collect state while the operation still has an opportunity to complete. Keep an absolute upper bound for cases where progress counters move but useful work never completes. Review tail events because changing workloads can invalidate old thresholds.

Compare ranks by logical operation

When a collective stalls, collect the sequence number, collective kind, tensor size, group identity, and launch time from every member. If rank 11 reports all-reduce sequence 402 while rank 12 reports broadcast sequence 402, the process group has a contract mismatch. If all ranks report the same operation but one has not reached it, walk backward through that rank's preceding device, input, and host work.

Arrival skew and transfer failure require different repairs. Late arrival points to imbalance, input delay, or earlier computation. Similar arrival followed by no completions points more strongly to communication, device, or runtime state. Cross-rank timelines established in the profiling chapter provide the same comparison during incidents.

Capture evidence before teardown

On the warning path, gather recent structured logs, process stacks, device error reports, collective state, network counters, topology identity, and a bounded trace window. Time-stamp the collection. Preserve the earliest asynchronous device error and first rank divergence separately from later peer-abort messages.

NCCL RAS provides an out-of-band Transmission Control Protocol mechanism for querying communicator health and identifying missing or unresponsive processes in supported NCCL versions. RAS helps establish communicator state; RAS does not by itself prove whether the initiating cause was invalid collective order, a device fault, a network fault, or stalled application code. Combine RAS output with application and system evidence.

Treat the communicator as invalid after serious failure

A process group depends on agreement among members and on the state of communication and device work. After a fatal asynchronous error, abrupt member loss, or incompatible collective sequence, continuing with the same communicator can propagate invalid state or wait forever. The safe recovery boundary is commonly the whole affected process group, even when one rank first reported the problem.

Recovery can terminate the group, allocate or select resources, create a new membership, construct new ranks, and restore a known checkpoint. Under elastic launch, rank numbers are not stable across restarts. Application state and data partitioning must not assume that a global or local rank always maps to the same host.

Recent NCCL releases also provide communicator-shrink functionality for supported recovery designs. Removing failed ranks from communication does not validate the interrupted training iteration or reshard model and optimizer state automatically. The application must still define the rollback boundary, reconstruct compatible ownership, and prove that surviving buffers do not contain partial collective or optimizer results before continuing.

Control retries and side effects

Record resource identities and incident signatures across attempts. Quarantine a node, GPU, NIC, or route after repeated matching faults according to policy. Limit automated retries and escalate when failures repeat, checkpoint restore fails, or no clean replacement exists. Immediate retries on the same placement can consume more time than diagnosis or reallocation.

Training usually repeats computation after restore, but distributed applications can also write metrics, samples, evaluation records, or external status. Make these effects idempotent or associate them with checkpoint and step identities. A restarted attempt should not publish the same supposedly final result twice or advance an external cursor beyond recoverable state.