PyTorch 2.15 tackles the 'one bad GPU' problem

PyTorch 2.15 rewrites its NCCL backend, the layer that handles GPU-to-GPU communication, to address "one bad GPU" failures in training clusters. The new design allows one communicator per process group, a faulty or slow GPU can be excluded and replaced by an external orchestrator, and a "flight recorder" aids diagnosis.…