Kubernetes 1.36: Handling GPU Device Failures without Infinite Pending

The article highlights that Kubernetes, by default, handles device failures by simply restarting the container, which is insufficient for GPU-accelerated workloads. Citing a Meta study on training Llama 3 405B, it notes that clusters with thousands of GPUs experience frequent interruptions, often due to GPU and memory failures. The author explains the limitations of the classic device plugin approach, which only reduces the allocatable count, and contrasts it with the newer DRA (Dynamic Resource Allocation) model that provides object-based device management. On a test cluster, the author simulates a GPU failure by hot-removing a device from the PCIe bus, resulting in Xid 79. The pod then enters a crash loop, restarting on the dead device, and the scheduler does not reschedule it because it is still bound to the node. Kubernetes 1.36 introduces several beta features to address this: device health status in the pod status, device taints and tolerations, and partitionable devices. Using these, the cluster can be made to recognize the failure and schedule the pod onto a healthy GPU, avoiding endless restarts.