GPU Failures in Kubernetes: Handling Device Failures Without Endless Pending

The article highlights Meta's experience training Llama 3 405B, where the cluster of 16,384 H100 GPUs saw 419 interruptions, 58.7% of which were GPU-related. It points out that Kubernetes traditionally treats device failures like stateless service failures, attempting a restart on the same broken GPU, which leads to a crash loop. To address this, Kubernetes 1.36 introduces several mechanisms: DRA (Dynamic Resource Allocation) is stable since 1.34, providing an object-based model instead of a simple counter; prioritized list (firstAvailable) for backup devices; device health status in pod status; device taints and tolerations; and device binding conditions, among others. The author builds a test stand on VK Cloud Managed Kubernetes 1.36 with a GPU node and simulates a GPU failure by hot-removing the PCIe device. They show that the node reports the device as unhealthy and the allocatable count drops, but the pod keeps restarting on the dead GPU without any rescheduling. The article concludes by explaining why Kubernetes offers no built-in descheduling for CrashLoopBackOff pods and mentions DIY patterns like node health controllers that kill the entire node, which is a blunt instrument.