INC-001INCIDENT ACTIVE
GPU Not Found
A GPU node joins the cluster, but no GPU resources ever appear on it.
ENVIRONMENTKubernetes / NVIDIA
DIFFICULTYBeginner
STATUSIncident Active
INCIDENT IDINC-001
SYMPTOMS
$ kubectl describe node gpu-node-03 | grep nvidia
$ # (no output)Mission
Make the GPU visible to Kubernetes as an allocatable resource — without rebuilding the node.
Skills tested
- · Linux driver diagnostics
- · Container runtime configuration
- · Kubernetes device plugins
- · Node labeling
POSSIBLE CAUSES
One or more of these is the real cause. The lab does not tell you which.
- NVIDIA driver not installed or mismatched with kernel
- Container runtime missing GPU support
- Device plugin DaemonSet not scheduled on the node
- Node labels missing for GPU discovery
METHOD
01
Observe
Confirm the GPU is visible to the OS but not to Kubernetes.
02
Hypothesize
Narrow down driver, runtime, or device plugin as the likely layer.
03
Diagnose
Inspect driver version, runtime config, and device plugin logs.
04
Fix
Correct the broken layer without touching the workload.
05
Verify
Confirm the GPU appears as an allocatable resource on the node.
Get the workload running. Then explain why it broke.
Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.