INC-005INCIDENT ACTIVE
GPU Utilization at 12%
A training job runs without errors, but GPUs sit at 12% utilization while the job crawls.
ENVIRONMENTKubernetes / NVIDIA / Training
DIFFICULTYAdvanced
STATUSIncident Active
INCIDENT IDINC-005
SYMPTOMS
$ nvidia-smi
GPU-Util: 12% Memory-Usage: 3204MiB / 40960MiBMission
Find the real bottleneck — the GPU is not the problem.
Skills tested
- · Performance profiling
- · Storage throughput analysis
- · Data pipeline diagnostics
- · GPU topology
POSSIBLE CAUSES
One or more of these is the real cause. The lab does not tell you which.
- Data loader bottleneck starving the GPU
- Storage throughput too slow to keep the pipeline fed
- Incorrect batch size or precision configuration
- PCIe or NVLink topology misconfiguration
METHOD
01
Observe
Confirm the GPU is healthy but idle relative to the workload.
02
Hypothesize
Suspect the data path rather than the compute path.
03
Diagnose
Profile the data loader and storage throughput feeding the GPU.
04
Fix
Relieve the bottleneck upstream of the GPU.
05
Verify
Confirm utilization and throughput both recover.
Get the workload running. Then explain why it broke.
Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.