CloudOps Foundry
INC-002INCIDENT ACTIVE

GPU Pod Stuck Pending

A training pod requests a GPU and sits in Pending with zero eligible nodes.

ENVIRONMENTKubernetes / NVIDIA
DIFFICULTYIntermediate
STATUSIncident Active
INCIDENT IDINC-002
SYMPTOMS
$ kubectl get pods
NAME        READY   STATUS    RESTARTS   AGE
trainer-0   0/1     Pending   0          4m

$ kubectl describe pod trainer-0
Warning  FailedScheduling  0/8 nodes are available

Mission

Get the workload running without changing the application.

Skills tested

  • · Kubernetes scheduling
  • · GPU resource requests
  • · Device plugins
  • · Node labels
  • · Taints and tolerations
POSSIBLE CAUSES

One or more of these is the real cause. The lab does not tell you which.

  • Missing GPU resources on candidate nodes
  • Device plugin failure or crash loop
  • Taints without matching tolerations
  • Resource exhaustion on GPU nodes
  • Node labels not matching nodeSelector/affinity rules
METHOD
01

Observe

Read the scheduler's FailedScheduling event carefully — it names the constraint.

02

Hypothesize

Decide whether this is a capacity, taint, or label problem.

03

Diagnose

Cross-reference node taints, labels, and allocatable GPU counts.

04

Fix

Resolve the scheduling constraint at the infrastructure layer.

05

Verify

Confirm the pod transitions to Running without a spec change.

Get the workload running. Then explain why it broke.

Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.

Request Lab Access