CloudOps Foundry
INC-004INCIDENT ACTIVE

Node NotReady

A GPU node flips to NotReady under load and every pod scheduled to it starts failing.

ENVIRONMENTKubernetes / Linux
DIFFICULTYIntermediate
STATUSIncident Active
INCIDENT IDINC-004
SYMPTOMS
$ kubectl get nodes
NAME          STATUS     ROLES    AGE
gpu-node-03   NotReady   <none>   14d

Mission

Bring the node back to Ready and identify why it dropped in the first place.

Skills tested

  • · Kubelet diagnostics
  • · systemd/journalctl
  • · Resource pressure analysis
  • · Control plane connectivity
POSSIBLE CAUSES

One or more of these is the real cause. The lab does not tell you which.

  • Kubelet crash or resource starvation
  • Container runtime hang
  • Disk pressure or PLEG (pod lifecycle event generator) failure
  • Network partition to the control plane
METHOD
01

Observe

Check node conditions and recent kubelet events.

02

Hypothesize

Separate a control-plane connectivity issue from a node-local failure.

03

Diagnose

Inspect kubelet logs, disk pressure, and PLEG status.

04

Fix

Resolve the underlying resource or connectivity failure.

05

Verify

Confirm the node returns to Ready and stays there under load.

Get the workload running. Then explain why it broke.

Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.

Request Lab Access