CloudOps Foundry
INC-009INCIDENT ACTIVE

Container Runtime Failure

New pods fail to start cluster-wide with runtime errors, though existing pods keep running.

ENVIRONMENTKubernetes / Linux
DIFFICULTYIntermediate
STATUSIncident Active
INCIDENT IDINC-009
SYMPTOMS
$ kubectl get events
Warning  FailedCreatePodSandBox  ... rpc error: code = Unknown
desc = failed to create containerd task

Mission

Restore pod creation cluster-wide without restarting the entire node fleet.

Skills tested

  • · Container runtime internals
  • · cgroup diagnostics
  • · systemd service management
POSSIBLE CAUSES

One or more of these is the real cause. The lab does not tell you which.

  • Container runtime service crashed or hung
  • Corrupted runtime state after a disk-full event
  • cgroup driver mismatch between kubelet and runtime
METHOD
01

Observe

Confirm existing containers run fine but new ones fail to start.

02

Hypothesize

Suspect the runtime daemon or its state rather than the scheduler.

03

Diagnose

Inspect runtime logs, disk state, and cgroup driver configuration.

04

Fix

Recover or restart the runtime cleanly, addressing the root state issue.

05

Verify

Confirm new pods schedule and start normally.

Get the workload running. Then explain why it broke.

Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.

Request Lab Access