FAILURE LAB

Production incidents. Recreated deliberately.

Students receive a working environment. Then something breaks. They must identify the symptoms, collect evidence, form a hypothesis, diagnose the root cause, repair the system, and verify recovery.

INC-001INCIDENT ACTIVE

GPU NOT FOUND

Kubernetes / NVIDIA

A GPU node joins the cluster, but no GPU resources ever appear on it.

SYMPTOMS

· NVIDIA driver not installed or mismatched with kernel
· Container runtime missing GPU support
· Device plugin DaemonSet not scheduled on the node

BEGINNEREnter the Lab

INC-002INCIDENT ACTIVE

GPU POD STUCK PENDING

Kubernetes / NVIDIA

A training pod requests a GPU and sits in Pending with zero eligible nodes.

SYMPTOMS

· Missing GPU resources on candidate nodes
· Device plugin failure or crash loop
· Taints without matching tolerations

INTERMEDIATEEnter the Lab

INC-003INCIDENT ACTIVE

CUDA VERSION MISMATCH

Linux / CUDA / Containers

A container that worked yesterday now fails with a CUDA driver/runtime version error after a host update.

SYMPTOMS

· Host driver upgraded without matching CUDA toolkit in the image
· Container image pinned to an incompatible CUDA version
· Multiple CUDA runtimes installed on the host

INTERMEDIATEEnter the Lab

INC-004INCIDENT ACTIVE

NODE NOTREADY

Kubernetes / Linux

A GPU node flips to NotReady under load and every pod scheduled to it starts failing.

SYMPTOMS

· Kubelet crash or resource starvation
· Container runtime hang
· Disk pressure or PLEG (pod lifecycle event generator) failure

INTERMEDIATEEnter the Lab

INC-005INCIDENT ACTIVE

GPU UTILIZATION AT 12%

Kubernetes / NVIDIA / Training

A training job runs without errors, but GPUs sit at 12% utilization while the job crawls.

SYMPTOMS

· Data loader bottleneck starving the GPU
· Storage throughput too slow to keep the pipeline fed
· Incorrect batch size or precision configuration

ADVANCEDEnter the Lab

INC-006INCIDENT ACTIVE

INFERENCE LATENCY SPIKE

Kubernetes / NVIDIA / Serving

p99 inference latency jumps 8x with no deployment change and no traffic spike.

SYMPTOMS

· GPU memory fragmentation from long-running processes
· Noisy-neighbor contention in a multi-tenant cluster
· Batching configuration drifted from expected traffic shape

ADVANCEDEnter the Lab

INC-007INCIDENT ACTIVE

STORAGE THROUGHPUT COLLAPSE

Kubernetes / Ceph / Storage

Read throughput to the training dataset falls from 4 GB/s to under 200 MB/s overnight.

SYMPTOMS

· Backing storage cluster rebalancing or degraded
· Client-side caching misconfigured after an upgrade
· Network path saturation between compute and storage

ADVANCEDEnter the Lab

INC-008INCIDENT ACTIVE

RDMA UNAVAILABLE

Linux / Networking / GPU

Multi-node training falls back to TCP after RDMA silently stops being available.

SYMPTOMS

· NIC firmware or driver mismatch after a host update
· Fabric-level link failure or misconfigured switch port
· Subnet manager not running or misconfigured

ADVANCEDEnter the Lab

INC-009INCIDENT ACTIVE

CONTAINER RUNTIME FAILURE

Kubernetes / Linux

New pods fail to start cluster-wide with runtime errors, though existing pods keep running.

SYMPTOMS

· Container runtime service crashed or hung
· Corrupted runtime state after a disk-full event
· cgroup driver mismatch between kubelet and runtime

INTERMEDIATEEnter the Lab

INC-010INCIDENT ACTIVE

CLUSTER UPGRADE FAILURE

Kubernetes / Control Plane

A minor-version cluster upgrade stalls halfway, leaving a mixed-version control plane.

SYMPTOMS

· API compatibility break between mixed control-plane versions
· etcd health degraded mid-upgrade
· Upgrade automation failed silently on one control-plane node

ADVANCEDEnter the Lab

LAB PLATFORM

CloudOps Foundry Labs

The labs are designed around real infrastructure rather than simulated terminal exercises. Our lab platform is being built around this model:

Launch → Break → Diagnose → Recover → Score

Labs will expand to cover Kubernetes, OpenStack, GPU infrastructure, Ceph, networking, observability, and AI workloads. We do not overpromise functionality that doesn't yet exist — access details are confirmed at signup.