INC-010INCIDENT ACTIVE
Cluster Upgrade Failure
A minor-version cluster upgrade stalls halfway, leaving a mixed-version control plane.
ENVIRONMENTKubernetes / Control Plane
DIFFICULTYAdvanced
STATUSIncident Active
INCIDENT IDINC-010
SYMPTOMS
$ kubectl get nodes -o wide
control-1 Ready v1.29.4
control-2 Ready v1.28.9
control-3 NotReady v1.28.9Mission
Complete the upgrade safely and bring the control plane back to a single consistent version.
Skills tested
- · Control plane upgrade procedure
- · etcd health diagnostics
- · API compatibility analysis
POSSIBLE CAUSES
One or more of these is the real cause. The lab does not tell you which.
- API compatibility break between mixed control-plane versions
- etcd health degraded mid-upgrade
- Upgrade automation failed silently on one control-plane node
METHOD
01
Observe
Establish the current version skew and control-plane health.
02
Hypothesize
Decide whether to roll forward or roll back safely.
03
Diagnose
Check etcd quorum health and the failed upgrade step.
04
Fix
Complete or safely reverse the upgrade on the affected node.
05
Verify
Confirm all control-plane nodes report a consistent, healthy version.
Get the workload running. Then explain why it broke.
Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.