INC-007INCIDENT ACTIVE
Storage Throughput Collapse
Read throughput to the training dataset falls from 4 GB/s to under 200 MB/s overnight.
ENVIRONMENTKubernetes / Ceph / Storage
DIFFICULTYAdvanced
STATUSIncident Active
INCIDENT IDINC-007
SYMPTOMS
$ fio --name=read --rw=read --bs=1M --size=4G
READ: bw=187MiB/sMission
Restore training-grade throughput and identify the layer that regressed.
Skills tested
- · Storage benchmarking
- · Ceph/cluster storage diagnostics
- · Network path analysis
POSSIBLE CAUSES
One or more of these is the real cause. The lab does not tell you which.
- Backing storage cluster rebalancing or degraded
- Client-side caching misconfigured after an upgrade
- Network path saturation between compute and storage
- Metadata service bottleneck under load
METHOD
01
Observe
Reproduce the throughput collapse with a controlled benchmark.
02
Hypothesize
Decide between storage-side, network, or client-side causes.
03
Diagnose
Check cluster health, rebalance status, and the network path.
04
Fix
Resolve the degraded layer without masking the symptom with a workaround.
05
Verify
Confirm throughput returns to baseline under the original workload.
Get the workload running. Then explain why it broke.
Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.