INC-006INCIDENT ACTIVE
Inference Latency Spike
p99 inference latency jumps 8x with no deployment change and no traffic spike.
ENVIRONMENTKubernetes / NVIDIA / Serving
DIFFICULTYAdvanced
STATUSIncident Active
INCIDENT IDINC-006
SYMPTOMS
$ curl -w "%{time_total}\n" -o /dev/null -s http://inference/v1/predict
8.412Mission
Restore p99 latency to baseline and explain the regression.
Skills tested
- · Latency profiling
- · GPU memory diagnostics
- · Multi-tenancy isolation
- · Network path tracing
POSSIBLE CAUSES
One or more of these is the real cause. The lab does not tell you which.
- GPU memory fragmentation from long-running processes
- Noisy-neighbor contention in a multi-tenant cluster
- Batching configuration drifted from expected traffic shape
- Network path change adding hops or saturation
METHOD
01
Observe
Establish the latency baseline and where the regression appears in the path.
02
Hypothesize
Weigh GPU contention against network and batching causes.
03
Diagnose
Correlate GPU memory, neighbor workloads, and request batching.
04
Fix
Isolate or reconfigure the contended resource.
05
Verify
Confirm p99 latency returns to baseline under real traffic.
Get the workload running. Then explain why it broke.
Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.