CloudOps Foundry
INC-008INCIDENT ACTIVE

RDMA Unavailable

Multi-node training falls back to TCP after RDMA silently stops being available.

ENVIRONMENTLinux / Networking / GPU
DIFFICULTYAdvanced
STATUSIncident Active
INCIDENT IDINC-008
SYMPTOMS
$ ibstat
CA 'mlx5_0'
        Physical state: LinkDown

Mission

Restore RDMA connectivity between training nodes.

Skills tested

  • · RDMA/InfiniBand diagnostics
  • · NIC driver and firmware checks
  • · Fabric troubleshooting
POSSIBLE CAUSES

One or more of these is the real cause. The lab does not tell you which.

  • NIC firmware or driver mismatch after a host update
  • Fabric-level link failure or misconfigured switch port
  • Subnet manager not running or misconfigured
METHOD
01

Observe

Confirm the fallback to TCP and identify the affected link.

02

Hypothesize

Separate a host-side driver issue from a fabric-side failure.

03

Diagnose

Inspect link state, firmware versions, and subnet manager status.

04

Fix

Restore the link and confirm the correct driver/firmware pairing.

05

Verify

Confirm RDMA traffic flows and multi-node throughput recovers.

Get the workload running. Then explain why it broke.

Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.

Request Lab Access