CloudOps Foundry
INC-003INCIDENT ACTIVE

CUDA Version Mismatch

A container that worked yesterday now fails with a CUDA driver/runtime version error after a host update.

ENVIRONMENTLinux / CUDA / Containers
DIFFICULTYIntermediate
STATUSIncident Active
INCIDENT IDINC-003
SYMPTOMS
$ python train.py
RuntimeError: CUDA error: no kernel image is available
for execution on the device

Mission

Restore a working driver/runtime/container alignment across the node.

Skills tested

  • · CUDA compatibility matrix
  • · Driver diagnostics
  • · Container image inspection
POSSIBLE CAUSES

One or more of these is the real cause. The lab does not tell you which.

  • Host driver upgraded without matching CUDA toolkit in the image
  • Container image pinned to an incompatible CUDA version
  • Multiple CUDA runtimes installed on the host
METHOD
01

Observe

Capture the exact driver and CUDA runtime versions in play.

02

Hypothesize

Suspect a version skew introduced by a recent change.

03

Diagnose

Compare host driver version against the container's CUDA toolkit requirement.

04

Fix

Realign versions — at the host, the image, or both.

05

Verify

Re-run the workload and confirm GPU kernels execute cleanly.

Get the workload running. Then explain why it broke.

Live lab access is provisioned as part of AI Infrastructure Troubleshooting and Production AI Infrastructure.

Request Lab Access