"enable_vms.py on -f" failed with the error: IOMMU groups not set up for VMs, aborting.

Issue summary


python3 /var/lib/vastai_kaalia/enable_vms.py on -f failed with IOMMU groups not set up for VMs, aborting.


A similar log can be found in /var/lib/vastai_kaalia/enable_vms.log



Debug steps

  1. Verify kernel parameters:

cat /proc/cmdline
Expected: includes amd_iommu=on iommu=pt nvidia_drm.modeset=0


  1. Verify NVIDIA DRM modeset is disabled:

cat /sys/module/nvidia_drm/parameters/modeset
Expected: N. The Vast script checks this directly.


  1. Run Vast validation:

python3 /var/lib/vastai_kaalia/enable_vms.py validate
If it prints Iommu Groups not set up for VMs, continue with group inspection.


  1. Enumerate IOMMU groups correctly:

for g in /sys/kernel/iommu_groups/*; do echo "Group ${g##*/}" for d in "$g"/devices/*; do [ -e "$d" ] || continue lspci -nns "${d##*/}" done echo done
This identifies whether GPUs share a group with each other or with other endpoint devices.


  1. Review the GPU group specifically:
  • If one group contains multiple GPUs, Vast VM enablement will fail.

Sample:

2c:00.0 NVIDIA GPU
2c:00.1 NVIDIA audio
2d:00.0 NVIDIA GPU
2d:00.1 NVIDIA audio


  • If a GPU shares a group with non-bridge devices, Vast VM enablement can also fail.

Updated on: 08/04/2026