I gave up with VFIO. The original error was me not passing through other 3 components of the GPU (audio, usb, scsi). Now, pytorch works sometimes only if the GPU was originally attached by nouveau, then bind to vfio. Eg, if GPU was originally only used by vfio-pci, pytorch will not work in guest. Instead, python3 binary will be frozen and unkillable, requiring a reset of the guest.
Seems like vfio is not ready for deep learning. I wonder how do colabs run the services? Will be running bare metal now, thanks.