CUDA11.1 does recognize Turing and Ampere GPUs and it seems that they are indeed used, but your code seems to hang?
nn.DataParallel should also be compatible with different architectures, but note that the slower GPUs would most likely be the bottleneck and the faster ones would have to wait.



 jaejun:

and pytorch recognize RTX3090 for cuda:0 even though RTX3090 is not PCI bus number 0


That’s expected and you can change it via export CUDA_DEVICE_ORDER=PCI_BUS_ID.