Hm, do you have an HDMI cable plugged into one of the cards to drive the GUI? If yes, can you try to run the code without DataParallel on just the card where you have the HDMI cable is plugged in? I am wondering if this really is a DataParallel issue or whether it’s due to exhausting all the GPU memory such that the GUI freezes.
PS: I have a machine with 8 2080Ti and when I use DataParallel on those, i don’t have any issues. I am running a headless install of Ubuntu there though, so no GUI. On my second machine I have 4 cards where I have an HDMI cable plugged into one of those for the GUI, and I notice that the interface is indeed slower when I run model training on this card – but I guess this is expected? (It never crashes/completely freezes though)