Thanks for the answers, I was able to solve it by re-installing pytorch in the same node where the script will be run (with a V100 card). Even if I install it with cuda but not within a node with V100 card, the script hangs (but works fine for other / older cards), but after re-installing, it works fine across the GPUs.