I’ve checked that using watch and that’s not it.
I think it’s related to some sort of cuda version mismatch. Do you know if it’s possible to use pytorch with cuda9.2 while nvidia-smi shows cuda10.0 ?
edit: It runs on a 1080ti and v100 but hangs on p40, version 1.3.1 cuda9.2
edit2: Nevermind, seems like the grid where I am is not setup correctly.