Hi, @mrshenli
I created a mini-repo: GitHub - SHu0421/Question-Repo. you can directly run it by
bash train.sh. my torch version is 1.8.1 and the cuda version is 10.2.
I run the code on four Tesla v100 GPUs (one node). It hung out as before with all GPUs usage 100%

image1324×902 64.9 KB