I cannot understand why until now, but it is solved by using map_location while I load the model.

I think this is because, all processes are trying to load the model to cuda:0 by default if you don’t set map_location or CUDA_VISIBLE_DEVICES. BTW, does directly setting map_location to device work for you?