A bit late here, but I had the exact same issue and the problem was that I was loading a state_dict (saved from the device cuda:0) from four different GPUs, and the resulting effect was that all the GPUs were loading the state_dict in the device cuda:0.
I solved loading the state_dict with:
torch.load(<state dict file path>, map_location=current_device)