I was suffering the same issue. But it seems  not to be caused by GPU0 problem. Because if you use CUDA_VISIBLE_DEVICES=1,3,4 to specify the used GPUS, they are reindexed in the pytorch process as range(#available_GPUS) (same as in tensorflow).
After many tries, I find the problem in the definition of a model and solve as fellows. I originally defined my model as :
self.features, self.layer4 = load_resnet()
            self.roi_fmap = self._roi_map

def forward(self, x):
   ...
   return self.roi_fmap(x)
....
# for convenient, I wrap the operation here
def _roi_map(self, features):
        return self.layer4(features).mean(3).mean(2) 

Then I solve the problem by modifying
self.features, self.layer4 = load_resnet()
            self.roi_fmap = self.layer4

def forward(self, x):
   ...
  out = self.roi_fmap(x)
   return out.mean(3).mean(2)

I guest that when I use a function  _roi_map() to wrap a module self.layer4(features).mean(3).mean(2), pytorch seems  not to assige this part to different gpus by using replicas = nn.parallel.replicate(model, devices=list(range(num_gpus))).
Any layer/module/model should be exposed in the def init() part of a  nn.Module class. pytorch seems only search relevant layer/module/model here and  not to dive into any functions within or outside the class definition.
I am not sure whether my analysis correct. If you know more please correct me. Thanks a lot.