thanks for your reply. According to what nvidia-smi shows the other gpu’s have zero utilization. I tried to use the DistributedDataParallel  the utilization problem is solved but seems there are problems with 1D batch normalization in distributed training ( I mentioned mine in 106969 6 but I see others also have the same problem)