HT_Liu:

nn.Dataparallel


First of all, check GPU utilization. can be by watch -n 1 nvidia-smi
The utilization should be ~90% on both GPU. If not:

The feeding of data is too slow (need more workers or SSD drive)
Your network is not too complicated to use all resources

I advice you to check training using ex. Resnet101 with random data and see if there is any speed up per epoch.