@bing it might be a good idea to try a framework. Lightning, Ignite, or Catalyst should take care of most of the boilerplate and get the DDP details right (not always easy).
Not following my own advice, I have iterated my own train script from the original PyTorch examples over various projects and experiments. It’s a bit sprawling, but fast and easy to hack once you are familiar. It supports both torch native and Nvidia Apex DDP. I haven’t done any multi-node work with it, but lots of multi-gpu single node. I know quite a few researchers and engineers have used it for training large nets on 4-8 GPU V100 systems. A version of it tailored for ImageNet w/ RandAugment, Mixup, Random Erasing here: https://github.com/rwightman/pytorch-image-models/blob/master/train.py 28
An example of hacking it for object detection training with MSCOCO: https://github.com/rwightman/efficientdet-pytorch/blob/master/train.py 13
Those scripts have no problem maxing out GPU utilization, so long as the drives for the dataset are fast enough, the data/network is large enough, and the CPU has enough cores for your augmentation/data decoding. I use a shell script for launching the distributed training mode of the train script to keep it simple.