okay, I tried wrapping my head around nn.DistributedDataParallel, following this link -
https://pytorch.org/docs/stable/nn.html#torch.nn.parallel.DistributedDataParallel
but I still not figure out how to do it as tutorial just tells to include following lines for single process multi-gpu.
torch.distributed.init_process_group(backend="nccl")
model = DistributedDataParallel(model)