Assume we have two nodes: node-A and node-B, each has 4gpus(i.e. ngpu_per_node=4). We set args.batch_size = 256 on each node, means that we want each node process 256 images in each forward.
(1) If we use DistributedDataparallel with 1gpu-per-process mode, shall we manually divide the batchsize by ngpu_per_node in torch.utils.data.DataLoader : torch.utils.data.DataLoader(batch_size = args.batch_size / 4)(the way used in pytorch-imagenet-official-example 75). In my original opinion, I think DistributedSampler can handle such thing, because we have passed world_size and rank to DistributedSampler. . If I was wrong, please point it out, thanks!
(2) If dividing the batchsize by ngpu_per_node is a correct way, I wonder what will happen if we do not that.


Does it means in each node, 4*batch_size images are processed per forward-process?


Will 4*len(dataset)  images are processed in one epoch, or the forward frequency are four times less than usual(i.e. the total number images proceeded per epoch keep same)?