In facebook paper “Accurate, Large Minibatch SGD: Training ImageNet in 1 Hour”, https://research.fb.com/wp-content/uploads/2017/06/imagenet1kin1h5.pdf 44, we can use large batch size if we scale the learning rate correctly. I want to do the following experiment:

use a 4x1080Ti single machine with multiple gpu cards
define a resnet101 model
for each training iteration, sample a batch = 4096 images.
divide batch among 4gpu card, i.e. each gpu process 4096/4 = 1024 images
I cannot fit 1024 images into a gpu in one time, so 1024 is divided in 16 x 64 sub batches.
the single gpu accumulate the gradients of these 16 sub batches.
after each gpu accumulate 16 sub batches, they send the value to a master gpu card
the master card accumulate gradients from all card and do a parameter update.

How can I do this in pytorch?