this is the current approach i am following. But i think this method does this:

for each sub iteration from 1 to 16:

scatter data(256 images to 4 gpu)
each gpu compute (64 images)
gather gradients. master gpu accumulate gradients from other 4 gpus.


master gpu update.

I am trying to cut down the scatter/gather time. How can I do:

scatter data (4096 images to 4 gpus)
each gpu compute and accumulate for 16 times (16x64 images)
gather gradients. master gpu accumulate gradients from other gpu.
master gpu update.

Thanks!