Hi All
I’m LUA\Torch7 user.
Unfortunately Torch7 community seems dead, so Im here with my dumb question.
So main question is followings:

In multi-gpu data parallel, is every GPU calculate Batchnorm independently?
If so, can we “emulate” big batch simply by accumulating gradients for 8 batches and update network only every 8th iteration?
Will it be the same as using 8GPUs in terms of result?