Hi I’m trying to do a learning a certain way and I need a big batch sufficiently big for it.
Indeed I need to compute a mean vector that converges weekly in O(1/sqrt(n)) (cf. CLT).
I have a powerful machine to use: 8 RTXA6000
When I run my model with a certain batch n it will share n/8 sub-batch on each GPU and then compute 8 different vectors. I’d like to compute only one vector to optimize the weak 1/sqrt(n)-convergence I deal with is it possible to use a trick to do that?