Ah, you’d probably want to look at the internals of nn.DataParallel–the forward method has a very straightforward (pun intended) scatter-replicate-apply-gather loop–I think it’s possible to scatter a large batch, apply N times on N slices of the scattered batch, and then gather the gradients? Not sure, might need to ask the Adam or another pro.