With Stochastic Gradient Descent(SGD) you’re just subracting the gradient from each weight/bias on the fly.

  

      en.m.wikipedia.org
  

  
    

Stochastic gradient descent

Stochastic gradient descent (often abbreviated SGD) is an iterative method for optimizing an objective function with suitable smoothness properties (e.g. differentiable or subdifferentiable). It can be regarded as a stochastic approximation of gradient descent optimization, since it replaces the actual gradient (calculated from the entire data set) by an estimate thereof (calculated from a randomly selected subset of the data). Especially in high-dimensional optimization problems this reduces the...

  

  
    
    
  

  


Why not just get the gradients for each batch and sum those together(so you just have one set of gradients) and then manually subtract those(times a learning rate, I’d set this much higher, since you’re running it all at once) from your parameters once your data has all been run through the model? That is what you’re effectively doing, anyway. I mean, you have to keep each layer of gradients separate. Don’t sum across layers, just across batches.
You won’t need momentum, since you are doing it all in one go.
Because you’re adding them at each iteration, the overhead would be equal to your model size. So you could keep it on GPU.