Hi Frank!
Originally, I thought L2 regularization already acts as the additive noise, but that’s to the loss rather than to the gradient itself. I’ll definitely try to play around adding noise to runs with larger batch sizes and see how it goes. Thanks a lot for the suggestions!
I also had another idea of using multiprocessing during the training phase, where I have a Pool of processes and I can just send multiple small mini batches to it to send to the GPU, so that gradients can be calculated in parallel. It is contrasting HogWild, where there are multiple processes training the same model on the same mini-batch. Have you seen a pattern like this?