Just for reference about the batch size problem, here’s a recent OpenAI post/paper about just that. TL;DR You wanna get the gradient noise scale / batch size near 1.

  
      
      OpenAI Blog – 14 Dec 18
  
  
    

How AI Training Scales 57

We've discovered that the gradient noise scale, a simple statistical metric, predicts the parallelizability of neural network training.