Solved by ptrblck in post #4 
              
              
                I’ve seen some experiments for large scale systems, where the learning rate was adapted to the batch size as seen in Training ImageNet in 1 hour. 
However, this effect should be much smaller in your setup. Might still be worth a try to lower it and see, if it changes the convergence.