Solved by ptrblck in post #2 
              
              
                The loss curve of GD is usually less noisy and the loss should decrease in each step. 
Using SGD on the other hand gives you more noise, but also the final accuracy is often better than in GD. One might claim that the noisy updates add some regularization, but I’m not sure what the current theory i…