I just wanted to report about the spikes and the slow learning. I have tried with a learning rate of 0.0001 (compared to the 0.001 I had before) and it didn’t help at all. It took REALLY LONG to learn (compared to SGD 0.0001), it got rid of the spikes, but accuracy at test time was really slow.
I then changed it to Adam optimizer: accuracy at test time is still high, and it takes 1/10th of the time to train!