I do encounter the same problem.
After several trial-and-error, I changed my optimizer from SGD back to the traditional GD and it worked. However, I am yet to find out why. In both cases, the learning rate is fixed as 0.1.