The problem may be because of the learning rate or the optimizer you are using. The optimizer is alright but I think the learning rate is too small. Once try between 5e-3 or 1e-2 for 10 to 20 epochs and check if the loss is decreasing or not.