Also, be careful with the momentum argument, since it is different from the one used in optimizer classes and the conventional notion of momentum. Have a look at this note 34. You probably want to lower it, i.e. setting it to something like 0.001.