Solved by ptrblck in post #6 
              
              
                From skimming your code, it looks like you are not zeroing out the gradients after the weight update. 
In this case the gradients get accumulated and the weight updates will be quite useless. 
Add this line into your for loop and run it again: 
self.optimizer.zero_grad()

It is also recommended to c…