You were right. I was storing the gradients. Thank you! It works now 
Note that I used the negative learning rate approach.
Now, the next step is to compute new gradients at each point along the gradient vector line. However, loss.backwards is not an option as that would update the model. Is there any way of computing the current gradients using a function that returns them, rather than changing them in-place?