Thanks for your advice Jaya.
I decreased the learning rate to 0.001 and moved the optimizer.zero_grad()  as you instructed. The loss seemed to be going fine at first but became unstable after the 5th epoch.