optimizer.zero_grad()
loss.backward()
optimizer.step()

I believe this may fix your issue. You need to clear all the gradients before you perform a backward pass again. Also as Alex mentioned, using L1Loss() may exacerbate the issue in some cases.