Solved by colesbury in post #2 
              
              
                You’re missing an optimizer.zero_grad() call. Without it the gradients get accumulated forever. 
Also, h0 and c0 shouldn’t be declared Variable (no more Variable since PyTorch 0.4). They also don’t need to have requires_grad=True because you’re not getting the derivative w.r.t h0 or c0 – just w.r.t.…