You could set the gradients to None instead of resetting them to zero in order to save more memory, as I guess you are running into the OOM in the second step since the gradients are now allocated and use additional memory. Use optimzer.zero_grad(set_to_none=True) and rerun the code.