One possible approach would be to create a copy of the state_dict before calling the initial step() and then restoring the model to the initial state.
However, this should remove all grad attributes, so you could either store them separately or rerun the loss calculation.