Did you check the values in the good run and saw that they are not close to zero?
I would start with a comparison between the “good” model after training vs. the model after reloading.
Also, could you use the latest nightly binary, as I’ve isolated a checkpoint issue a few weeks ago (which should be fixed by now), which corrupted the loading of state_dicts if CUDATensors were stored.