Hey @ptrblck,

Did you check the values in the good run and saw that they are not close to zero?

Yeah, that’s is exactly what I did, as explained in my initial post. In the attached images of my first post you can see parameters of the ‘good’ checkpoint and the ‘faulty’ checkpoint. As you can see the parameter tensors in the good checkpoint do not start with values close to zero.

Also, could you use the latest nightly binary, as I’ve isolated a checkpoint issue a few weeks ago (which should be fixed by now), which corrupted the loading of state_dict s if CUDATensors were stored.

Will do! Is there a merge request and/or a related issue I can read about this issue?