@sgugger
do you by any chance know what this could be?
The Strange thing is, i resumed training from last valid checkpoint and the model normally generates checkpoints as expected, but at some point the checkpoint folder only contains rng_state.pth as shown on the screenshot.