The error point to a broken checkpoint, not the distributed launch.
It seems you’ve saved some checkpoints in the previous runs without making sure only a single process (e.g. rank0) writes to the files.
This might yield to multiple processes writing to the same checkpoint file and thus breaking it.
Could this be the case?