I think your problem is due to a bug in the current checkpoints saving: when you have load_best_model_at_end set at True, it accidentally deletes the oldest checkpoint if the best checkpoint is before it, instead of deleting the oldest checkpoint (this is because you have save_total_limit > 0.
Will try to fix this today.