changing the way checkpoint is done in the new release 
Hi
Currently HuggingFace library checks the path for the latest checkpoint and then starts the training from there, this approach is not working due to following reason:
- lets assume you train the model with limit of 1 checkpoint and then you checkpoint only when the best eval accuracy is achieved, then if library loads the model from the last checkpoint, this is not training from the place training is stopped, and this can goes back a lot in time, since the last checkpoint is not necessarily  the last model we need to load from, and this is only the last model with best accuracy. 

To solve the issue, in addition to the checkpoint folders, one needs to introduce a "save_path_folder", and then when resuming the model trainining, loading from this path and not from last checkpoint folder anymore.

Please let me know if any part is not clear. thanks 
I am not sure if I understand this correctly, how would the `Trainer` know if the last model is different from the actual last checkpoint, and if the model is only saved when the best eval accuracy reached, how would you save the last model to `save_path_folder`?

Also if you are saving the last model (and optimizer/scheduler) to some other directory, you could always pass that path as the `model_name_or_path` and change the `--output_dir` so the `Trainer` won't load the last checkpoint <|||||>cc @sgugger <|||||>> lets assume you train the model with limit of 1 checkpoint and then you checkpoint only when the best eval accuracy is achieved, then if library loads the model from the last checkpoint, this is not training from the place training is stopped, and this can goes back a lot in time, since the last checkpoint is not necessarily the last model we need to load from, and this is only the last model with best accuracy.

Yes, you should not use a `save_total_limit` of 1 in conjunction with metric tracking (such as `load_bst_model_at_end=True`) or accept you won't be able to resume training from the last checkpoint.  If you want to restart from scratch, you should just pass `--overwrite_output_dir`.<|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/master/CONTRIBUTING.md) are likely to be ignored.