Problem while pretraining MLM from scratch using Transformers
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->
     
- `transformers` version: 3.1.0
- Platform: Linux-3.10.0-1127.19.1.el7.x86_64-x86_64-with-centos-7.8.2003-Core
- Python version: 3.6.9
- PyTorch version (GPU?): 1.6.0 (True)
- Tensorflow version (GPU?): 2.3.0 (True)
- Using GPU in script?: Yes(Two GPUs, Nvidia P100)
- Using distributed or parallel set-up in script?: No

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.
 

 Trainer: **@sgugger** 
 
 -->
 Trainer: @sgugger
## Information

Model I am using (RoBERTa): 

The problem arises when using:
* [* ] my own modified scripts: (give details below)
'''Python
config=RobertaConfig(vocab_size=30_000, max_position_embeddings=inputlen, num_attention_heads=12,
                     num_hidden_layers=6,hidden_dropout_prob=0.1,attention_probs_dropout_prob=0.1,
                     initializer_range=0.2, intermediate_size=3072, type_vocab_size=1)

training_args=TrainingArguments(output_dir="/LAB_SHARED/Projects/011-Language_Model/Data/LanguageModel/BadgerBERT",overwrite_output_dir=True,
                                num_train_epochs=10, do_train=True, do_eval=True, evaluate_during_training=True,
                                per_gpu_train_batch_size=64, learning_rate=0.0004,
                                gradient_accumulation_steps=32,
                                logging_steps=128,
                                warmup_steps=30000,
                                weight_decay=0.01,
                                eval_steps=128,
                                save_steps=128,
                                save_total_limit=2, prediction_loss_only=True)

trainer=Trainer(model=model,args=training_args,data_collator=data_collator,train_dataset=dataset,eval_dataset=dataset1,prediction_loss_only=True)

'''

The tasks I am working on is:
* [ *] my own task or dataset: (give details below)
Training masked language model

## To reproduce

Steps to reproduce the behavior:

1. Please use the following configuration
2. I used my own training dataset, but I guess there should be no difference. My input text file size is around 40GB


<!-- If you have code snippets, error messages, stack traces please provide them here as well.
     Important! Use code tags to correctly format your code. See https://help.github.com/en/github/writing-on-github/creating-and-highlighting-code-blocks#syntax-highlighting
     Do not use screenshots, as they are hard to read and (more importantly) don't allow others to copy-and-paste your code.-->

Traceback (most recent call last):█████████████████████████████████████████████████████████████████▉| 279691/279692 [5:22:01<00:00, 15.48it/s]
  File "CreateLanguageModel.py", line 84, in <module>
    trainer.train()
  File "**/lib64/python3.6/site-packages/transformers/trainer.py", line 785, in train
    torch.save(self.optimizer.state_dict(), os.path.join(output_dir, "optimizer.pt"))
  File "**/lib64/python3.6/site-packages/torch/serialization.py", line 361, in save
    with _open_file_like(f, 'wb') as opened_file:
  File "**/lib64/python3.6/site-packages/torch/serialization.py", line 229, in _open_file_like
    return _open_file(name_or_buffer, mode)
  File "**/lib64/python3.6/site-packages/torch/serialization.py", line 210, in __init__
    super(_open_file, self).__init__(open(name, mode))
FileNotFoundError: [Errno 2] No such file or directory: '**/checkpoint-128/optimizer.pt'

## Expected behavior
Instead of continue training, it stops. It seems that the problem is related to the check-points. I don't know why it tries to open (save) check-point-128 while it already saved 512!
Besides that it already saved check-point-160 while I did not set any parameter as 160, so I don't know where the 160 came from.
I expected the code to just report the evaluation loss periodically, save checkpoints and finish the training.

<!-- A clear and concise description of what you would expect to happen. -->

It looks like you're using Transformers v3.1.0. There have been quite q few improvements/bug fixes on Trainer since then. Could you check you still have the issue with the latest version?<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
