serious bug with trainer.py when restarting the training from a checkpoint
## Environment info
<!-- You can run the command `transformers-cli env` and copy-and-paste its output below.
     Don't forget to fill out the missing fields in that output! -->

- `transformers` version: 4.5.1
- Platform: Linux
- Python version: 3.8
- PyTorch version (GPU?): 1.8
- Tensorflow version (GPU?): - 
- Using GPU in script?: - 
- Using distributed or parallel set-up in script?: - 

### Who can help
<!-- Your issue will be replied to more quickly if you can figure out the right person to tag with @
 If you know how to use git blame, that is the easiest way, otherwise, here is a rough guide of **who to tag**.
 Please tag fewer than 3 people.

Models:

- albert, bert, xlm: @LysandreJik
- blenderbot, bart, marian, pegasus, encoderdecoder,  t5: @patrickvonplaten, @patil-suraj
- longformer, reformer, transfoxl, xlnet: @patrickvonplaten
- fsmt: @stas00
- funnel: @sgugger
- gpt2: @patrickvonplaten, @LysandreJik
- rag: @patrickvonplaten, @lhoestq
- tensorflow: @Rocketknight1

Library:

- benchmarks: @patrickvonplaten
- deepspeed: @stas00
- ray/raytune: @richardliaw, @amogkam
- text generation: @patrickvonplaten
- tokenizers: @LysandreJik
- trainer: @sgugger
- pipelines: @LysandreJik

Documentation: @sgugger

Model hub:

- for issues with a model report at https://discuss.huggingface.co/ and tag the model's creator.

HF projects:

- datasets: [different repo](https://github.com/huggingface/datasets)
- rust tokenizers: [different repo](https://github.com/huggingface/tokenizers)

Examples:

- maintained examples (not research project or legacy): @sgugger, @patil-suraj
- research_projects/bert-loses-patience: @JetRunner
- research_projects/distillation: @VictorSanh

 -->

trainer: @sgugger, @patil-suraj

## Information

Hi, I see this serious issue with trainer.py class, let please consider run_translation.py script [1] after you define the model, let freeze the encoder, or wrap the model in a class. So one can modify the model after this line https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/examples/seq2seq/run_translation.py#L331 

Then, during the training, one can stop the training, and now would like to continue the training from the place it is stopped, if you print the number of parameters inside trainer.py, right before this line:

https://github.com/huggingface/transformers/blob/d9c62047a8d75e18d2849d345ab3394875a712ef/src/transformers/trainer.py#L1062

like this 
```
for n,p in model.named_parameters():
   if p.requires_grad:
       print(n)
```

what would we see? We see all parameters are there, even  the ones we made frozen, this is a serious bug that if the user modify the model after creation, those modifications are not considered when restarting the training, could you kindly have a look?
thanks 

[1] https://github.com/huggingface/transformers/blob/master/examples/seq2seq/run_translation.py 


## Expected behavior

The user should be able to continue training the modified model as they are modified. 
You can also consider the modification in finetune_trainer.py

https://github.com/huggingface/transformers/blob/1c06240e1b3477728129bb58e7b6c7734bb5074e/examples/legacy/seq2seq/finetune_trainer.py#L233

If you freeze some parameters as done in the line above, those would not be there when you load the model and restarting the training, this is really a serious issue, thanks for the help 
<|||||>Here is the minimal code to generate this bug, we make a model, we freeze, then we save it (as done in trainer checkpoint), then we load it (as done in train() in trainer.py) and we see if the feezed parameters are there or not

```
from transformers import T5ForConditionalGeneration
from typing import Optional 
import torch
import os

# This is copied from trainer.py
def _save(model, output_dir: Optional[str] = None):
        os.makedirs(output_dir, exist_ok=True)
        print(f"Saving model checkpoint to {output_dir}")
        # Save a trained model and configuration using `save_pretrained()`.
        # They can then be reloaded using `from_pretrained()`
        state_dict = model.state_dict()
        model.save_pretrained(output_dir, state_dict=state_dict)

def print_num_parameters(model):
   for n,p in model.named_parameters():
      if (p.requires_grad):
        print("n ", n) 

def freeze_params(model):
   for n,p in model.named_parameters():
      p.requires_grad = False


model = T5ForConditionalGeneration.from_pretrained("t5-base")
freeze_params(model)
print("#### parameters before saving ####")
print_num_parameters(model)
_save(model, "temp_model")

# Now lets load the model as done in trainer from the checkpoint.
model = model.from_pretrained("temp_model")

# Now lets print the number of parameters
print("#### parameters after saving ####")
print_num_parameters(model)
```

surprisingly, no, the freezed parameters are not freezed anymore after loading the checkpoint:
```
#### parameters before saving ####
Saving model checkpoint to temp_model
#### parameters after saving ####
n  shared.weight
n  encoder.block.0.layer.0.SelfAttention.q.weight
n  encoder.block.0.layer.0.SelfAttention.k.weight
n  encoder.block.0.layer.0.SelfAttention.v.weight
n  encoder.block.0.layer.0.SelfAttention.o.weight
n  encoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight
n  encoder.block.0.layer.0.layer_norm.weight
n  encoder.block.0.layer.1.DenseReluDense.wi.weight
n  encoder.block.0.layer.1.DenseReluDense.wo.weight
n  encoder.block.0.layer.1.layer_norm.weight
n  encoder.block.1.layer.0.SelfAttention.q.weight
n  encoder.block.1.layer.0.SelfAttention.k.weight
n  encoder.block.1.layer.0.SelfAttention.v.weight
n  encoder.block.1.layer.0.SelfAttention.o.weight
n  encoder.block.1.layer.0.layer_norm.weight
n  encoder.block.1.layer.1.DenseReluDense.wi.weight
n  encoder.block.1.layer.1.DenseReluDense.wo.weight
n  encoder.block.1.layer.1.layer_norm.weight
n  encoder.block.2.layer.0.SelfAttention.q.weight
n  encoder.block.2.layer.0.SelfAttention.k.weight
n  encoder.block.2.layer.0.SelfAttention.v.weight
n  encoder.block.2.layer.0.SelfAttention.o.weight
n  encoder.block.2.layer.0.layer_norm.weight
n  encoder.block.2.layer.1.DenseReluDense.wi.weight
n  encoder.block.2.layer.1.DenseReluDense.wo.weight
n  encoder.block.2.layer.1.layer_norm.weight
n  encoder.block.3.layer.0.SelfAttention.q.weight
n  encoder.block.3.layer.0.SelfAttention.k.weight
n  encoder.block.3.layer.0.SelfAttention.v.weight
n  encoder.block.3.layer.0.SelfAttention.o.weight
n  encoder.block.3.layer.0.layer_norm.weight
n  encoder.block.3.layer.1.DenseReluDense.wi.weight
n  encoder.block.3.layer.1.DenseReluDense.wo.weight
n  encoder.block.3.layer.1.layer_norm.weight
n  encoder.block.4.layer.0.SelfAttention.q.weight
n  encoder.block.4.layer.0.SelfAttention.k.weight
n  encoder.block.4.layer.0.SelfAttention.v.weight
n  encoder.block.4.layer.0.SelfAttention.o.weight
n  encoder.block.4.layer.0.layer_norm.weight
n  encoder.block.4.layer.1.DenseReluDense.wi.weight
n  encoder.block.4.layer.1.DenseReluDense.wo.weight
n  encoder.block.4.layer.1.layer_norm.weight
n  encoder.block.5.layer.0.SelfAttention.q.weight
n  encoder.block.5.layer.0.SelfAttention.k.weight
n  encoder.block.5.layer.0.SelfAttention.v.weight
n  encoder.block.5.layer.0.SelfAttention.o.weight
n  encoder.block.5.layer.0.layer_norm.weight
n  encoder.block.5.layer.1.DenseReluDense.wi.weight
n  encoder.block.5.layer.1.DenseReluDense.wo.weight
n  encoder.block.5.layer.1.layer_norm.weight
n  encoder.block.6.layer.0.SelfAttention.q.weight
n  encoder.block.6.layer.0.SelfAttention.k.weight
n  encoder.block.6.layer.0.SelfAttention.v.weight
n  encoder.block.6.layer.0.SelfAttention.o.weight
n  encoder.block.6.layer.0.layer_norm.weight
n  encoder.block.6.layer.1.DenseReluDense.wi.weight
n  encoder.block.6.layer.1.DenseReluDense.wo.weight
n  encoder.block.6.layer.1.layer_norm.weight
n  encoder.block.7.layer.0.SelfAttention.q.weight
n  encoder.block.7.layer.0.SelfAttention.k.weight
n  encoder.block.7.layer.0.SelfAttention.v.weight
n  encoder.block.7.layer.0.SelfAttention.o.weight
n  encoder.block.7.layer.0.layer_norm.weight
n  encoder.block.7.layer.1.DenseReluDense.wi.weight
n  encoder.block.7.layer.1.DenseReluDense.wo.weight
n  encoder.block.7.layer.1.layer_norm.weight
n  encoder.block.8.layer.0.SelfAttention.q.weight
n  encoder.block.8.layer.0.SelfAttention.k.weight
n  encoder.block.8.layer.0.SelfAttention.v.weight
n  encoder.block.8.layer.0.SelfAttention.o.weight
n  encoder.block.8.layer.0.layer_norm.weight
n  encoder.block.8.layer.1.DenseReluDense.wi.weight
n  encoder.block.8.layer.1.DenseReluDense.wo.weight
n  encoder.block.8.layer.1.layer_norm.weight
n  encoder.block.9.layer.0.SelfAttention.q.weight
n  encoder.block.9.layer.0.SelfAttention.k.weight
n  encoder.block.9.layer.0.SelfAttention.v.weight
n  encoder.block.9.layer.0.SelfAttention.o.weight
n  encoder.block.9.layer.0.layer_norm.weight
n  encoder.block.9.layer.1.DenseReluDense.wi.weight
n  encoder.block.9.layer.1.DenseReluDense.wo.weight
n  encoder.block.9.layer.1.layer_norm.weight
n  encoder.block.10.layer.0.SelfAttention.q.weight
n  encoder.block.10.layer.0.SelfAttention.k.weight
n  encoder.block.10.layer.0.SelfAttention.v.weight
n  encoder.block.10.layer.0.SelfAttention.o.weight
n  encoder.block.10.layer.0.layer_norm.weight
n  encoder.block.10.layer.1.DenseReluDense.wi.weight
n  encoder.block.10.layer.1.DenseReluDense.wo.weight
n  encoder.block.10.layer.1.layer_norm.weight
n  encoder.block.11.layer.0.SelfAttention.q.weight
n  encoder.block.11.layer.0.SelfAttention.k.weight
n  encoder.block.11.layer.0.SelfAttention.v.weight
n  encoder.block.11.layer.0.SelfAttention.o.weight
n  encoder.block.11.layer.0.layer_norm.weight
n  encoder.block.11.layer.1.DenseReluDense.wi.weight
n  encoder.block.11.layer.1.DenseReluDense.wo.weight
n  encoder.block.11.layer.1.layer_norm.weight
n  encoder.final_layer_norm.weight
n  decoder.block.0.layer.0.SelfAttention.q.weight
n  decoder.block.0.layer.0.SelfAttention.k.weight
n  decoder.block.0.layer.0.SelfAttention.v.weight
n  decoder.block.0.layer.0.SelfAttention.o.weight
n  decoder.block.0.layer.0.SelfAttention.relative_attention_bias.weight
n  decoder.block.0.layer.0.layer_norm.weight
n  decoder.block.0.layer.1.EncDecAttention.q.weight
n  decoder.block.0.layer.1.EncDecAttention.k.weight
n  decoder.block.0.layer.1.EncDecAttention.v.weight
n  decoder.block.0.layer.1.EncDecAttention.o.weight
n  decoder.block.0.layer.1.layer_norm.weight
n  decoder.block.0.layer.2.DenseReluDense.wi.weight
n  decoder.block.0.layer.2.DenseReluDense.wo.weight
n  decoder.block.0.layer.2.layer_norm.weight
n  decoder.block.1.layer.0.SelfAttention.q.weight
n  decoder.block.1.layer.0.SelfAttention.k.weight
n  decoder.block.1.layer.0.SelfAttention.v.weight
n  decoder.block.1.layer.0.SelfAttention.o.weight
n  decoder.block.1.layer.0.layer_norm.weight
n  decoder.block.1.layer.1.EncDecAttention.q.weight
n  decoder.block.1.layer.1.EncDecAttention.k.weight
n  decoder.block.1.layer.1.EncDecAttention.v.weight
n  decoder.block.1.layer.1.EncDecAttention.o.weight
n  decoder.block.1.layer.1.layer_norm.weight
n  decoder.block.1.layer.2.DenseReluDense.wi.weight
n  decoder.block.1.layer.2.DenseReluDense.wo.weight
n  decoder.block.1.layer.2.layer_norm.weight
n  decoder.block.2.layer.0.SelfAttention.q.weight
n  decoder.block.2.layer.0.SelfAttention.k.weight
n  decoder.block.2.layer.0.SelfAttention.v.weight
n  decoder.block.2.layer.0.SelfAttention.o.weight
n  decoder.block.2.layer.0.layer_norm.weight
n  decoder.block.2.layer.1.EncDecAttention.q.weight
n  decoder.block.2.layer.1.EncDecAttention.k.weight
n  decoder.block.2.layer.1.EncDecAttention.v.weight
n  decoder.block.2.layer.1.EncDecAttention.o.weight
n  decoder.block.2.layer.1.layer_norm.weight
n  decoder.block.2.layer.2.DenseReluDense.wi.weight
n  decoder.block.2.layer.2.DenseReluDense.wo.weight
n  decoder.block.2.layer.2.layer_norm.weight
n  decoder.block.3.layer.0.SelfAttention.q.weight
n  decoder.block.3.layer.0.SelfAttention.k.weight
n  decoder.block.3.layer.0.SelfAttention.v.weight
n  decoder.block.3.layer.0.SelfAttention.o.weight
n  decoder.block.3.layer.0.layer_norm.weight
n  decoder.block.3.layer.1.EncDecAttention.q.weight
n  decoder.block.3.layer.1.EncDecAttention.k.weight
n  decoder.block.3.layer.1.EncDecAttention.v.weight
n  decoder.block.3.layer.1.EncDecAttention.o.weight
n  decoder.block.3.layer.1.layer_norm.weight
n  decoder.block.3.layer.2.DenseReluDense.wi.weight
n  decoder.block.3.layer.2.DenseReluDense.wo.weight
n  decoder.block.3.layer.2.layer_norm.weight
n  decoder.block.4.layer.0.SelfAttention.q.weight
n  decoder.block.4.layer.0.SelfAttention.k.weight
n  decoder.block.4.layer.0.SelfAttention.v.weight
n  decoder.block.4.layer.0.SelfAttention.o.weight
n  decoder.block.4.layer.0.layer_norm.weight
n  decoder.block.4.layer.1.EncDecAttention.q.weight
n  decoder.block.4.layer.1.EncDecAttention.k.weight
n  decoder.block.4.layer.1.EncDecAttention.v.weight
n  decoder.block.4.layer.1.EncDecAttention.o.weight
n  decoder.block.4.layer.1.layer_norm.weight
n  decoder.block.4.layer.2.DenseReluDense.wi.weight
n  decoder.block.4.layer.2.DenseReluDense.wo.weight
n  decoder.block.4.layer.2.layer_norm.weight
n  decoder.block.5.layer.0.SelfAttention.q.weight
n  decoder.block.5.layer.0.SelfAttention.k.weight
n  decoder.block.5.layer.0.SelfAttention.v.weight
n  decoder.block.5.layer.0.SelfAttention.o.weight
n  decoder.block.5.layer.0.layer_norm.weight
n  decoder.block.5.layer.1.EncDecAttention.q.weight
n  decoder.block.5.layer.1.EncDecAttention.k.weight
n  decoder.block.5.layer.1.EncDecAttention.v.weight
n  decoder.block.5.layer.1.EncDecAttention.o.weight
n  decoder.block.5.layer.1.layer_norm.weight
n  decoder.block.5.layer.2.DenseReluDense.wi.weight
n  decoder.block.5.layer.2.DenseReluDense.wo.weight
n  decoder.block.5.layer.2.layer_norm.weight
n  decoder.block.6.layer.0.SelfAttention.q.weight
n  decoder.block.6.layer.0.SelfAttention.k.weight
n  decoder.block.6.layer.0.SelfAttention.v.weight
n  decoder.block.6.layer.0.SelfAttention.o.weight
n  decoder.block.6.layer.0.layer_norm.weight
n  decoder.block.6.layer.1.EncDecAttention.q.weight
n  decoder.block.6.layer.1.EncDecAttention.k.weight
n  decoder.block.6.layer.1.EncDecAttention.v.weight
n  decoder.block.6.layer.1.EncDecAttention.o.weight
n  decoder.block.6.layer.1.layer_norm.weight
n  decoder.block.6.layer.2.DenseReluDense.wi.weight
n  decoder.block.6.layer.2.DenseReluDense.wo.weight
n  decoder.block.6.layer.2.layer_norm.weight
n  decoder.block.7.layer.0.SelfAttention.q.weight
n  decoder.block.7.layer.0.SelfAttention.k.weight
n  decoder.block.7.layer.0.SelfAttention.v.weight
n  decoder.block.7.layer.0.SelfAttention.o.weight
n  decoder.block.7.layer.0.layer_norm.weight
n  decoder.block.7.layer.1.EncDecAttention.q.weight
n  decoder.block.7.layer.1.EncDecAttention.k.weight
n  decoder.block.7.layer.1.EncDecAttention.v.weight
n  decoder.block.7.layer.1.EncDecAttention.o.weight
n  decoder.block.7.layer.1.layer_norm.weight
n  decoder.block.7.layer.2.DenseReluDense.wi.weight
n  decoder.block.7.layer.2.DenseReluDense.wo.weight
n  decoder.block.7.layer.2.layer_norm.weight
n  decoder.block.8.layer.0.SelfAttention.q.weight
n  decoder.block.8.layer.0.SelfAttention.k.weight
n  decoder.block.8.layer.0.SelfAttention.v.weight
n  decoder.block.8.layer.0.SelfAttention.o.weight
n  decoder.block.8.layer.0.layer_norm.weight
n  decoder.block.8.layer.1.EncDecAttention.q.weight
n  decoder.block.8.layer.1.EncDecAttention.k.weight
n  decoder.block.8.layer.1.EncDecAttention.v.weight
n  decoder.block.8.layer.1.EncDecAttention.o.weight
n  decoder.block.8.layer.1.layer_norm.weight
n  decoder.block.8.layer.2.DenseReluDense.wi.weight
n  decoder.block.8.layer.2.DenseReluDense.wo.weight
n  decoder.block.8.layer.2.layer_norm.weight
n  decoder.block.9.layer.0.SelfAttention.q.weight
n  decoder.block.9.layer.0.SelfAttention.k.weight
n  decoder.block.9.layer.0.SelfAttention.v.weight
n  decoder.block.9.layer.0.SelfAttention.o.weight
n  decoder.block.9.layer.0.layer_norm.weight
n  decoder.block.9.layer.1.EncDecAttention.q.weight
n  decoder.block.9.layer.1.EncDecAttention.k.weight
n  decoder.block.9.layer.1.EncDecAttention.v.weight
n  decoder.block.9.layer.1.EncDecAttention.o.weight
n  decoder.block.9.layer.1.layer_norm.weight
n  decoder.block.9.layer.2.DenseReluDense.wi.weight
n  decoder.block.9.layer.2.DenseReluDense.wo.weight
n  decoder.block.9.layer.2.layer_norm.weight
n  decoder.block.10.layer.0.SelfAttention.q.weight
n  decoder.block.10.layer.0.SelfAttention.k.weight
n  decoder.block.10.layer.0.SelfAttention.v.weight
n  decoder.block.10.layer.0.SelfAttention.o.weight
n  decoder.block.10.layer.0.layer_norm.weight
n  decoder.block.10.layer.1.EncDecAttention.q.weight
n  decoder.block.10.layer.1.EncDecAttention.k.weight
n  decoder.block.10.layer.1.EncDecAttention.v.weight
n  decoder.block.10.layer.1.EncDecAttention.o.weight
n  decoder.block.10.layer.1.layer_norm.weight
n  decoder.block.10.layer.2.DenseReluDense.wi.weight
n  decoder.block.10.layer.2.DenseReluDense.wo.weight
n  decoder.block.10.layer.2.layer_norm.weight
n  decoder.block.11.layer.0.SelfAttention.q.weight
n  decoder.block.11.layer.0.SelfAttention.k.weight
n  decoder.block.11.layer.0.SelfAttention.v.weight
n  decoder.block.11.layer.0.SelfAttention.o.weight
n  decoder.block.11.layer.0.layer_norm.weight
n  decoder.block.11.layer.1.EncDecAttention.q.weight
n  decoder.block.11.layer.1.EncDecAttention.k.weight
n  decoder.block.11.layer.1.EncDecAttention.v.weight
n  decoder.block.11.layer.1.EncDecAttention.o.weight
n  decoder.block.11.layer.1.layer_norm.weight
n  decoder.block.11.layer.2.DenseReluDense.wi.weight
n  decoder.block.11.layer.2.DenseReluDense.wo.weight
n  decoder.block.11.layer.2.layer_norm.weight
n  decoder.final_layer_norm.weight
```
<|||||>Your end example is not surprising: you are re-loading a new model so of course modifications done on the original model are erased.
We'll look at how we can fix this inside the trainer, to load the weights differently instead of using `from_pretrained`.<|||||>Hi @sgugger  thank you for the response, my intention from the example was showing that the procedure happening inside the trainer if the user resume training from a modified model. thank you. <|||||>@sgugger  I also see some more issues with the trainer.py when I load the model from the checkpoint, the model which previously was training fine on the GPU gets out of memory issue, there must be a leakge of memory during the loading from a checkpoint, shall I make a separate ticket for this issue? thanks <|||||>If it's because you add frozen parameters previously, that would explain the OOM error.<|||||>Hi @LysandreJik @sgugger 
thanks for the respose, but I do not see how this is relevant. 
During training, I also load the model and then freeze some of the parameters, then this trains fine, only during loading from a checkpoint, this goes to memory issues, but the procedure remains the same, by loading the model and then freezing params, i personally think there must be a bug somewhere in checkpoint loading, resulting in extra usage of memory,. thanks for your help <|||||>One simple test to check this @sgugger  would be get a model, and choose a batch size in a way that it just fit the memory but larger than that wont, then resume the training from a checkpoint, then I am sure you would also see the memory issue, even without any modification, just the baseline t5 I do see this issue with huggingface codes. thanks for your help <|||||>> We'll look at how we can fix this inside the trainer, to load the weights differently instead of using `from_pretrained`.

@sgugger, this is definitely related to what I need for deepspeed checkpoint resume. Currently we first load the model `from_pretrained` and then it gets dropped and replaced by the deepspeed checkpoint, which for huge models is a huge slowdown.
So let's coordinate this work.

My preliminary idea was to pass a new flag to `from_pretrained` which will do everything except actually loading the weights. I was planning to work on this this week.

(plus we need to stop random init the weights when they are replaced with pre-trained weights, so this is related too but not directly to this particular issue)<|||||>Thanks @stas00  for your attention to this issue, this would be really awesome to have this fixed, thanks a lot for the great work you do <|||||>You're in good hands, @dorooddorood606 - @sgugger is taking care of it already, I was just commenting that something similar needs to be done for deepspeed, so once @sgugger's PR goes in I will work on doing the same for deepspeed.<|||||>Dear @stas00 
Thank you very much both of you @sgugger to your great efforts and the great job you do, I also observe the vanilla t5-base checkpointing gets very different results after resume, I reported the bug here, this can be a relevant bug to this one:

https://github.com/huggingface/transformers/issues/11323 

So this seems there are some randomness in t5-base model which is not considered in trainer.py resume from checkpointing part, if you also think these bugs can be related, I would greatly appreciate if you could also consider the losses when one resume from a checkpoint.
 I would like to thank you so much for your time and your efforts and the great and awesome job you do. 