assertEqual of non-frozen parameters in test_resume_training_with_frozen_params
### System Info

```shell
- `transformers` version: 4.20.0
- Platform: Linux-5.13.0-1031-aws-x86_64-with-glibc2.29
- Python version: 3.8.10
- Huggingface_hub version: 0.7.0
- PyTorch version (GPU?): 1.11.0a0+gita4c10ee (False)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: no
- Using distributed or parallel set-up in script?: no
- Specific hardware: Habana HPU
```


### Who can help?

@sgugger 

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Steps to reproduce the behaviour:

1. Setup an AWS DL1 instance
2. Clone [optimum-habana](https://github.com/huggingface/optimum-habana)
3. Install the package with `pip install optimum-habana/[tests]`
4. Uncomment `test_resume_training_with_frozen_params` in `optimum-habana/tests/test_trainer.py`
5. Run `pytest tests/test_trainer.py -k "frozen"`

### Expected behavior

`test_resume_training_with_frozen_params` in `tests/trainer/test_trainer.py` should not assert if `b` and `b1` are equal [here](https://github.com/huggingface/transformers/blob/eb16be415a74328e5ab62e050330a43054f6bd11/tests/trainer/test_trainer.py#L1429). While `a` is frozen, `b` is not so why should `b` and `b1` be equal?

I understand I use a very specific hardware and the test passes on a GPU, but I actually think it should not pass on the latter and would like to understand why it does :)

Please use the forums for questions like this as we keep issues for bugs and feature requests only.

b and b1 should be equal because they are the results of the same trainings. One started from scratch and one resumed from an intermediate checkpoint. If that test fails, you have a problem in reproducibility on HPUs.<|||||>Sure, sorry for the inconvenience!

Yep I thought `checkpoint-5` comes from a completely different run, closing this as this issue does not come from the test definition.