Issue with ESM and DDP due to unused positional embeddings when rotary embeddings specified
### System Info

Copy-and-paste the text below in your GitHub issue and FILL OUT the two last points.

- `transformers` version: 4.25.1
- Platform: Linux-5.4.0-136-generic-x86_64-with-glibc2.31
- Python version: 3.10.4
- Huggingface_hub version: 0.11.1
- PyTorch version (GPU?): 1.12.0 (True)
- Tensorflow version (GPU?): not installed (NA)
- Flax version (CPU?/GPU?/TPU?): not installed (NA)
- Jax version: not installed
- JaxLib version: not installed
- Using GPU in script?: Yes, 1 GPU
- Using distributed or parallel set-up in script?: Yes, DDP


### Who can help?

@ArthurZucker @Rocketknight1

### Information

- [ ] The official example scripts
- [X] My own modified scripts

### Tasks

- [ ] An officially supported task in the `examples` folder (such as GLUE/SQuAD, ...)
- [X] My own task or dataset (give details below)

### Reproduction

Using class `ESMModel` results in an unused parameter creation. In the DDP setting, this causes errors as some parameters don't receive grad. Within `EsmEmbeddings` , self.positional_embeddings should not be instantiated as an `nn.Embedding` if `config. position_embedding_type` is not `absolute`.

### Expected behavior

We would expect instantiaton of `EsmEmbeddings` to not create the unused `nn.Embedding` module as this can create cryptic errors later due to unused parameters.
Note that you can avoid any error by passing `find_unused_parameters=True` when wrapping your model in DDP.<|||||>This is part of the model design that I believe is in the original repo as well, but if it's still causing issues with @sgugger's fix, we can look into making a PR to remove the base positional embeddings when the model is using rotary embeddings instead!

If it's working for you with that fix, though, feel free to close the issue.<|||||>Hi @Rocketknight1, it would fix the issue but due to other features of the particular model (ESM within a larger module) in use, the flag results in `RuntimeError: Expected to mark a variable ready only once`. At any rate, it turned out this portion of ESM was the only problematic portion, but feel free to close the issue since that find unused params fix will (probably) be sufficient for most users.<|||||>Alright, I'll do that for now - but if anyone comes across this issue and `find_unused_parameters=True` is not helping, feel free to reopen and comment here!