Splitting the transformers dependencies
### Feature request

Right now this Python packages has a lot of dependencies including major DL frameworks like PyTorch, TensorFlow and Jax. This causes some complexity in the downstream packages that use `transformers` e.g. CI/CD environments get gigantic and it likely creates dependency issues e.g.

![image](https://github.com/huggingface/transformers/assets/1140359/b9651a1a-bf67-4674-ba0f-866cea465c91)

and

![image](https://github.com/huggingface/transformers/assets/1140359/a6a14ba7-bee4-40f7-9b62-08552ec551d6)


This is a CI exception of a Torch code using enformer-pytorch, which depends on transformers. Although there is nothing using Jax either in the Torch code or in the enformer-pytorch, we have to solve this Jax-related issue now.

I was wondering if you can somehow split the dependencies into groups e.g. `'pip install transformers[jax]'` or `'pip install transformers[pytorch]'`. Let me know what you think.

### Motivation

I think splitting the dependencies into reasonable groups would improve installation or testing of the downstream packages using transformers.

### Your contribution

I am not able to contribute, but I think my suggestions is relatively simple to implement.
This is already the case.<|||||>Oh I didn't know! So `'pip install transformers[torch]'` doesn't install jax or tensorflow?<|||||>No. You may have it in your environment from other installs, but `pip install transformers[torch]` will only install Transformers and its core dependencies (very light) and torch.<|||||>Thanks so much!