Module import takes too long
### Feature request

Make
```python
import transformers
```

faster 



### Motivation

When I import the transformers package, it always takes > 5 seconds on my MacBook Pro machine. 

I'm not sure if it's just me but how can I make this faster? 

This can get a bit annoying when my debug script is pretty fast to run. 

MWE to illustrate the problem

```python
from datetime import datetime
start = datetime.now()
import transformers
print(datetime.now() - start)
```


### Your contribution

NA
Hey @StefanHeng, it should not take that long. What version of `transformers` do you have installed?<|||||>Interesting. 

I have python version `3.9.7` and `transformers` version 4.18.0 in a conda environment. <|||||>You may have tools that you do not directly need that slow down your instantiation: are you using torch, tensorflow or flax? If you're only using one, can you try to uninstall the other two if they're installed and checking if it speeds up the install?<|||||>I'm using `torch`. Now that you mentioned it, I do also have `tensorflow` installed in the environment too. 

Because I use `torch` for the architectures but I need the `tensorboard` under `tensorflow` for visualizations. <|||||>When I uninstall `tensorflow`, I'm seeing ~1 seconds for the `transformers` import. Nice! 

but is it possible to speed up loading 2 backends? cos I do need the `tensorboard` functionalities under `tensorflow`... <|||||>Unfortunately, it isn't `transformers` taking all that time, it's likely `tensorflow`. Loading `tensorflow` on its own will likely take most of the time that you saw when you had both frameworks installed, as we load a tiny subset of the items when doing `import transformers`. <|||||>This issue has been automatically marked as stale because it has not had recent activity. If you think this still needs to be addressed please comment on this thread.

Please note that issues that do not follow the [contributing guidelines](https://github.com/huggingface/transformers/blob/main/CONTRIBUTING.md) are likely to be ignored.