Add adapter support
# 🚀 Feature request

Add [adapter](https://arxiv.org/abs/1902.00751) support to transformers.

## Motivation

Adapters are great time-and-memory-savers for multitask use cases and would be a great addition to this library. Some very kind folks added support for them ([AdapterHub](https://adapterhub.ml/)) on top of transformers library but unfortunately in order to use it one needs to use their [fork](https://github.com/Adapter-Hub/adapter-transformers) which is slightly inconvenient.

## Your contribution

They've done the integration already so I hope it's straightforward. I've posted [an issue](https://github.com/Adapter-Hub/adapter-transformers/issues/65) on their end as well and would be happy to help in any way I can.

That'd be great!<|||||>This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
<|||||>Just voicing my support of this too. Not sure yet if adapter-hub will end up having the resources to merge back into `transformers`. If they become inactive, me (and a lot of the community I'm sure) will want to try to take up that effort.