Make classifier backbone dynamic maskformer, mask2former
### Feature request

Make classifier backbone dynamic maskformer, mask2former. Currently it only supports swin transformer. 

link example:  https://github.com/huggingface/transformers/blob/main/src/transformers/models/mask2former/modeling_mask2former.py#L1393


### Motivation

If this feature request successful, it will help to benchmark for new clasifier easily.
Thank you for solving this issue. But there is still problem.
The supported backbones variable hinders to allow any backbones other than
swin.
Please check this line. I had to write child class and had to modify this
line to solve this problem.

https://github.com/huggingface/transformers/blob/main/src/transformers/models/mask2former/configuration_mask2former.py#L124


On Tue, 27 Jun 2023, 11:34 pm Sylvain Gugger, ***@***.***>
wrote:

> Closed #24244 <https://github.com/huggingface/transformers/issues/24244>
> as completed via #24259
> <https://github.com/huggingface/transformers/pull/24259>.
>
> —
> Reply to this email directly, view it on GitHub
> <https://github.com/huggingface/transformers/issues/24244#event-9655252543>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AQSQGCPK433TDSJZKGBL7VDXNMKRDANCNFSM6AAAAAAZEYRWOA>
> .
> You are receiving this because you authored the thread.Message ID:
> ***@***.***>
>
<|||||>cc @amyeroberts <|||||>Hi @tanzzilaalam, you're completely right. I've opened a PR - #24532 -  which should resolve this and allow you to pass in any `backbone_config`. 