Proper way to monkey patch a customized model no in transformers?
### Feature request

Hi, nowadays more and more model emerging out, many of them hard to merge transformers, but we have more and more customized patchs such as condensing rotary, xformers attn etc.

What is the best to monkey patch a customized model which not inside transformers library but just loaded with AutoModel?

### Motivation

need guidance

### Your contribution

current not
Hi @lucasjinreal, thanks for raising an issue! 

This is a question best placed in our [forums](https://discuss.huggingface.co/). We try to reserve the github issues for feature requests and bug reports.<|||||>@amyeroberts Please help me find someone answer this issue, I posted serveral on forum none of them got response. than k u<|||||>@lucasjinreal Another place to ask questions like this is in [our discord](https://discord.com/invite/hugging-face-879548962464493619). 