Hello everyone,
Although I am quite new here (first post!), I feel like I know huggingface transformers pretty well since I read the documentation and tutorials, and I even made a small project by myself recently.
My goal is to create a module that receives a pretrained language model, fine tunes/trains it on some downstream task, NER for example, and then saves only the language model but without the NER classifier at the end (only the underlying finetuned language model) .
Some requirements that make this task non-trivial:

The module should be general enough (inherit from PretrainedModel) so it can be trained, saved to disk, loaded from disk, and then trained, all using huggingface Trainer.
The module must be flexible enough to do this for all huggingface’s models with as little code changes as possible.
The module should be as forward compatible as possible, meaning that I prefer using AutoModel as much as I can.

I hope I got myself clear. if you have any questions regarding my post, please don’t hesitate to reply.