Removing Head Layer/Model Conversion
I am currently working on some research in which I am to delve into the analysis of decision boundaries in text classification tasks and I am aiming to use recent work from the `ExBERT` paper, allowing me to visualise the importance of particular features across sentences.

Since the library is built on top of models from the Transformers library and requires that *The model architecture must be supported by the `AutoModelWithLMHead`*, I was wondering if it was possible to modify a fine-tuned model to work with that architecture. I am currently using `DistilBERTForSequenceClassification` in my pipeline and was wondering if it were possible to essentially fine-tune for a classification task and use the underlying `DistilBERT` model, as I assume all of the attention weights etc. will still be included in the model? ie. Could I change the loaded model after training to work with the library and to work with the `AutoModelWithLMHead` architecture so that I could inspect the attention heads?

I wasn't sure if I was only able to use models trained for Masked LM or if I could use models trained for downstream tasks? Apologies if this is a question best for the ExBERT github but since it was built into the library, thought I'd ask.
A model trained for sequence classification can definitely be loaded with a different head. Here's an example:

```py
from transformers import DistilBertForSequenceClassification, DistilBertForMaskedLM
sequence_classifier = DistilBertForSequenceClassification.from_pretrained("...")
# Do stuff with your model, train it, do what you like

# Save the weights in a local directory
sequence_classifier.save_pretrained("model-trained-on-xxx")

# Load the weights in the *ForMaskedLM model.
language_model = DistilBertForMaskedLM.from_pretrained("model-trained-on-xxx")
```
This `language_model` has kept all the weights of the base transformer model, has discarded the sequence classification layers, and has randomly initialized the new layers. This model can be loaded in an `AutoModelWithLMHead`.<|||||>Oh wow, did not expect it to be this easy. Thanks very much!