Question: CANINE with (pre-trained) LM head
# 🚀 Feature request

This is more a question than a request.

Would it be possible to have [CANINE](https://huggingface.co/docs/transformers/model_doc/canine) model with a (pre-trained) LM head to perform word/character prediction? (Just like BERTForMaskedLM and other similar models)

## Motivation

The character-based paradigm of the CANINE model is very promising and currently it is possible to perform TokenClassification, SequenceClassification, etc. using it. But I don't see the way of using it to emit LM probabilities over characters of a masked input. In fact, unless I am mistaken, I don't see that any LM pre-trained weights come within the available [pretrained-checkpoints](https://huggingface.co/models?sort=downloads&search=canine). I also see that, in the source code, there are [LM related CANINE heads](https://github.com/huggingface/transformers/blob/v4.15.0/src/transformers/models/canine/modeling_canine.py#L878), but they do not seem to be in use.

So, in summary, CANINE only outputs character embeddings. I have tried to add a fresh LM head on top of it and fine-tune to predict characters, but that lacks all the knowledge that such a LM head would have acquired during a large self-supervised pre-training.

My question is, am I missing or misunderstanding something? Is CANINE suitable to be used for masked words/characters prediction? Will there be, eventually, a pre-trained LM head available for CANINE? (Or maybe this is a question for the original authors of the model?)

## Your contribution

N/A. I am afraid I cannot contribute anything in this regard.

Thank youy so much.

Hi,

CANINE can do masked language modeling (that's how it was pre-trained). However, the authors did not release their pre-training code (yet) as seen [here](https://github.com/google-research/language/tree/master/language/canine#pre-training-code-coming-later).

See also #12892.<|||||>Ok, that was my understanding of the situation so far.

I just wanted some clarification or confirmation from a more knowledgeable person :-)

Since it has more to do with the authors of the model and not with the Transformers library itself, I close this question for now.

Thank you very much for the quick answer!