[Data2Vec] Incompatibility with the original implementation
Hello dear HuggingFace team!
According to the original paper, data2vec is not an actual model but more of a self-distilling training strategy. It takes an encoder model as backbone (RoBERTa for text, BEiT for vision, wav2vec for audio as mentioned in the paper) and pre-trains the encoder (student) to predict representations extracted from the EMA instance of the encoder (teacher), meaning the encoder can be any Transformer-based encoder model.
After pretraining, in order to finetune or get predictions, the encoder itself is what matters and data2vec is of no use! (as seen [here](https://github.com/pytorch/fairseq/tree/main/examples/data2vec#finetuning-data2vec-text-on-glue))
I reviewed data2vec implementation in HF transformers and noticed that you decided to use static encoders (BERT for text, BEiT for vision and wav2vec2 for audio) so for example, using Data2VecVisionModel for any task would be the same as using BEiTModel. 
Also I noticed that the encoders used for HF Data2Vec are not exactly the same models I mentioned above and there are some minor differences. The reason I'm wondering this, is because I was trying to copy the weights from your models to apply them to my own models in [my own repo](https://github.com/AryanShekarlaban/data2vec-pytorch) and found out that I can't due to those incompatibilities. 
So my question is, what was the purpose behind all this? and did you train all those models or copied the weights from the original checkpoints in fairseq?

Regards,
Aryan