<p dir="auto">Hello dear HuggingFace team!<br>
According to the original paper, data2vec is not an actual model but more of a self-distilling training strategy. It takes an encoder model as backbone (RoBERTa for text, BEiT for vision, wav2vec for audio as mentioned in the paper) and pre-trains the encoder (student) to predict representations extracted from the EMA instance of the encoder (teacher), meaning the encoder can be any Transformer-based encoder model.<br>
After pretraining, in order to finetune or get predictions, the encoder itself is what matters and data2vec is of no use! (as seen <a href="https://github.com/pytorch/fairseq/tree/main/examples/data2vec#finetuning-data2vec-text-on-glue">here</a>)<br>
I reviewed data2vec implementation in HF transformers and noticed that you decided to use static encoders (BERT for text, BEiT for vision and wav2vec2 for audio) so for example, using Data2VecVisionModel for any task would be the same as using BEiTModel.<br>
Also I noticed that the encoders used for HF Data2Vec are not exactly the same models I mentioned above and there are some minor differences. The reason I'm wondering this, is because I was trying to copy the weights from your models to apply them to my own models in <a href="https://github.com/AryanShekarlaban/data2vec-pytorch">my own repo</a> and found out that I can't due to those incompatibilities.<br>
So my question is, what was the purpose behind all this? and did you train all those models or copied the weights from the original checkpoints in fairseq?</p>
<p dir="auto">Regards,<br>
Aryan</p>
<h1 dir="auto"><g-emoji class="g-emoji" alias="rocket" fallback-src="https://github.githubassets.com/images/icons/emoji/unicode/1f680.png">🚀</g-emoji> Not a Feature request, what am I here for then?</h1>
<p dir="auto">Well, mainly I'd just like to get some feedback from fellow software engineers. Some of the frustrations I've experienced might not have been big issues at all or there could have been easy ways to get around them which I've failed to notice. Getting roasted can be a good way for us to identify obvious flaws in our thought processes that aren't so obvious from our own often tunneled point of view.</p>
<h2 dir="auto">Motivation</h2>
<p dir="auto">Data loading in PyTorch requires the user to define the Dataset, collator, as well as a sampling strategy.<br>
I found it rather hard to stick to the framework when I have to deal with</p>
<ul dir="auto">
<li>Extremely large datasets that do not fit in system memory</li>
<li>The previous point + training with multiple processes and nodes</li>
</ul>
<p dir="auto">For Language Modeling and Machine Translation,</p>
<ul dir="auto">
<li>We often have to deal with large text/CSV files that are multiple times the size of system memory.</li>
<li>Also running training on a single GPU is often slow to the point of frustration.</li>
<li>Then we have to duplicate the extremely large dataset across each machine.</li>
<li>To format the data we have into a PyTorch Dataset, we simply just do a hack where we wrap an IO stream into a PyTorch Dataset</li>
<li>Or preprocess the entire dataset beforehand to speed up data loading.</li>
<li>Sometimes we would keep an index table to "lookup" the position of a data entry in a file which can be slow if not using SSD to perform random access.</li>
</ul>
<p dir="auto">Instead of trying to write code that fits into the format expected by PyTorch. I simply threw everything out the window and just made data loading a standalone service instead...</p>
<h2 dir="auto">Your contribution</h2>
<p dir="auto">Well, I've uploaded my work on <a href="https://github.com/mingruimingrui/data-loader-as-a-service-demo">https://github.com/mingruimingrui/data-loader-as-a-service-demo</a>.</p>
<p dir="auto">If you have taken your time to read up to this point, I would like to give you my gratitude as it had made me quite happy (^o^)<br>
I would also like to ask for you to leave some comments for me which can be any of the following.</p>
<ol dir="auto">
<li>Can you relate to the problems I've faced?</li>
<li>Which part of the Data Loading as a Service do you like?</li>
<li>Which part of the Data Loading as a Service do you not like or have problems agreeing with?</li>
<li>Any other comments would also be appreciated.</li>
</ol>