Deep Learning, Artificial Intelligence, Image Classification, Self Supervised Learning, NLP.
self-attention over the entire input which results in contextualized targets. This distinguishes data2vec from other self-supervised approaches which construct a learning task by predicting or reconstructing local parts of the input. Larger context sizes lead to better downstream performance.