Answers.Body: <p>I agree with the answer by @Funzo. The corresponding section that clarifies this in the paper can be found on page three:</p>
  <blockquote>
  <p><strong>Multimodal pre-training.</strong> [...] Our work does not perform multimodal training but aims to unifiy the learning objective for self-supervised
  learning in different modalities. We hope that this will enable better multimodal representations in the future.
  <a href="https://arxiv.org/abs/2202.03555" rel="nofollow noreferrer">[ArXiv]</a></p>
  </blockquote>
  <p>So what the authors contribute is a framework for learning rich data representations, given any type of data. The framework is the training pipeline that uses the same network in a teacher and student setup. The network first generates data representations from unmasked data (teacher mode) and then does the same with a masked version of the data (student mode). The training objective is minimizing the error of all latent representations between the network in student mode and in teacher mode. If you were to apply the approach you have to choose a model suitable for your data (e.g. the authors use a Vision Transformer for Images).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-21T10:41:50.750Z
Answers.LastActivityDate: 2022-03-21T10:41:50.750Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>it's not multimodal.
  It's trying to be a standard way of training a for a model.
  But it is not suppose to work across modalities</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-02T03:48:46.080Z
Answers.Text: Hi @Funzo, welcome to AI.SE. Do you think you could perhaps expand this answer a bit? As it stands, it makes some claims, but doesn't explain them or provide any sources. Adding sources and explaining your answer some more would greatly improve the quality of the post. Thank you!
Answers.UserDisplayName: Mithical
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-03-20T20:47:54.807Z
Answers.LastActivityDate: 2022-03-20T20:47:54.807Z
Answers.UserDisplayName: Funzo
Body: <p>I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-supervised learning by masking parts of the input and having the network predict the hidden states if the input hadn't been masked. This allows these modes to share a common representation.</p>
  <p>However, I don't understand how the representations of different modes can be connected. For example, how are the hidden state representations of an image of a banana and the word banana trained to be similar, if they are at all?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-21T09:10:10.897Z
Comments.Text: One option would be an *association network* between the various individual latent spaces, having its own multi-modal latent space, where the shared phenomena reside.
Comments.UserDisplayName: Michael
Tags: <deep-learning><hidden-layers><embeddings><self-supervised-learning>
Title: How do multimodal models establish connections between different modes?
UserDisplayName: user3576467
ThreadText: # How do multimodal models establish connections between different modes?
  
  (Asked by: username_0 on 2022-01-22)
  
  I am specifically interested in data2vec, Meta's new model that can convert image, text, and sound data into a unified neural network representation. To my understanding, they did this through self-supervised learning by masking parts of the input and having the network predict the hidden states if the input hadn't been masked. This allows these modes to share a common representation.
  
  However, I don't understand how the representations of different modes can be connected. For example, how are the hidden state representations of an image of a banana and the word banana trained to be similar, if they are at all?
  
  **Tags:** deep-learning, hidden-layers, embeddings, self-supervised-learning
  > **Comments:**
  > **username_1**: One option would be an *association network* between the various individual latent spaces, having its own multi-modal latent space, where the shared phenomena reside.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2022-03-21. Score: 0
  
  I agree with the answer by @Funzo. The corresponding section that clarifies this in the paper can be found on page three:
  
  > **Multimodal pre-training.** [...] Our work does not perform multimodal training but aims to unifiy the learning objective for self-supervised
  > learning in different modalities. We hope that this will enable better multimodal representations in the future.
  > [[ArXiv]](https://arxiv.org/abs/2202.03555)
  
  So what the authors contribute is a framework for learning rich data representations, given any type of data. The framework is the training pipeline that uses the same network in a teacher and student setup. The network first generates data representations from unmasked data (teacher mode) and then does the same with a masked version of the data (student mode). The training objective is minimizing the error of all latent representations between the network in student mode and in teacher mode. If you were to apply the approach you have to choose a model suitable for your data (e.g. the authors use a Vision Transformer for Images).
  
  
  ---
  
  ### Answer by username_3 on 2022-03-20. Score: -1
  
  it's not multimodal.
  It's trying to be a standard way of training a for a model.
  But it is not suppose to work across modalities
  
  > **Comments:**
  > **username_4**: Hi @Funzo, welcome to AI.SE. Do you think you could perhaps expand this answer a bit? As it stands, it makes some claims, but doesn't explain them or provide any sources. Adding sources and explaining your answer some more would greatly improve the quality of the post. Thank you!  
  
  ---
  
  