Answers.Body: <p>Well, I suppose one can use some kind of <code>contrastive learning</code> in this case.</p>
  <p>A famous example of the establishment of relation between two different representations is the <a href="https://openai.com/blog/clip/" rel="nofollow noreferrer">CLIP</a>  - <em>Contrastive Language–Image Pre-training</em>, where model gets a huge corpus of image captions and images and the image caption is passed through the language model, and the image itself through convolutional or ViT backbone, and the model learns to make the image embedding (output of the visual model) to be similar to the output embedding (output of text model) and dissimilar (in the sense of cosine distance) to the embedding of captions, that belong to other images in the training dataset.</p>
  <p><a href="https://i.sstatic.net/g5D5i.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/g5D5i.png" alt="enter image description here" /></a></p>
  <p>In case your two data representations allow for the notion of similar and dissimilar, I expect you can apply a similar procedure.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-18T20:33:56.090Z
Answers.LastActivityDate: 2021-09-18T20:33:56.090Z
Answers.UserDisplayName: spiridon_the_sun_rotator
Body: <p>I am a researcher in a field, and new to the whole of AI and machine learning techniques.  May the following question is trivial or not framed in the ML language but I try my best.</p>
  <p>I have two sets of representations (I can extract feature vectors, etc., from the datasets) from vastly different domains. I want to find, if any, a relationship exists between these two sets.  In other words, I want an algorithm (the idea of an algorithm) to learn both representations and find the connections and convert one representation to another.</p>
  <p>There is neither apparent one-to-one correspondence nor both need to be the same lengths.</p>
  <p>Any suggestion on how to approach this problem is appreciated.</p>
  <p>I thought of one method; write an encoder-decoder for each of these presentations separately and swap the decoders.  I am not sure whether it works or not, and besides I may not have any idea what's going on there.</p>
  <p>I prefer a general approach if it exists.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-18T18:59:17.200Z
Comments.Text: Have you tried a simple design where you input one representation to an NN and make it predict the other?
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-19T00:43:11.070Z
Comments.Text: You say "There is neither apparent one-to-one correspondence nor both need to be the same lengths", but then I don't understand why you want to convert one representation to the other. _What's your ultimate goal or why do you want to do this_? Could you also provide an example of 2 vastly different feature vectors that you would want the algorithm to convert from/to, so that we have a better idea of what you really want to achieve?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-09-30T22:01:05.853Z
Comments.Text: Yes, I am sorry that the message was kind of cryptic.  I am comparing chemical vibrations to musical notes.   I have a dataset of chemicals, let's say proteins, polymers, crystals etc. and another dataset of musical notes, ragas etc.  
  
  I need to find the correspondence or any hidden structural similarities between these two domains.  
  
  I can assign one amino acid to one note manually and create some kind of music but that's not what I want. 
  In the end, the algorithm needs to convert chemicals to music or music to chemicals vice versa.
Comments.UserDisplayName: bababee
Tags: <machine-learning><features><algorithm-request>
Title: How to find "relationships" between two data representations?
UserDisplayName: bababee
ThreadText: # How to find "relationships" between two data representations?
  
  (Asked by: username_0 on 2021-09-18)
  
  I am a researcher in a field, and new to the whole of AI and machine learning techniques. May the following question is trivial or not framed in the ML language but I try my best.
  
  I have two sets of representations (I can extract feature vectors, etc., from the datasets) from vastly different domains. I want to find, if any, a relationship exists between these two sets. In other words, I want an algorithm (the idea of an algorithm) to learn both representations and find the connections and convert one representation to another.
  
  There is neither apparent one-to-one correspondence nor both need to be the same lengths.
  
  Any suggestion on how to approach this problem is appreciated.
  
  I thought of one method; write an encoder-decoder for each of these presentations separately and swap the decoders. I am not sure whether it works or not, and besides I may not have any idea what's going on there.
  
  I prefer a general approach if it exists.
  
  **Tags:** machine-learning, features, algorithm-request
  
  > **Comments:**
  > **username_1**: Have you tried a simple design where you input one representation to an NN and make it predict the other?  
  > **username_2**: You say "There is neither apparent one-to-one correspondence nor both need to be the same lengths", but then I don't understand why you want to convert one representation to the other. _What's your ultimate goal or why do you want to do this_? Could you also provide an example of 2 vastly different feature vectors that you would want the algorithm to convert from/to, so that we have a better idea of what you really want to achieve?  
  > **username_0**: Yes, I am sorry that the message was kind of cryptic.  I am comparing chemical vibrations to musical notes.   I have a dataset of chemicals, let's say proteins, polymers, crystals etc. and another dataset of musical notes, ragas etc.    I need to find the correspondence or any hidden structural similarities between these two domains.    I can assign one amino acid to one note manually and create some kind of music but that's not what I want.  In the end, the algorithm needs to convert chemicals to music or music to chemicals vice versa.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2021-09-18. Score: 1 | Accepted answer
  
  Well, I suppose one can use some kind of `contrastive learning` in this case.
  
  A famous example of the establishment of relation between two different representations is the [CLIP](https://openai.com/blog/clip/) - *Contrastive Language–Image Pre-training*, where model gets a huge corpus of image captions and images and the image caption is passed through the language model, and the image itself through convolutional or ViT backbone, and the model learns to make the image embedding (output of the visual model) to be similar to the output embedding (output of text model) and dissimilar (in the sense of cosine distance) to the embedding of captions, that belong to other images in the training dataset.
  
  [![enter image description here](https://i.sstatic.net/g5D5i.png)](https://i.sstatic.net/g5D5i.png)
  
  In case your two data representations allow for the notion of similar and dissimilar, I expect you can apply a similar procedure.
  
  
  ---
  
  