Answers.Body: <p>One problem with such an approach is if you would know how <code>chest X-ray</code> and <code>cancer biomarkers</code> interact you would likely use that model or knowledge directly instead of using it to generate data. And aligning data among subjects without such knowledge might even give you less information than working with each of the two modalities separately since your model might pick up on some incidental relations. It's like having a regression problem but being unaware of the interaction term.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-26T08:48:37.223Z
Answers.LastActivityDate: 2022-11-26T08:48:37.223Z
Answers.UserDisplayName: Mariusmarten
Body: <p><strong>BACKGROUND:</strong> To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., <code>chest X-ray</code> (<strong>modality 1</strong>) and <code>cancer biomarkers</code> (<strong>modality 2</strong>) come from the <em>same</em> <code>patient</code> (<strong>the example</strong>)).  The problem is that we often don't have that in public datasets.  Instead, datasets are more commonly independent (e.g., a dataset of <code>chest X-rays</code> from one set of patients and a dataset of <code>cancer biomarkers</code> from a second set of patients).</p>
  <p><strong>QUESTION:</strong>  Is there any validity to artificially creating &quot;co-registered&quot; datasets from such independent datasets for the ultimate purpose of leveraging multimodal ML (e.g., generate all possible pairs of <code>chest X-rays</code> and <code>cancer biomarkers</code> from <em>different</em> patients <em>with cancer</em> and generate the same for <em>different</em> patients <em>without cancer</em>)?</p>
  <p><strong>NOTE:</strong>  It is understood that this approach is non-canonical and has flaws, but that is not the question here.  I am more interested in learning whether this could be a <em>second best option</em> for researchers who lack co-registered samples but still want to develop multimodal ML models.  Please provide your response along with some justification as to why this would or would not be valid &quot;second best option&quot;.</p>
  <hr />
  <p>11/26/2022</p>
  <p><strong>New NOTE:</strong>  I thought it was implied in the question post that I am aware that the interaction between modalities is not available to be leveraged by multimodal ML since this is really the crux of the problem with fusing independent datasets.  However, the question still stands:  &quot;Would this be a valid second best option?&quot;</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-11-23T11:09:25.563Z
Comments.Text: A keyword that may be useful to you is the concept of "semi-synthetic" data generation. It seems like that is what you are trying to do :)
Comments.UserDisplayName: Scriddie
Tags: <machine-learning><datasets><semi-synthetic>
Title: Can independent datasets be artificially combined for multimodal learning (semi-synthetic data generation)?
UserDisplayName: Snehal Patel
ThreadText: # Can independent datasets be artificially combined for multimodal learning (semi-synthetic data generation)?
  
  (Asked by: username_0 on 2022-11-22)
  
  **BACKGROUND:** To apply multimodal machine learning (ML), the various data modalities typically come from the same example (e.g., `chest X-ray` (**modality 1**) and `cancer biomarkers` (**modality 2**) come from the *same* `patient` (**the example**)). The problem is that we often don't have that in public datasets. Instead, datasets are more commonly independent (e.g., a dataset of `chest X-rays` from one set of patients and a dataset of `cancer biomarkers` from a second set of patients).
  
  **QUESTION:** Is there any validity to artificially creating "co-registered" datasets from such independent datasets for the ultimate purpose of leveraging multimodal ML (e.g., generate all possible pairs of `chest X-rays` and `cancer biomarkers` from *different* patients *with cancer* and generate the same for *different* patients *without cancer*)?
  
  **NOTE:** It is understood that this approach is non-canonical and has flaws, but that is not the question here. I am more interested in learning whether this could be a *second best option* for researchers who lack co-registered samples but still want to develop multimodal ML models. Please provide your response along with some justification as to why this would or would not be valid "second best option".
  
  ---
  
  11/26/2022
  
  **New NOTE:** I thought it was implied in the question post that I am aware that the interaction between modalities is not available to be leveraged by multimodal ML since this is really the crux of the problem with fusing independent datasets. However, the question still stands: "Would this be a valid second best option?"
  
  **Tags:** machine-learning, datasets, semi-synthetic
  
  > **Comments:**
  > **username_1**: A keyword that may be useful to you is the concept of "semi-synthetic" data generation. It seems like that is what you are trying to do :)  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-11-26. Score: 0
  
  One problem with such an approach is if you would know how `chest X-ray` and `cancer biomarkers` interact you would likely use that model or knowledge directly instead of using it to generate data. And aligning data among subjects without such knowledge might even give you less information than working with each of the two modalities separately since your model might pick up on some incidental relations. It's like having a regression problem but being unaware of the interaction term.
  
  
  ---
  
  