Answers.Body: <p>You can try doing image segmentation the traditional way, just using the image data. If you want to use the non-image data, then, you can introduce classification as another task for your network. It will provide some regularization to your model. But, this is one way you can still use non-image data whilst still working with image outputs.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-25T19:45:49.667Z
Answers.LastActivityDate: 2021-02-25T19:45:49.667Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I have a task for which I have to do image segmentation (cancer detection on MRIs). If possible, I would also like to include clinical data (i.e. numeric/categorical data which comes in the form of a table with features such as age, gender, ...).</p>
  <p>I know that for classification purposes, it's possible to create a model that uses both numeric data as well as image data (as mentioned in the paper by Huang et al. : <a href="https://www.nature.com/articles/s41598-020-78888-w.pdf" rel="nofollow noreferrer">&quot;Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case‑study in pulmonary embolism detection&quot;</a></p>
  <p>The problem I have is that, for image segmentation tasks, it doesn't really make sense to me as to how to use both types of data.</p>
  <p>In the above-mentioned paper, they create one model with only the image data and another with only the numeric data, and then they fuse them (there are multiple strategies for fusing them together). For classification tasks, it makes sense. However, for my task, it does not make sense to have a model which only uses the clinical data for image segmentation and that's where I get confused.</p>
  <p>How do you think I should proceed with my task? Is it even possible to mix both types of data for image segmentation tasks?</p>
  
Tags: <deep-learning><image-segmentation><u-net>
Title: How to use mixed data for image segmentation?
UserDisplayName: Skyris
ThreadText: # How to use mixed data for image segmentation?
  
  (Asked by: username_0 on 2021-02-25)
  
  I have a task for which I have to do image segmentation (cancer detection on MRIs). If possible, I would also like to include clinical data (i.e. numeric/categorical data which comes in the form of a table with features such as age, gender, ...).
  
  I know that for classification purposes, it's possible to create a model that uses both numeric data as well as image data (as mentioned in the paper by Huang et al. : ["Multimodal fusion with deep neural networks for leveraging CT imaging and electronic health record: a case‑study in pulmonary embolism detection"](https://www.nature.com/articles/s41598-020-78888-w.pdf)
  
  The problem I have is that, for image segmentation tasks, it doesn't really make sense to me as to how to use both types of data.
  
  In the above-mentioned paper, they create one model with only the image data and another with only the numeric data, and then they fuse them (there are multiple strategies for fusing them together). For classification tasks, it makes sense. However, for my task, it does not make sense to have a model which only uses the clinical data for image segmentation and that's where I get confused.
  
  How do you think I should proceed with my task? Is it even possible to mix both types of data for image segmentation tasks?
  
  **Tags:** deep-learning, image-segmentation, u-net
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-02-25. Score: 1 | Accepted answer
  
  You can try doing image segmentation the traditional way, just using the image data. If you want to use the non-image data, then, you can introduce classification as another task for your network. It will provide some regularization to your model. But, this is one way you can still use non-image data whilst still working with image outputs.
  
  
  ---
  
  