Answers.Body: <p>What you want to do is called multi-task learning. Here's what you do:</p>
  <ol>
  <li>Create a second Input.</li>
  <li>Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.</li>
  <li>Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.</li>
  <li>Put a dense layer of 2 after this.</li>
  <li>Put softmax with units = number of classes.</li>
  <li>Add CE loss calculated with this one and the ground labels to the loss of U-Net.</li>
  </ol>
  <p>This is in regards to TensorFlow. The same can be done in PyTorch easily.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-03-18T17:29:08.310Z
Answers.LastActivityDate: 2021-03-18T17:29:08.310Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the form of a table containing features such as gender, age, etc.). So far, my researches have led me nowhere. What can I do in order to achieve this?</p>
  
Tags: <deep-learning><data-preprocessing><u-net><semantic-segmentation><multi-task-learning>
Title: How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?
UserDisplayName: Skyris
ThreadText: # How should I incorporate numerical and categorical data as part of the inputs to the U-net for semantic segmentation?
  
  (Asked by: username_0 on 2021-03-18)
  
  I am using a U-Net to segment cancer cells in images of patients' arms. I would like to add patient data to it in order to see if it is possible to enhance the segmentation (patient data comes in the form of a table containing features such as gender, age, etc.). So far, my researches have led me nowhere. What can I do in order to achieve this?
  
  **Tags:** deep-learning, data-preprocessing, u-net, semantic-segmentation, multi-task-learning
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-03-18. Score: 2 | Accepted answer
  
  What you want to do is called multi-task learning. Here's what you do:
  
  1. Create a second Input.
  2. Attach it to 1D CNN (2-3 layers), so it aggregates this tabular information.
  3. Concatenate this feature with the intermediate feature generated by the U-Net using Concatenate layer.
  4. Put a dense layer of 2 after this.
  5. Put softmax with units = number of classes.
  6. Add CE loss calculated with this one and the ground labels to the loss of U-Net.
  
  This is in regards to TensorFlow. The same can be done in PyTorch easily.
  
  
  ---
  
  