Answers.Body: <p>First of all you don't need "text" input as in Christopher Bonnett's blog. Your case is more easy - demographic is table data, which can be expressed as vector of numeric values. This data should be processed - pushed through one or two fully connected layers. The trick is where, to what part of yolo to concatenate results of processing of this vector. Because it's vector data it should concatenated to fully connected layer. Where exactly should be found by experiments, but as starting point it could be concatenated to before-last (before output) fully connected layer (I think for yolo it's 4096-size layer). </p>
  
  <p>Overall I'd say that is not a trivial task. It require some experience with deep learning, good understanding of yolo design, yolo algorithm and a lot of experimentation, both with architecture and hyperparameters. Probably worth solid paper. Good luck.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T00:29:30.823Z
Answers.Text: So basically you are saying that for testing I should concatenate the processed vector with the vector produced by the second to last fully connected layer for yolo.
Answers.UserDisplayName: Hozaifa Bhutta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T04:45:31.020Z
Answers.Text: Essentially yes,  but as the first stage of experiment. Final version may differ
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-30T21:33:36.617Z
Answers.Text: Do you know how I can access the Yolo structure or where I can learn to do this? Thank you.
Answers.UserDisplayName: Hozaifa Bhutta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T05:07:45.983Z
Answers.Text: Read Yolo paper and github code, for example: https://github.com/ayooshkathuria/pytorch-yolo-v3 ;You have to know python and pytorch or tensorflow
Answers.UserDisplayName: mirror2image
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-29T17:16:04.607Z
Answers.LastActivityDate: 2019-03-29T17:16:04.607Z
Answers.UserDisplayName: mirror2image
Answers.Body: <blockquote>
    <p>you need to take into account the demographics of the patient</p>
  </blockquote>
  
  <p>How, exactly? </p>
  
  <ul>
  <li>Is it a difference of, say, threshold? In this case you can do this serially (as @mirror2image mentions): process the image and then conclude by comparing the size of what you saw to, say, an age-dependent threshold.</li>
  <li>Or has the whole processing to be different? In the extreme, you would not wait until the very end before asking whether the patient is a man if you are looking for prostate cancer.</li>
  </ul>
  
  <p>To design the model, you need enough medical understanding to make such choices. The model can handle the parameters, but you have to choose the architecture.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T23:01:39.700Z
Answers.Text: Hi. This seems to be more of a comment than an answer to the question.
Answers.UserDisplayName: Philip Raeisghasem
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T06:06:06.933Z
Answers.Text: @PhilipRaeisghasem Perhaps, but one too lengthy to fit as a comment.
Answers.UserDisplayName: Mathieu Bouville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T12:35:29.533Z
Answers.Text: Thank you for the response, but I am confused by what you mean when you say ‘difference of threshold’. Also what is an age-dependent threshold.
Answers.UserDisplayName: Hozaifa Bhutta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T13:13:58.023Z
Answers.Text: You say the output would be: `class, center_x, center_y, width and height`. There may be a minimum size (threshold) for what you see to be meaningful, and this value may be different for an old man compared to a young woman: a different threshold.
Answers.UserDisplayName: Mathieu Bouville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T16:51:03.637Z
Answers.Text: So one suggestion (I think) you made is maybe that I merge the two vectors earlier than the place mirror2image suggested. I know this is just an example but I just want to make sure I understand what you are saying.
Answers.UserDisplayName: Hozaifa Bhutta
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-01T17:42:01.443Z
Answers.Text: It depends on what makes sense: `To design the model, you need enough medical understanding to make such choices.`
Answers.UserDisplayName: Mathieu Bouville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T17:05:14.057Z
Answers.LastActivityDate: 2019-03-31T17:05:14.057Z
Answers.UserDisplayName: Mathieu Bouville
Body: <p>This is my first post so please forgive me for any mistakes.</p>
  
  <p>I am working on an object detection algorithm that can detect abnormalities in an x-ray. As a prototype, I will be using yolov3 (more about yolo here: '<a href="https://pjreddie.com/darknet/yolo/" rel="nofollow noreferrer">https://pjreddie.com/darknet/yolo/</a>')
  However, one radiologist mentioned that in order to produce a good result you need to take into account the demographics of the patient.
  In order to do that, my neural network must take into account both text an an image. Some suggestions have been made by other people for this question. For example, someone recommended taking the result of a convolution neural network and a seperate text neural network.
  Here is an image for clarification:
  <a href="https://i.sstatic.net/ctYuj.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ctYuj.png" alt="Here is an image for clarification"></a></p>
  
  <p>Image Credits: This image (<a href="https://cdn-images-1.medium.com/max/1600/1" rel="nofollow noreferrer">https://cdn-images-1.medium.com/max/1600/1</a>*oiLg3C3-7Ocklg9_xubRRw.jpeg) from Christopher Bonnett's article (<a href="https://blog.insightdatascience.com/classifying-e-commerce-products-based-on-images-and-text-14b3f98f899e" rel="nofollow noreferrer">https://blog.insightdatascience.com/classifying-e-commerce-products-based-on-images-and-text-14b3f98f899e</a>)</p>
  
  <p>For more details, please refer to above-mentioned article. It has explained how e-commerce products can be classified into various category hierarchies using both image and text data.</p>
  
  <p>However, a when convolution  neural network is mention it usssualy means it is used for classification instead of detection 
  <a href="https://www.quora.com/What-is-the-difference-between-detection-and-classification-in-computer-vision" rel="nofollow noreferrer">https://www.quora.com/What-is-the-difference-between-detection-and-classification-in-computer-vision</a> (Link for comparison between detection and classification)</p>
  
  <p>In my case, when I am using yolov3, how would it work. Would I be using yolov3 output vector which would be like this format 
  class, center_x, center_y, width and height</p>
  
  <p>My main question is how would the overall structure of my neural network be like if I have both image and text as input while using yolov3.
  Thank you for taking the time to read this.</p>
  
Tags: <neural-networks><convolutional-neural-networks><computer-vision><object-recognition><automation>
Title: Detecting abnormalities in x-rays while taking into account demographics of a patient -automated
UserDisplayName: Hozaifa Bhutta
ThreadText: # Detecting abnormalities in x-rays while taking into account demographics of a patient -automated
  
  (Asked by: username_0 on 2019-03-28)
  
  This is my first post so please forgive me for any mistakes.
  
  I am working on an object detection algorithm that can detect abnormalities in an x-ray. As a prototype, I will be using yolov3 (more about yolo here: '<https://pjreddie.com/darknet/yolo/>')
  However, one radiologist mentioned that in order to produce a good result you need to take into account the demographics of the patient.
  In order to do that, my neural network must take into account both text an an image. Some suggestions have been made by other people for this question. For example, someone recommended taking the result of a convolution neural network and a seperate text neural network.
  Here is an image for clarification:
  [![Here is an image for clarification](https://i.sstatic.net/ctYuj.png)](https://i.sstatic.net/ctYuj.png)
  
  Image Credits: This image (<https://cdn-images-1.medium.com/max/1600/1>\*oiLg3C3-7Ocklg9\_xubRRw.jpeg) from Christopher Bonnett's article (<https://blog.insightdatascience.com/classifying-e-commerce-products-based-on-images-and-text-14b3f98f899e>)
  
  For more details, please refer to above-mentioned article. It has explained how e-commerce products can be classified into various category hierarchies using both image and text data.
  
  However, a when convolution neural network is mention it usssualy means it is used for classification instead of detection
  <https://www.quora.com/What-is-the-difference-between-detection-and-classification-in-computer-vision> (Link for comparison between detection and classification)
  
  In my case, when I am using yolov3, how would it work. Would I be using yolov3 output vector which would be like this format
  class, center\_x, center\_y, width and height
  
  My main question is how would the overall structure of my neural network be like if I have both image and text as input while using yolov3.
  Thank you for taking the time to read this.
  
  **Tags:** neural-networks, convolutional-neural-networks, computer-vision, object-recognition, automation
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-03-29. Score: 0 | Accepted answer
  
  First of all you don't need "text" input as in Christopher Bonnett's blog. Your case is more easy - demographic is table data, which can be expressed as vector of numeric values. This data should be processed - pushed through one or two fully connected layers. The trick is where, to what part of yolo to concatenate results of processing of this vector. Because it's vector data it should concatenated to fully connected layer. Where exactly should be found by experiments, but as starting point it could be concatenated to before-last (before output) fully connected layer (I think for yolo it's 4096-size layer).
  
  Overall I'd say that is not a trivial task. It require some experience with deep learning, good understanding of yolo design, yolo algorithm and a lot of experimentation, both with architecture and hyperparameters. Probably worth solid paper. Good luck.
  
  > **Comments:**
  > **username_0**: So basically you are saying that for testing I should concatenate the processed vector with the vector produced by the second to last fully connected layer for yolo.  
  > **username_1**: Essentially yes,  but as the first stage of experiment. Final version may differ  
  > **username_0**: Do you know how I can access the Yolo structure or where I can learn to do this? Thank you.  
  > **username_1**: Read Yolo paper and github code, for example: https://github.com/ayooshkathuria/pytorch-yolo-v3 ;You have to know python and pytorch or tensorflow  
  
  ---
  
  ### Answer by username_2 on 2019-03-31. Score: 0
  
  > you need to take into account the demographics of the patient
  
  How, exactly?
  
  * Is it a difference of, say, threshold? In this case you can do this serially (as @mirror2image mentions): process the image and then conclude by comparing the size of what you saw to, say, an age-dependent threshold.
  * Or has the whole processing to be different? In the extreme, you would not wait until the very end before asking whether the patient is a man if you are looking for prostate cancer.
  
  To design the model, you need enough medical understanding to make such choices. The model can handle the parameters, but you have to choose the architecture.
  
  > **Comments:**
  > **username_3**: Hi. This seems to be more of a comment than an answer to the question.  
  > **username_2**: @PhilipRaeisghasem Perhaps, but one too lengthy to fit as a comment.  
  > **username_0**: Thank you for the response, but I am confused by what you mean when you say ‘difference of threshold’. Also what is an age-dependent threshold.  
  > **username_2**: You say the output would be: `class, center_x, center_y, width and height`. There may be a minimum size (threshold) for what you see to be meaningful, and this value may be different for an old man compared to a young woman: a different threshold.  
  > **username_0**: So one suggestion (I think) you made is maybe that I merge the two vectors earlier than the place mirror2image suggested. I know this is just an example but I just want to make sure I understand what you are saying.  
  > **username_2**: It depends on what makes sense: `To design the model, you need enough medical understanding to make such choices.`  
  
  ---
  
  