Answers.Body: <p>I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options. </p>
  
  <p>I see a little thing that might be making your model a little worse, though:</p>
  
  <ul>
  <li>You're concatenating "relu" with "sigmoid". </li>
  </ul>
  
  <p>Placing two different nature values in the same array may make it more difficult for updating weights properly.</p>
  
  <p>A few independent suggestions:</p>
  
  <ul>
  <li>Make the output of the images model have "relu" activation as well before the concatenation. Preferrably, use a batch normalization before the image relu and before the number relu (this way you concatenate values that are in very similar ranges).    </li>
  <li>Instead of concatenating, you can try <code>Multiply()([img_output, numeric_output])</code>, in this case, both outputs must have the same size, one of them uses "relu" or "linear", and the other uses "sigmoid".   </li>
  </ul>
  
  <hr>
  
  <p>Now, something important when using AUC: you need big batch sizes, because AUC is dependent on the whole data, it's not like usual metrics/losses that you can take the mean from the results of each batch. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-05T06:27:31.053Z
Answers.Text: Thank you for insights. let me try them. BTW, why do you think, if multiplied, one should have "relu" or "linear", and the other "sigmoid"?
Answers.UserDisplayName: bit_scientist
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-05T13:56:13.840Z
Answers.Text: To keep roughly the same range as the other layers. Sigmoid is between 0 and 1, and ReLU is from 0 to unknown. Multiplying two ReLUs may produce blowing values, too high, multiplying two Sigmoids may decrease the values too much. Use a balanced multiplication.
Answers.UserDisplayName: Daniel Möller
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-03T11:31:39.277Z
Answers.LastActivityDate: 2020-04-03T11:31:39.277Z
Answers.UserDisplayName: Daniel Möller
Body: <p>I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be.</p>
  
  <p>Image-only mean AUC after 5-fold cross-validation is about <strong><em>0.73</em></strong> and numeric data only 5-fold mean AUC is <strong><em>0.65</em></strong>. I was hoping to improve the mean AUC by combining the models into one and merging output layers using <code>concatenate</code> in Keras. </p>
  
  <pre><code>img_output = Dense(256, activation="sigmoid")(x_1) 
  </code></pre>
  
  <p>and </p>
  
  <pre><code>numeric_output = Dense(128, activation="relu")(x_2) 
  </code></pre>
  
  <p>are the output layers of the two models. And, </p>
  
  <pre><code>concat = concatenate([img_output, numeric_output])
  hidden1 = Dense(64, activation="relu")(concat)
  main_output = Dense(1, activation='sigmoid', name='main_output')(hidden1)
  </code></pre>
  
  <p>is the way I concatenated them.</p>
  
  <p>Since image-only performance was better I decided that it might be reasonable to have more dense layers for <code>image_output</code> (<code>256</code>) and ended up using <code>128</code> in <code>numeric_output</code>.I could only reach up to mean AUC of 0.67 using a combined model. I think I should rearrange the concatenation of two outputs somehow (by introducing another learnable parameter (like the formula (10) at 3.3 section of <a href="https://www.hindawi.com/journals/cin/2018/2061516/" rel="nofollow noreferrer">this work</a>?, bias?, or something else) to get more boost on mean AUC. However, I was not able to find what options were available.</p>
  
  <p>Hope you have some ideas worth trying.</p>
  
Tags: <deep-learning><convolutional-neural-networks><keras><performance>
Title: How can I merge outputs of two separate layers so that the overall performance improves?
UserDisplayName: bit_scientist
ThreadText: # How can I merge outputs of two separate layers so that the overall performance improves?
  
  (Asked by: username_0 on 2020-03-09)
  
  I am training a combined model (fine-tuned VGG16 for images and shallow FCN for numerical data) to do a binary classification. However, the overall AUC score is not what I expected it to be.
  
  Image-only mean AUC after 5-fold cross-validation is about ***0.73*** and numeric data only 5-fold mean AUC is ***0.65***. I was hoping to improve the mean AUC by combining the models into one and merging output layers using `concatenate` in Keras.
  
  ```
  img_output = Dense(256, activation="sigmoid")(x_1) 
  
  ```
  
  and
  
  ```
  numeric_output = Dense(128, activation="relu")(x_2) 
  
  ```
  
  are the output layers of the two models. And,
  
  ```
  concat = concatenate([img_output, numeric_output])
  hidden1 = Dense(64, activation="relu")(concat)
  main_output = Dense(1, activation='sigmoid', name='main_output')(hidden1)
  
  ```
  
  is the way I concatenated them.
  
  Since image-only performance was better I decided that it might be reasonable to have more dense layers for `image_output` (`256`) and ended up using `128` in `numeric_output`.I could only reach up to mean AUC of 0.67 using a combined model. I think I should rearrange the concatenation of two outputs somehow (by introducing another learnable parameter (like the formula (10) at 3.3 section of [this work](https://www.hindawi.com/journals/cin/2018/2061516/)?, bias?, or something else) to get more boost on mean AUC. However, I was not able to find what options were available.
  
  Hope you have some ideas worth trying.
  
  **Tags:** deep-learning, convolutional-neural-networks, keras, performance
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-04-03. Score: 2
  
  I'm not sure it's possible to help much because this is an experimental question. I'm afraid the only answer comes with testing many different options.
  
  I see a little thing that might be making your model a little worse, though:
  
  * You're concatenating "relu" with "sigmoid".
  
  Placing two different nature values in the same array may make it more difficult for updating weights properly.
  
  A few independent suggestions:
  
  * Make the output of the images model have "relu" activation as well before the concatenation. Preferrably, use a batch normalization before the image relu and before the number relu (this way you concatenate values that are in very similar ranges).
  * Instead of concatenating, you can try `Multiply()([img_output, numeric_output])`, in this case, both outputs must have the same size, one of them uses "relu" or "linear", and the other uses "sigmoid".
  
  ---
  
  Now, something important when using AUC: you need big batch sizes, because AUC is dependent on the whole data, it's not like usual metrics/losses that you can take the mean from the results of each batch.
  
  > **Comments:**
  > **username_0**: Thank you for insights. let me try them. BTW, why do you think, if multiplied, one should have "relu" or "linear", and the other "sigmoid"?  
  > **username_1**: To keep roughly the same range as the other layers. Sigmoid is between 0 and 1, and ReLU is from 0 to unknown. Multiplying two ReLUs may produce blowing values, too high, multiplying two Sigmoids may decrease the values too much. Use a balanced multiplication.  
  
  ---
  
  