Body: <p>I'm a newbie in neural networks. I'm trying to fit my neural network that has 3 different outputs: </p>
  
  <ol>
  <li>semantic segmentation, </li>
  <li>box mask and </li>
  <li>box coordinates.</li>
  </ol>
  
  <p>When my model is training, the loss of semantic segmentation and box coordinates are decreasing, but the loss of the box mask is increasing too much. </p>
  
  <p>My neural network is a CNN and it's based on Chargrid from <a href="https://arxiv.org/abs/1809.08799" rel="nofollow noreferrer">here</a>. The architecture is this:</p>
  
  <p><a href="https://i.sstatic.net/f9L0E.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/f9L0E.png" alt="enter image description here"></a></p>
  
  <p>For semantic segmentation outputs, it's expected to have 15 classes, for box mask it's expected to have 2 classes (foreground vs background) and for box coordinates it's expected to have 5 classes (1 for each corner of bounding box + 1 for None). </p>
  
  <h2>Loss details</h2>
  
  <h3>Step 1</h3>
  
  <p>Here's the loss/accuracy for each of the three outputs at the end of the first step.</p>
  
  <ol>
  <li>Semantic segmentation (Output 1) - 0.0181/0.958</li>
  <li>Box mask (Output 2) - 13.88/0.946</li>
  <li>Box coordinates (Output 3) - 0.2174/0.0000000867</li>
  </ol>
  
  <h3>Last step</h3>
  
  <p>Here's the loss/accuracy at the last step.             </p>
  
  <ol>
  <li>Semantic segmentation (Output 1) - 0.0157/0.963</li>
  <li>Box mask (Output 2) - 73.02/0.935</li>
  <li>Box coordinates (Output 3) - 0.06/0.82</li>
  </ol>
  
  <hr>
  
  <p><strong>Is that normal? How can I interpret these results?</strong></p>
  
  <p>I will leave below, the output of the model fit.</p>
  
  <p><a href="https://i.sstatic.net/c7fQs.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/c7fQs.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/wtGBc.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wtGBc.png" alt="enter image description here"></a>
  <a href="https://i.sstatic.net/fNWl8.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/fNWl8.png" alt="enter image description here"></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T16:10:22.023Z
Comments.Text: Hi, you ought to be a bit more specific and provide some more details, otherwise nobody will be able to answer your question. Also, the images are pretty much unreadable, so a summary as text would be better.
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T16:23:14.393Z
Comments.Text: @OliverMason what do you mean about being more specific?
Comments.UserDisplayName: João Castilho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T16:25:58.990Z
Comments.Text: Give us more detail about your data, architecture, etc
Comments.UserDisplayName: Oliver Mason
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-02-20T16:36:54.857Z
Comments.Text: @OliverMason is this better now?
Comments.UserDisplayName: João Castilho
Tags: <machine-learning><convolutional-neural-networks><training><overfitting><loss>
Title: Why is the loss of one of the outputs of a model with multiple outputs increasing while the others are decreasing?
UserDisplayName: João Castilho
ThreadText: # Why is the loss of one of the outputs of a model with multiple outputs increasing while the others are decreasing?
  
  (Asked by: username_0 on 2020-02-20)
  
  I'm a newbie in neural networks. I'm trying to fit my neural network that has 3 different outputs:
  
  1. semantic segmentation,
  2. box mask and
  3. box coordinates.
  
  When my model is training, the loss of semantic segmentation and box coordinates are decreasing, but the loss of the box mask is increasing too much.
  
  My neural network is a CNN and it's based on Chargrid from [here](https://arxiv.org/abs/1809.08799). The architecture is this:
  
  [![enter image description here](https://i.sstatic.net/f9L0E.png)](https://i.sstatic.net/f9L0E.png)
  
  For semantic segmentation outputs, it's expected to have 15 classes, for box mask it's expected to have 2 classes (foreground vs background) and for box coordinates it's expected to have 5 classes (1 for each corner of bounding box + 1 for None).
  
  ## Loss details
  
  ### Step 1
  
  Here's the loss/accuracy for each of the three outputs at the end of the first step.
  
  1. Semantic segmentation (Output 1) - 0.0181/0.958
  2. Box mask (Output 2) - 13.88/0.946
  3. Box coordinates (Output 3) - 0.2174/0.0000000867
  
  ### Last step
  
  Here's the loss/accuracy at the last step.
  
  1. Semantic segmentation (Output 1) - 0.0157/0.963
  2. Box mask (Output 2) - 73.02/0.935
  3. Box coordinates (Output 3) - 0.06/0.82
  
  ---
  
  **Is that normal? How can I interpret these results?**
  
  I will leave below, the output of the model fit.
  
  [![enter image description here](https://i.sstatic.net/c7fQs.png)](https://i.sstatic.net/c7fQs.png)
  [![enter image description here](https://i.sstatic.net/wtGBc.png)](https://i.sstatic.net/wtGBc.png)
  [![enter image description here](https://i.sstatic.net/fNWl8.png)](https://i.sstatic.net/fNWl8.png)
  
  **Tags:** machine-learning, convolutional-neural-networks, training, overfitting, loss
  
  > **Comments:**
  > **username_1**: Hi, you ought to be a bit more specific and provide some more details, otherwise nobody will be able to answer your question. Also, the images are pretty much unreadable, so a summary as text would be better.  
  > **username_0**: @username_1 what do you mean about being more specific?  
  > **username_1**: Give us more detail about your data, architecture, etc  
  > **username_0**: @username_1 is this better now?  
  
  ---
  
  ## 0 answers
  
  