Body: <p>Typically in supervised learning, a neural networks' output is compared to the targets through a loss function, and the gradients are backpropagated. Is it a bad idea to also have a loss function on intermediate layers? I feel that restricts the networks ability to actually optimize the final, global loss function. What are the instances in which an additional loss on intermediate layers would work?</p>
  <p><strong>----EDIT----</strong></p>
  <p>Clarification of my thought process for supervising intermediate layers, via examples:</p>
  <ol>
  <li><p>Merging contrastive loss to a Variational AutoEncoder (VAE) or a Classifier. I would have the original VAE loss function (or classifier's cross entropy loss), but additionally supervise the output of the encoder or some intermediate layer with a contrastive loss (for example the triplet loss, where the targets would be positive and negative samples from the batch)</p>
  </li>
  <li><p>A more classic example would be teacher-student distillation, supervising student's intermediate layers with the teacher's intermediate layers.</p>
  </li>
  <li><p>When intermediate layers provide a prediction itself. For instance, let's say we have an object detector where the <em>M</em>th layer outputs segmentation masks, and the <em>N</em>th layer (<em>N&gt;M</em>) outputs the bounding boxes.</p>
  </li>
  </ol>
  <p>So the question is, when would these additional losses work, or will they confuse the network just more?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-21T11:18:33.083Z
Comments.Text: Can you explain how you would compute such a loss? What would be the targets, for example? Maybe your idea makes sense, but right now it's a bit unclear. So, backpropagation is a global update rule. Maybe you have some local update rule in mind?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-11-21T17:54:30.227Z
Comments.Text: @nbro yes thank you for the question. I edited my post with to exemplify my idea in mind. Let me know if I should elaborate more.
Comments.UserDisplayName: Liubove
Tags: <neural-networks><deep-learning><objective-functions>
Title: Loss function on intermediate layers of the networks
UserDisplayName: Liubove
ThreadText: # Loss function on intermediate layers of the networks
  
  (Asked by: username_0 on 2024-11-20)
  
  Typically in supervised learning, a neural networks' output is compared to the targets through a loss function, and the gradients are backpropagated. Is it a bad idea to also have a loss function on intermediate layers? I feel that restricts the networks ability to actually optimize the final, global loss function. What are the instances in which an additional loss on intermediate layers would work?
  
  **----EDIT----**
  
  Clarification of my thought process for supervising intermediate layers, via examples:
  
  1. Merging contrastive loss to a Variational AutoEncoder (VAE) or a Classifier. I would have the original VAE loss function (or classifier's cross entropy loss), but additionally supervise the output of the encoder or some intermediate layer with a contrastive loss (for example the triplet loss, where the targets would be positive and negative samples from the batch)
  2. A more classic example would be teacher-student distillation, supervising student's intermediate layers with the teacher's intermediate layers.
  3. When intermediate layers provide a prediction itself. For instance, let's say we have an object detector where the *M*th layer outputs segmentation masks, and the *N*th layer (*N>M*) outputs the bounding boxes.
  
  So the question is, when would these additional losses work, or will they confuse the network just more?
  
  **Tags:** neural-networks, deep-learning, objective-functions
  
  > **Comments:**
  > **username_1**: Can you explain how you would compute such a loss? What would be the targets, for example? Maybe your idea makes sense, but right now it's a bit unclear. So, backpropagation is a global update rule. Maybe you have some local update rule in mind?  
  > **username_0**: @username_1 yes thank you for the question. I edited my post with to exemplify my idea in mind. Let me know if I should elaborate more.  
  
  ---
  
  ## 0 answers
  
  