<p>Typically in supervised learning, a neural networks' output is compared to the targets through a loss function, and the gradients are backpropagated. Is it a bad idea to also have a loss function on intermediate layers? I feel that restricts the networks ability to actually optimize the final, global loss function. What are the instances in which an additional loss on intermediate layers would work?</p>
<p><strong>----EDIT----</strong></p>
<p>Clarification of my thought process for supervising intermediate layers, via examples:</p>
<ol>
<li><p>Merging contrastive loss to a Variational AutoEncoder (VAE) or a Classifier. I would have the original VAE loss function (or classifier's cross entropy loss), but additionally supervise the output of the encoder or some intermediate layer with a contrastive loss (for example the triplet loss, where the targets would be positive and negative samples from the batch)</p>
</li>
<li><p>A more classic example would be teacher-student distillation, supervising student's intermediate layers with the teacher's intermediate layers.</p>
</li>
<li><p>When intermediate layers provide a prediction itself. For instance, let's say we have an object detector where the <em>M</em>th layer outputs segmentation masks, and the <em>N</em>th layer (<em>N&gt;M</em>) outputs the bounding boxes.</p>
</li>
</ol>
<p>So the question is, when would these additional losses work, or will they confuse the network just more?</p>

CC BY-SA 4.0
2024-11-21T11:18:33.083Z
Can you explain how you would compute such a loss? What would be the targets, for example? Maybe your idea makes sense, but right now it's a bit unclear. So, backpropagation is a global update rule. Maybe you have some local update rule in mind?
nbro
CC BY-SA 4.0
2024-11-21T17:54:30.227Z
@nbro yes thank you for the question. I edited my post with to exemplify my idea in mind. Let me know if I should elaborate more.
Liubove
<neural-networks><deep-learning><objective-functions>
Loss function on intermediate layers of the networks
Liubove
# Loss function on intermediate layers of the networks

(Asked by: username_0 on 2024-11-20)

Typically in supervised learning, a neural networks' output is compared to the targets through a loss function, and the gradients are backpropagated. Is it a bad idea to also have a loss function on intermediate layers? I feel that restricts the networks ability to actually optimize the final, global loss function. What are the instances in which an additional loss on intermediate layers would work?

**----EDIT----**

Clarification of my thought process for supervising intermediate layers, via examples:

1. Merging contrastive loss to a Variational AutoEncoder (VAE) or a Classifier. I would have the original VAE loss function (or classifier's cross entropy loss), but additionally supervise the output of the encoder or some intermediate layer with a contrastive loss (for example the triplet loss, where the targets would be positive and negative samples from the batch)
2. A more classic example would be teacher-student distillation, supervising student's intermediate layers with the teacher's intermediate layers.
3. When intermediate layers provide a prediction itself. For instance, let's say we have an object detector where the *M*th layer outputs segmentation masks, and the *N*th layer (*N>M*) outputs the bounding boxes.

So the question is, when would these additional losses work, or will they confuse the network just more?

**Tags:** neural-networks, deep-learning, objective-functions

> **Comments:**
> **username_1**: Can you explain how you would compute such a loss? What would be the targets, for example? Maybe your idea makes sense, but right now it's a bit unclear. So, backpropagation is a global update rule. Maybe you have some local update rule in mind?  
> **username_0**: @username_1 yes thank you for the question. I edited my post with to exemplify my idea in mind. Let me know if I should elaborate more.  

---

## 0 answers

