Answers.Body: <blockquote>
  <p>Normaly, the output layer is only connected to the second last layer.</p>
  </blockquote>
  <p>This is only normal for standard MLPs. As soon as you look into slightly more sophisticated neural networks, you'll find a variety of different schemes to connect the layers within a neural network.</p>
  <blockquote>
  <p>Is there any model that the output layer is connected to multiple layers</p>
  </blockquote>
  <p>There are many examples of such a case, like <a href="https://arxiv.org/abs/1609.03499" rel="nofollow noreferrer">WaveNet</a>, <a href="https://arxiv.org/abs/1505.04597" rel="nofollow noreferrer">U-Net</a>, <a href="https://arxiv.org/abs/1908.07919" rel="nofollow noreferrer">HRNet</a>, to name a just few. The first two networks use <em>skip-connections</em>. Mostly summation or concatenation is applied to combine the multilayer results.
  And like @nbro mentioned in his comment, <a href="https://arxiv.org/abs/1512.03385" rel="nofollow noreferrer">ResNet</a> is also comparable which uses residual connections to bypass layers.</p>
  <p>There are a lot more examples than that because this practice has a very beneficial aspect to it: It shortcuts the backward pass and thus mitigates the <a href="https://en.wikipedia.org/wiki/Vanishing_gradient_problem" rel="nofollow noreferrer">vanishing gradient problem</a>.</p>
  <p>A more general answer to your question is that you can wire up your layers however you want as long as the network stays differentiable. It should somehow make sense though: In the case of Wavenet these skip-connections help to capture temporal features from different granularities (short-term temporal features detected in the early layers / long-term temporal features detected in later layers).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-06-26T12:05:02.570Z
Answers.LastActivityDate: 2022-06-26T12:05:02.570Z
Answers.UserDisplayName: Chillston
Body: <p>Normaly, the output layer is only connected to the second last layer.
  Is there any model that the output layer is connected to multiple layers (For example, the second last layer AND the layer before it.)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-25T17:35:21.323Z
Comments.Text: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.
Comments.UserDisplayName: Community
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-26T09:20:50.490Z
Comments.Text: Residual networks have connections that skip layers, the obviously called _skip connections_. I don't know if skip connections also happen for that specific case you're mentioning, but you should look into ResNets because they may be one answer to your question.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><model-request>
Title: Can the output layer be connected to multiple layers?
UserDisplayName: kaka
ThreadText: # Can the output layer be connected to multiple layers?
  
  (Asked by: username_0 on 2022-06-25)
  
  Normaly, the output layer is only connected to the second last layer.
  Is there any model that the output layer is connected to multiple layers (For example, the second last layer AND the layer before it.)
  
  **Tags:** neural-networks, machine-learning, model-request
  
  > **Comments:**
  > **username_1**: Please edit the question to limit it to a specific problem with enough detail to identify an adequate answer.  
  > **username_2**: Residual networks have connections that skip layers, the obviously called _skip connections_. I don't know if skip connections also happen for that specific case you're mentioning, but you should look into ResNets because they may be one answer to your question.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2022-06-26. Score: 1
  
  > Normaly, the output layer is only connected to the second last layer.
  
  This is only normal for standard MLPs. As soon as you look into slightly more sophisticated neural networks, you'll find a variety of different schemes to connect the layers within a neural network.
  
  > Is there any model that the output layer is connected to multiple layers
  
  There are many examples of such a case, like [WaveNet](https://arxiv.org/abs/1609.03499), [U-Net](https://arxiv.org/abs/1505.04597), [HRNet](https://arxiv.org/abs/1908.07919), to name a just few. The first two networks use *skip-connections*. Mostly summation or concatenation is applied to combine the multilayer results.
  And like @username_2 mentioned in his comment, [ResNet](https://arxiv.org/abs/1512.03385) is also comparable which uses residual connections to bypass layers.
  
  There are a lot more examples than that because this practice has a very beneficial aspect to it: It shortcuts the backward pass and thus mitigates the [vanishing gradient problem](https://en.wikipedia.org/wiki/Vanishing_gradient_problem).
  
  A more general answer to your question is that you can wire up your layers however you want as long as the network stays differentiable. It should somehow make sense though: In the case of Wavenet these skip-connections help to capture temporal features from different granularities (short-term temporal features detected in the early layers / long-term temporal features detected in later layers).
  
  
  ---
  
  