Answers.Body: <p>Yes, you can do that, and it is a standard practice. One famous example is the <a href="https://arxiv.org/abs/1409.4842" rel="nofollow noreferrer">&quot;Inception&quot; network architecture</a>. To keep inner subnets from &quot;dying out&quot;,  several outputs from inner layers are extracted and passed into FC-&gt;Softmax. Then all the outputs are averaged in the loss function.</p>
  <p>From practical point of view, you won't be able to implement such things with basic <code>Sequential</code> model construction. Most libraries allow you to move beyond it, though, and this distinction is usually well-documented. For example in the <a href="https://www.tensorflow.org/guide/keras/functional" rel="nofollow noreferrer">tensorflow guide on functional API</a> it states right in the introduction:</p>
  <blockquote>
  <p>The functional API is a way to create models that are more flexible than the <code>tf.keras.Sequential</code> API. The functional API can handle models with non-linear topology, shared layers, and even multiple inputs or outputs.</p>
  </blockquote>
  <p>Hope that'll help you to get started.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-11T17:46:54.307Z
Answers.LastActivityDate: 2021-04-11T17:46:54.307Z
Answers.UserDisplayName: Kostya
Body: <p>For a network of the form:</p>
  <pre><code>Input(10)
  Dense(200)
  Dense(100+10)
  Dense(20)
  Output()
  </code></pre>
  <p>Those <code>+10</code> outputs are what I want to add to the standard <code>20</code> outputs, for my loss function.</p>
  <p>Is this possible - in theory or even with some pre-existing library?</p>
  
Tags: <neural-networks><tensorflow><keras><objective-functions><generative-adversarial-networks>
Title: Is it possible to use an internal layer's outputs in a loss function?
UserDisplayName: Tobi Akinyemi
ThreadText: # Is it possible to use an internal layer's outputs in a loss function?
  
  (Asked by: username_0 on 2021-04-10)
  
  For a network of the form:
  
  ```
  Input(10)
  Dense(200)
  Dense(100+10)
  Dense(20)
  Output()
  
  ```
  
  Those `+10` outputs are what I want to add to the standard `20` outputs, for my loss function.
  
  Is this possible - in theory or even with some pre-existing library?
  
  **Tags:** neural-networks, tensorflow, keras, objective-functions, generative-adversarial-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2021-04-11. Score: 2
  
  Yes, you can do that, and it is a standard practice. One famous example is the ["Inception" network architecture](https://arxiv.org/abs/1409.4842). To keep inner subnets from "dying out", several outputs from inner layers are extracted and passed into FC->Softmax. Then all the outputs are averaged in the loss function.
  
  From practical point of view, you won't be able to implement such things with basic `Sequential` model construction. Most libraries allow you to move beyond it, though, and this distinction is usually well-documented. For example in the [tensorflow guide on functional API](https://www.tensorflow.org/guide/keras/functional) it states right in the introduction:
  
  > The functional API is a way to create models that are more flexible than the `tf.keras.Sequential` API. The functional API can handle models with non-linear topology, shared layers, and even multiple inputs or outputs.
  
  Hope that'll help you to get started.
  
  
  ---
  
  