Answers.Body: <p>For the above stated artificial network, these two training scenarios are similar.</p>
  
  <ul>
  <li>Training to converge to the ideal output vector at the point after the last layer's activation functions are applied</li>
  <li>Training to converge to the vector formed by applying <span class="math-container">$tan$</span> functions to each component of the ideal output vector, when convergence occurs at the point just after the vector-matrix multiplication with the last layer's parameters, prior to the last layer's <span class="math-container">$atan$</span> activation functions</li>
  </ul>
  
  <p>Distinctions between them include these.</p>
  
  <ul>
  <li>The applications of gradients and associated code must be adapted to the modification of the starting point of back propagation to before the final <span class="math-container">$atan$</span> activation functions.</li>
  <li>The slope and curvature of the loss function will differ if the same loss function is used for the two scenarios, so the accuracy, speed, and reliability of convergence will also be different.</li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-14T05:10:07.937Z
Answers.LastActivityDate: 2018-10-14T05:10:07.937Z
Answers.UserDisplayName: Douglas Daseeco
Answers.Body: <p>The main difference your change would have is to allow you to apply a loss function to a different part of the network. This may affect training.</p>
  
  <p>If you keep the same loss function (e.g. MSE), but apply it to the pre-transformed values, then you will have changed the objective of the network, perhaps significantly. Whether or not this is a good thing depends on how much you needed the original loss function. However, the fact that it would result in a different training target is usually going to be a bad thing if your original training target was correct. This will also be true if you pick a new arbitrary loss function that seems to fit the pre-transform representation better.</p>
  
  <p>If you engineer a "correct" loss function such that the objective of the network remains unchanged, then the behaviour of the network will not change much - probably not at all. However, in some cases this can lead to more stable and/or faster training - it is often used for classifiers to avoid need to use exponentiation, see <a href="https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits" rel="nofollow noreferrer">tf.nn.softmax_cross_entropy_with_logits</a> in TensorFlow, which does exactly this.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-11T11:03:23.170Z
Answers.LastActivityDate: 2018-10-11T11:09:07.640Z
Answers.LastEditDate: 2018-10-11T11:09:07.640Z
Answers.UserDisplayName: Neil Slater
Answers.Body: <p>Be careful to specify exactly what you mean by <code>adjust weights and biases of the network so that the "pre-activated output" ... approaches ...</code>. </p>
  
  <p>When training a neural network, one minimizes a loss function. This loss function is determines how important the deviation from <code>0.01396</code> is compared to the deviation of the other node from <code>-0.00524</code>. By transforming the target labels backwards you should also express the original loss function in terms of the back-transformed labels.</p>
  
  <hr>
  
  <p>What one can do in some cases is to combine the input to the last layer's activation function with the loss function and algebraically simplify the resulting expression.</p>
  
  <p>This concept is for example implemented in Tensorflow's <a href="https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits" rel="nofollow noreferrer">tf.nn.sigmoid_cross_entropy_with_logits</a>. This function can be used for the case of a single output with sigmoid activation function and binary cross-entropy loss (a similar function also exists for the case of multiple output nodes with a softmax activation function).</p>
  
  <p>Instead of first passing the values through a sigmoid activation and then calculating the binary cross-entropy with respect to the target label, it combines the two expressions and uses an equivalent but simpler expression.</p>
  
  <p>If you look at the documentation of this function, you'll see that the number of transcendental functions (which are computationally expensive to calculate) can be reduced from</p>
  
  <p><code>loss(x,z) = z * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x)))
  </code></p>
  
  <p>to</p>
  
  <p><code>loss(x,z) = max(x, 0) - x * z + log(1 + exp(-abs(x)))
  </code></p>
  
  <p>where <code>x</code> is the input to the (sigmoid) nonlinearity (corresponding to the output of the green nodes in your diagram), also called 'logits' and <code>z</code> is the target label.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T12:04:15.123Z
Answers.Text: So, based on this discourse, is it suitable or not?
Answers.UserDisplayName: Douglas Daseeco
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-15T14:57:22.400Z
Answers.Text: it is not clear from the question whether the OP is transforming his loss function to account for the missing tanh nonlinearity or not -- as I mentioned above he did not write how he defines the new minimization goal before the last nonlinearity. If he keeps the same loss as at the output nodes, the method is finding a different type of minimum (and therefore the method is unsuitable). If he adapts the loss to account for the removed tanh nonlinearity then it depends if one gets some simplification in the new loss function or not. It is not clear what loss function he uses and why tanh .
Answers.UserDisplayName: Andre Holzner
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-11T20:37:12.757Z
Answers.LastActivityDate: 2018-10-11T20:37:12.757Z
Answers.UserDisplayName: Andre Holzner
Answers.Body: <p>I think your idea would work... fine, but I don't necessarily see any advantages to it. I haven't actually tried it (that'd be the best way for you to also see whether it works!), so I'm mostly going by first thoughts and intuition here.</p>
  
  <p>Anyway, what you are essentially doing with your idea is "cutting off" the last layer of the Neural Network from the perspective of your learning algorithm (typically backpropagation). Whatever weights you have between the last hidden layer and the output layer will be fixed to their initial values. The last hidden layer can actually be viewed as an "output" layer, since you also have fixed targets that you want to converge towards for them.</p>
  
  <p>Whether this makes your learning process better/faster/easier, or worse/slower/harder seems to be very much dependent on how the weights between your last hidden layer and your output layer are initialized. For example:</p>
  
  <ul>
  <li>If those weights are initialized to all-zero, your "real" output layer is doomed to always predict zeros, so your problem becomes impossible to solve.</li>
  <li>If those weights are initialized to implement the identity function, this becomes 100% equivalent to the case you would have if you'd simply cut off the last layer and train that in the traditional sense (i.e. you effectively have one layer less than you really do).</li>
  <li>If those weights are initialized randomly, it looks to me like you have a post-processing step consisting of a random projection. Such a random projection may be beneficial for training (random projections can be useful for dimensionality reduction, or for, in combination with the subsequent non-linear function, turning an otherwise linear function into a non-linear function). </li>
  </ul>
  
  <p>I don't think it'd very often be better than actually have an extra "real", trainable layer with a non-linear activation function though. I suspect such a "non-trainable" extra layer can sometimes be better than not having anything there, but I don't think it'd often be better than having a real, trainable layer.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-11T10:21:03.643Z
Answers.LastActivityDate: 2018-10-11T10:28:00.567Z
Answers.LastEditDate: 2018-10-11T10:28:00.567Z
Answers.UserDisplayName: Dennis Soemers
Body: <p>I have a neural network with the following structure:
  <a href="https://i.sstatic.net/eHMGH.jpg" rel="noreferrer"><img src="https://i.sstatic.net/eHMGH.jpg" alt="Neural Network with target outputs: 0.8,-0.3"></a></p>
  
  <p>I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the upper output node and -0.3 for the lower output node.</p>
  
  <p>The activations function used for the first 2 layers are ReLu or LeakyReLu while the last layer uses atan as an activation function.</p>
  
  <p>For back propogation, instead of adjusting values to make the network's output approach 0.8,-0.3. is it suitable if I use the inverse function for atan -which is tan itself- to get "the ideal input to the output layer multiplied by weights and adjusted by biases". </p>
  
  <p>The tan of 0.8 and -0.3 is 0.01396 and -0.00524 approximately. </p>
  
  <p>My algorithm would then adjust weights and biases of the network so that the "pre-activated output" of the output layer -which is basically (sum(output_layer_weight*output_layer's inputs)+output_layer_biases)- approaches 0.01396 and -0.00524.</p>
  
  <p>Is this suitable </p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-09T15:35:32.760Z
Comments.Text: As mentioned in the topic of the question I am only considering to apply this on the last layer since there is no 'nesting' of multiple activation functions.
Comments.UserDisplayName: Vikhyat Agarwal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-09T15:39:58.620Z
Comments.Text: Your third point on over fitting might be invalid since the method i suggest only finds inverse values of the last activation function to apply back propogation to the layers before it so that the input values of the last layer approaches the inverse function values of the target.
Comments.UserDisplayName: Vikhyat Agarwal
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-13T08:54:03.037Z
Comments.Text: What do you want to achieve? What is your goal?
Comments.UserDisplayName: Martin Thoma
Tags: <neural-networks><activation-functions>
Title: Is it suitable to find inverse of last layer's activation function and apply it on the target output?
UserDisplayName: Vikhyat Agarwal
ThreadText: # Is it suitable to find inverse of last layer's activation function and apply it on the target output?
  
  (Asked by: username_0 on 2018-10-06)
  
  I have a neural network with the following structure:
  [![Neural Network with target outputs: 0.8,-0.3](https://i.sstatic.net/eHMGH.jpg)](https://i.sstatic.net/eHMGH.jpg)
  
  I am expecting specific outputs from the neural network which are the target values for my training. Let's say the target values are 0.8 for the upper output node and -0.3 for the lower output node.
  
  The activations function used for the first 2 layers are ReLu or LeakyReLu while the last layer uses atan as an activation function.
  
  For back propogation, instead of adjusting values to make the network's output approach 0.8,-0.3. is it suitable if I use the inverse function for atan -which is tan itself- to get "the ideal input to the output layer multiplied by weights and adjusted by biases".
  
  The tan of 0.8 and -0.3 is 0.01396 and -0.00524 approximately.
  
  My algorithm would then adjust weights and biases of the network so that the "pre-activated output" of the output layer -which is basically (sum(output\_layer\_weight\*output\_layer's inputs)+output\_layer\_biases)- approaches 0.01396 and -0.00524.
  
  Is this suitable
  
  **Tags:** neural-networks, activation-functions
  
  > **Comments:**
  > **username_0**: As mentioned in the topic of the question I am only considering to apply this on the last layer since there is no 'nesting' of multiple activation functions.  
  > **username_0**: Your third point on over fitting might be invalid since the method i suggest only finds inverse values of the last activation function to apply back propogation to the layers before it so that the input values of the last layer approaches the inverse function values of the target.  
  > **username_1**: What do you want to achieve? What is your goal?  
  
  ---
  
  ## 4 answers
  
  ### Answer by username_2 on 2018-10-14. Score: 1 | Accepted answer
  
  For the above stated artificial network, these two training scenarios are similar.
  
  * Training to converge to the ideal output vector at the point after the last layer's activation functions are applied
  * Training to converge to the vector formed by applying $tan$ functions to each component of the ideal output vector, when convergence occurs at the point just after the vector-matrix multiplication with the last layer's parameters, prior to the last layer's $atan$ activation functions
  
  Distinctions between them include these.
  
  * The applications of gradients and associated code must be adapted to the modification of the starting point of back propagation to before the final $atan$ activation functions.
  * The slope and curvature of the loss function will differ if the same loss function is used for the two scenarios, so the accuracy, speed, and reliability of convergence will also be different.
  
  
  ---
  
  ### Answer by username_3 on 2018-10-11. Score: 2
  
  The main difference your change would have is to allow you to apply a loss function to a different part of the network. This may affect training.
  
  If you keep the same loss function (e.g. MSE), but apply it to the pre-transformed values, then you will have changed the objective of the network, perhaps significantly. Whether or not this is a good thing depends on how much you needed the original loss function. However, the fact that it would result in a different training target is usually going to be a bad thing if your original training target was correct. This will also be true if you pick a new arbitrary loss function that seems to fit the pre-transform representation better.
  
  If you engineer a "correct" loss function such that the objective of the network remains unchanged, then the behaviour of the network will not change much - probably not at all. However, in some cases this can lead to more stable and/or faster training - it is often used for classifiers to avoid need to use exponentiation, see [tf.nn.softmax\_cross\_entropy\_with\_logits](https://www.tensorflow.org/api_docs/python/tf/nn/softmax_cross_entropy_with_logits) in TensorFlow, which does exactly this.
  
  
  ---
  
  ### Answer by username_4 on 2018-10-11. Score: 1
  
  Be careful to specify exactly what you mean by `adjust weights and biases of the network so that the "pre-activated output" ... approaches ...`.
  
  When training a neural network, one minimizes a loss function. This loss function is determines how important the deviation from `0.01396` is compared to the deviation of the other node from `-0.00524`. By transforming the target labels backwards you should also express the original loss function in terms of the back-transformed labels.
  
  ---
  
  What one can do in some cases is to combine the input to the last layer's activation function with the loss function and algebraically simplify the resulting expression.
  
  This concept is for example implemented in Tensorflow's [tf.nn.sigmoid\_cross\_entropy\_with\_logits](https://www.tensorflow.org/api_docs/python/tf/nn/sigmoid_cross_entropy_with_logits). This function can be used for the case of a single output with sigmoid activation function and binary cross-entropy loss (a similar function also exists for the case of multiple output nodes with a softmax activation function).
  
  Instead of first passing the values through a sigmoid activation and then calculating the binary cross-entropy with respect to the target label, it combines the two expressions and uses an equivalent but simpler expression.
  
  If you look at the documentation of this function, you'll see that the number of transcendental functions (which are computationally expensive to calculate) can be reduced from
  
  `loss(x,z) = z * -log(1 / (1 + exp(-x))) + (1 - z) * -log(exp(-x) / (1 + exp(-x)))`
  
  to
  
  `loss(x,z) = max(x, 0) - x * z + log(1 + exp(-abs(x)))`
  
  where `x` is the input to the (sigmoid) nonlinearity (corresponding to the output of the green nodes in your diagram), also called 'logits' and `z` is the target label.
  
  > **Comments:**
  > **username_2**: So, based on this discourse, is it suitable or not?  
  > **username_4**: it is not clear from the question whether the OP is transforming his loss function to account for the missing tanh nonlinearity or not -- as I mentioned above he did not write how he defines the new minimization goal before the last nonlinearity. If he keeps the same loss as at the output nodes, the method is finding a different type of minimum (and therefore the method is unsuitable). If he adapts the loss to account for the removed tanh nonlinearity then it depends if one gets some simplification in the new loss function or not. It is not clear what loss function he uses and why tanh .  
  
  ---
  
  ### Answer by username_5 on 2018-10-11. Score: 1
  
  I think your idea would work... fine, but I don't necessarily see any advantages to it. I haven't actually tried it (that'd be the best way for you to also see whether it works!), so I'm mostly going by first thoughts and intuition here.
  
  Anyway, what you are essentially doing with your idea is "cutting off" the last layer of the Neural Network from the perspective of your learning algorithm (typically backpropagation). Whatever weights you have between the last hidden layer and the output layer will be fixed to their initial values. The last hidden layer can actually be viewed as an "output" layer, since you also have fixed targets that you want to converge towards for them.
  
  Whether this makes your learning process better/faster/easier, or worse/slower/harder seems to be very much dependent on how the weights between your last hidden layer and your output layer are initialized. For example:
  
  * If those weights are initialized to all-zero, your "real" output layer is doomed to always predict zeros, so your problem becomes impossible to solve.
  * If those weights are initialized to implement the identity function, this becomes 100% equivalent to the case you would have if you'd simply cut off the last layer and train that in the traditional sense (i.e. you effectively have one layer less than you really do).
  * If those weights are initialized randomly, it looks to me like you have a post-processing step consisting of a random projection. Such a random projection may be beneficial for training (random projections can be useful for dimensionality reduction, or for, in combination with the subsequent non-linear function, turning an otherwise linear function into a non-linear function).
  
  I don't think it'd very often be better than actually have an extra "real", trainable layer with a non-linear activation function though. I suspect such a "non-trainable" extra layer can sometimes be better than not having anything there, but I don't think it'd often be better than having a real, trainable layer.
  
  
  ---
  
  