Answers.Body: <p>You can limit the absolute values of the outputs by "punishing" large values in the loss function. This is done by adding an extra term to the loss function. </p>
  
  <p>For example, if your existing loss function is <em>L(yhat, y)</em> where <em>yhat</em> is the output and <em>y</em> the correct value, create a new loss function </p>
  
  <p><em>L'(yhat, y) = L(yhat,y) + max(0, ||yhat||<sub>1</sub> - k)</em> </p>
  
  <p>When the outputs are "small" the L<sub>1</sub> norm of your output <em>yhat</em> is less than <em>k</em> and imposes no extra loss (keeping everything as before). However once the norm grows above <em>k</em> it starts to impose a loss that guides the training to reduce the absolute values of the outputs values.</p>
  
  <p>You can also limit the values in the network itself by selecting a bounded activation function for the output, for example <em>tanh</em> or <em>sigmoid</em>. However, you need to pick an activation function that suits your problem.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-12-04T14:59:44.620Z
Answers.LastActivityDate: 2017-12-04T14:59:44.620Z
Answers.UserDisplayName: mjul
Body: <p>I'm having the following problem: `</p>
  
  <p>I'm training a multi-output CNN and using the relative values of the outputs in my loss function. The net is learning well, but as the absolute values of the outputs are not regularized in anyway in the loss function, the values of the outputs keep rising. This causes a situation where the result I need (values of the outputs relative to each other) are quite right, but the huge absolute values produce underflow resulting NaN values at some point of the training. Is there some way to constrain these absolute values (perhaps elsewhere than the loss function?)</p>
  
  <p>I'm using a custom loss function implementing a recovery angular error metric, so the loss function is somewhat complex. A weighed mean of the net outputs is fed to this function, so using both the net outputs and the weighed mean would lead to some quite problematic gradient derivation. Would it be maybe possible to constrain the value range of the net outputs in the net structure itself?</p>
  
Tags: <convolutional-neural-networks>
Title: Constraining the output value range of a CNN independent of the loss function
UserDisplayName: Alfred J Kwak
ThreadText: # Constraining the output value range of a CNN independent of the loss function
  
  (Asked by: username_0 on 2017-10-31)
  
  I'm having the following problem: `
  
  I'm training a multi-output CNN and using the relative values of the outputs in my loss function. The net is learning well, but as the absolute values of the outputs are not regularized in anyway in the loss function, the values of the outputs keep rising. This causes a situation where the result I need (values of the outputs relative to each other) are quite right, but the huge absolute values produce underflow resulting NaN values at some point of the training. Is there some way to constrain these absolute values (perhaps elsewhere than the loss function?)
  
  I'm using a custom loss function implementing a recovery angular error metric, so the loss function is somewhat complex. A weighed mean of the net outputs is fed to this function, so using both the net outputs and the weighed mean would lead to some quite problematic gradient derivation. Would it be maybe possible to constrain the value range of the net outputs in the net structure itself?
  
  **Tags:** convolutional-neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-12-04. Score: 1
  
  You can limit the absolute values of the outputs by "punishing" large values in the loss function. This is done by adding an extra term to the loss function.
  
  For example, if your existing loss function is *L(yhat, y)* where *yhat* is the output and *y* the correct value, create a new loss function
  
  *L'(yhat, y) = L(yhat,y) + max(0, ||yhat||1 - k)*
  
  When the outputs are "small" the L1 norm of your output *yhat* is less than *k* and imposes no extra loss (keeping everything as before). However once the norm grows above *k* it starts to impose a loss that guides the training to reduce the absolute values of the outputs values.
  
  You can also limit the values in the network itself by selecting a bounded activation function for the output, for example *tanh* or *sigmoid*. However, you need to pick an activation function that suits your problem.
  
  
  ---
  
  