Answers.Body: <h2>There are many ways of constraining the network's output.</h2>
  <p>Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by 180 to adapt to your output. Since the layer is part of the optimization process, gradients will be learned correctly.</p>
  <p>However, there are a few issues with it, namely sigmoid is not a linear curve, and is quite steep around 0. Maybe you can consider something like a ReLU, but with a maximum as well. This makes your output linearly between [0,180], and is also learned during optimization.</p>
  <p>You could also just clip the output without any activation layers. This is simple and works, but the optimization process sees no difference between outputting 180 or 180000. This <em>can</em> lead to problems, depending on your problem.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-10-26T11:06:46.973Z
Answers.LastActivityDate: 2018-10-26T11:06:46.973Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: BlueMoon93
Body: <p>I am training a deep neural network. There is a constraint on the output value of the neural network (e.g. the output has to be between 0 and 180). I think some possible solutions are using sigmoid, tanh activation at the end of the layer.</p>
  <p><strong>Are there better ways to put constraints on the output value of a neural network?</strong></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2018-10-26T07:43:13.790Z
Comments.Text: Please give more detail about how the output value is interpreted. This will help to understand what kind of loss function and activation of the last layer are suitable. Do you have 181 classes ? Is it a position or an angle ?
Comments.UserDisplayName: Andre Holzner
Tags: <neural-networks><deep-learning><keras><activation-functions><network-design>
Title: How to constraint the output value of a neural network?
UserDisplayName: raemoii
ThreadText: # How to constraint the output value of a neural network?
  
  (Asked by: username_0 on 2018-10-23)
  
  I am training a deep neural network. There is a constraint on the output value of the neural network (e.g. the output has to be between 0 and 180). I think some possible solutions are using sigmoid, tanh activation at the end of the layer.
  
  **Are there better ways to put constraints on the output value of a neural network?**
  
  **Tags:** neural-networks, deep-learning, keras, activation-functions, network-design
  
  > **Comments:**
  > **username_1**: Please give more detail about how the output value is interpreted. This will help to understand what kind of loss function and activation of the last layer are suitable. Do you have 181 classes ? Is it a position or an angle ?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2018-10-26. Score: 3
  
  ## There are many ways of constraining the network's output.
  
  Using an activation layer is a good one. If you sigmoid the output layer, the output is constrained between [0,1] and you can multiply that by 180 to adapt to your output. Since the layer is part of the optimization process, gradients will be learned correctly.
  
  However, there are a few issues with it, namely sigmoid is not a linear curve, and is quite steep around 0. Maybe you can consider something like a ReLU, but with a maximum as well. This makes your output linearly between [0,180], and is also learned during optimization.
  
  You could also just clip the output without any activation layers. This is simple and works, but the optimization process sees no difference between outputting 180 or 180000. This *can* lead to problems, depending on your problem.
  
  
  ---
  
  