Body: <p><strong>CONTEXT</strong></p>
  <p>I am trying to build a regression model that finds the optimal parameters for a given input. The data I am using are point clouds, with <code>N</code> points and <code>3</code> coordinates (x,y,z) each. Each point cloud is divided into neighborhoods of constant size and, during inference, a batch of these neighborhoods are fed into the model which outputs a set of parameters. The parameters represent a family of surfaces and the goal is to find parameters such that the surface fits the neighborhood of points as tightly as possible (in the least squares sense).</p>
  <p><strong>THE ISSUE</strong></p>
  <p>The problem is that each type of parameter must fall into a specific range, otherwise it has no meaning. For example the first two parameters must lie inside [0.1, 1.9], the next three must be strictly positive etc.. I have tried restraining the outputs by adding a scaled sigmoid activation or simply clamping the output to the range that I want. However, it seems that such hacks result in saturation, the model outputs negative values and all the outputs become 0 from clamping.</p>
  <p>I can't imagine I'm the first one to encounter such a problem, but I haven't been able to find out a way to solve it. Is there a defacto way of dealing with this situation?</p>
  <p>P.S. I am not including details of the model architecture to keep this question general interest, but I will include them upon request, if it helps.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-24T17:13:15.200Z
Comments.Text: Hi @VlassisFo, I think you could try making a custom loss function, which would add extra punishment when the parameters go over the limit - this might help to discourage your model from spitting out large/wrong values.
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-24T17:24:53.667Z
Comments.Text: @markmark That's an interesting idea! How would i go about making the loss as strict as possible? This constrain needs to be enforced in as little iterations as possible, as the rest of the training is based on the correctness of these parameters.
Comments.UserDisplayName: VlassisFo
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-24T18:22:45.153Z
Comments.Text: For numbers within the acceptable range($[0.1, 1.9]$) your loss could work as normal. If a value, say,  $5.9$, you could add  normal loss, plus squared distance from the upper bound(in this case it would be $normal loss + (5.9-1.9)^2$). The further away the parameter is, the larger the loss will be. The neural net should be able to pick that up and learn quicker, in theory.
Comments.UserDisplayName: mark mark
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-24T18:24:52.170Z
Comments.Text: Beware of the numbers that are below 1 ($0.5^2 = 0.25$), as they become smaller when squared. You could add 1 to all the distances to ensure that they always become larger.
Comments.UserDisplayName: mark mark
Tags: <deep-learning><regression><geometric-deep-learning>
Title: How to restrain a model's outputs to a certain range without affecting its representative capacity?
UserDisplayName: VlassisFo
ThreadText: # How to restrain a model's outputs to a certain range without affecting its representative capacity?
  
  (Asked by: username_0 on 2021-02-24)
  
  **CONTEXT**
  
  I am trying to build a regression model that finds the optimal parameters for a given input. The data I am using are point clouds, with `N` points and `3` coordinates (x,y,z) each. Each point cloud is divided into neighborhoods of constant size and, during inference, a batch of these neighborhoods are fed into the model which outputs a set of parameters. The parameters represent a family of surfaces and the goal is to find parameters such that the surface fits the neighborhood of points as tightly as possible (in the least squares sense).
  
  **THE ISSUE**
  
  The problem is that each type of parameter must fall into a specific range, otherwise it has no meaning. For example the first two parameters must lie inside [0.1, 1.9], the next three must be strictly positive etc.. I have tried restraining the outputs by adding a scaled sigmoid activation or simply clamping the output to the range that I want. However, it seems that such hacks result in saturation, the model outputs negative values and all the outputs become 0 from clamping.
  
  I can't imagine I'm the first one to encounter such a problem, but I haven't been able to find out a way to solve it. Is there a defacto way of dealing with this situation?
  
  P.S. I am not including details of the model architecture to keep this question general interest, but I will include them upon request, if it helps.
  
  **Tags:** deep-learning, regression, geometric-deep-learning
  
  > **Comments:**
  > **username_1**: Hi @username_0, I think you could try making a custom loss function, which would add extra punishment when the parameters go over the limit - this might help to discourage your model from spitting out large/wrong values.  
  > **username_0**: @username_1 That's an interesting idea! How would i go about making the loss as strict as possible? This constrain needs to be enforced in as little iterations as possible, as the rest of the training is based on the correctness of these parameters.  
  > **username_1**: For numbers within the acceptable range($[0.1, 1.9]$) your loss could work as normal. If a value, say,  $5.9$, you could add  normal loss, plus squared distance from the upper bound(in this case it would be $normal loss + (5.9-1.9)^2$). The further away the parameter is, the larger the loss will be. The neural net should be able to pick that up and learn quicker, in theory.  
  > **username_1**: Beware of the numbers that are below 1 ($0.5^2 = 0.25$), as they become smaller when squared. You could add 1 to all the distances to ensure that they always become larger.  
  
  ---
  
  ## 0 answers
  
  