Body: <p>I am wondering if it makes sense to parameterize simple weight inequalities, for example if the weights should be <span class="math-container">$w\geq 0$</span>, one cound train <span class="math-container">$\exp w$</span> over the unconstrained set instead. Also, if <span class="math-container">$\sum w_i=1$</span> one could parameterize <span class="math-container">$\frac{e^{w_i}}{\sum e^{w_i}}$</span> and optimize over the unconstrained set.</p>
  <p>While the solutions should be the same, I wonder if it makes numerically a difference, and what way is the right one. I guess the values of <span class="math-container">$\exp$</span> might explode and it becomes numerically unstable.</p>
  <p>Question: should I use something like tf.keras.constraints.NonNeg() or should I replace each weight <span class="math-container">$w$</span>, on which I want to apply the constraint with <span class="math-container">$\exp w$</span> (&quot;reparameterization&quot;) (since exp&gt;0)or is there no general rule?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-16T09:20:15.293Z
Comments.Text: Can you please reformulate the title in the form of a **specific** question, so that it's clear what you're really asking here? I don't see any question in your post!
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-06-16T10:17:15.640Z
Comments.Text: Sorry if it is not clear. I want to use a non-negative weight constraint in one layer. Question: should I use something like tf.keras.constraints.NonNeg() or should I replace each weight $w$ on which I want to apply the constraint with $\exp w$ ("reparameterization") (since $\exp >0$)or is there no general rule?
Comments.UserDisplayName: Philipp123
Tags: <constrained-optimization><numerical-algorithms>
Title: For simple weight constraints: Add constraint directly or use parameterization without constraint
UserDisplayName: Philipp123
ThreadText: # For simple weight constraints: Add constraint directly or use parameterization without constraint
  
  (Asked by: username_0 on 2022-06-14)
  
  I am wondering if it makes sense to parameterize simple weight inequalities, for example if the weights should be $w\geq 0$, one cound train $\exp w$ over the unconstrained set instead. Also, if $\sum w\_i=1$ one could parameterize $\frac{e^{w\_i}}{\sum e^{w\_i}}$ and optimize over the unconstrained set.
  
  While the solutions should be the same, I wonder if it makes numerically a difference, and what way is the right one. I guess the values of $\exp$ might explode and it becomes numerically unstable.
  
  Question: should I use something like tf.keras.constraints.NonNeg() or should I replace each weight $w$, on which I want to apply the constraint with $\exp w$ ("reparameterization") (since exp>0)or is there no general rule?
  
  **Tags:** constrained-optimization, numerical-algorithms
  > **Comments:**
  > **username_1**: Can you please reformulate the title in the form of a **specific** question, so that it's clear what you're really asking here? I don't see any question in your post!  
  > **username_0**: Sorry if it is not clear. I want to use a non-negative weight constraint in one layer. Question: should I use something like tf.keras.constraints.NonNeg() or should I replace each weight $w$ on which I want to apply the constraint with $\exp w$ ("reparameterization") (since $\exp >0$)or is there no general rule?  
  
  ---
  
  ## 0 answers
  
  