Answers.Body: <p>Well let's consider one, Ridge regression.<br />
  We have 2 terms:</p>
  <ul>
  <li>the regression loss <span class="math-container">$L^{pred} = \sum(f(x) - y)^2$</span>, which we can see that it is a sum of squared values, thus <span class="math-container">$L^{pred} \ge 0$</span></li>
  <li>the regularization loss <span class="math-container">$L^{reg} = \sum w_i^2$</span>, which we can also see that it's composed by a sum of squared values, thus <span class="math-container">$L^{reg} \ge 0$</span></li>
  </ul>
  <p>Now, we have Ridge, which would sum them <span class="math-container">$L = L^{pred} + L^{reg}$</span>, which will ask the model to &quot;balance&quot; the use of the parameters with the predictions.</p>
  <p>However, considering the case where we multiply them, we have <span class="math-container">$L = L^{pred} \cdot L^{reg}$</span>, and knowing that the two terms are greater or equal then zero, we have <span class="math-container">$L \ge 0$</span>.</p>
  <p>Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.</p>
  <p>How can we make <span class="math-container">$L = L^{pred} \cdot L^{reg} = 0$</span>?<br />
  Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how <span class="math-container">$L^{reg}$</span> can be easily be zeroed, just by putting the parameters of your model (<span class="math-container">$\beta$</span> in your formula) to 0</p>
  <p>So, multiplying the loss by a regularization based on norms (at least with <span class="math-container">$p\ge1$</span>) will just drive the model to learn the 0-function, by setting all the weights to 0</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T15:02:50.650Z
Answers.Text: Yes, seems very obvious in retrospect, thanks. (A fix may be to do something like objective = RSS + (RSS*p-norm))
Answers.UserDisplayName: BigMistake
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-20T15:23:51.197Z
Answers.Text: @BigMistake yes, most of the things in optimization I would say that look obvious in retrospect, but I knew the answer because it was a thing I also though long time ago, so i already went down the rabbit hole (and wrt the fix, idk, that's equivalent to $(1 + norm)*RSS$ and yes it might work, but idk which property would that have)
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-19T11:49:10.313Z
Answers.LastActivityDate: 2023-11-19T11:49:10.313Z
Answers.UserDisplayName: Alberto
Body: <p>The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term <em>added</em> to a penalty term based on a norm of the coefficients.</p>
  <p>What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?</p>
  <p>For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:</p>
  <ul>
  <li>Lasso (L1 norm): <span class="math-container">$$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta_j| \right)$$</span></li>
  <li>Ridge (L2 norm): <span class="math-container">$$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta_j^2 \right)$$</span></li>
  </ul>
  <p>Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-20T12:11:37.480Z
Comments.Text: Sorry, just wanted to add a small point on the answer I gave... there exists a similar thing to what you are proposing in the optimization community https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares, but the function that you are multiplying must be something non trivial (like regularization).... maybe you are interested in, so wanted to let you know
Comments.UserDisplayName: Alberto
Tags: <regression><regularization><normalisation><l2-regularization><l1-regularization>
Title: What are the consequences when we multiply, instead of add, a penalty term?
UserDisplayName: BigMistake
ThreadText: # What are the consequences when we multiply, instead of add, a penalty term?
  
  (Asked by: username_0 on 2023-11-19)
  
  The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term *added* to a penalty term based on a norm of the coefficients.
  
  What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?
  
  For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:
  
  * Lasso (L1 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta\_j| \right)$$
  * Ridge (L2 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta\_j^2 \right)$$
  
  Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?
  
  **Tags:** regression, regularization, normalisation, l2-regularization, l1-regularization
  
  > **Comments:**
  > **username_1**: Sorry, just wanted to add a small point on the answer I gave... there exists a similar thing to what you are proposing in the optimization community https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares, but the function that you are multiplying must be something non trivial (like regularization).... maybe you are interested in, so wanted to let you know  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-19. Score: 2 | Accepted answer
  
  Well let's consider one, Ridge regression.  
  We have 2 terms:
  
  * the regression loss $L^{pred} = \sum(f(x) - y)^2$, which we can see that it is a sum of squared values, thus $L^{pred} \ge 0$
  * the regularization loss $L^{reg} = \sum w\_i^2$, which we can also see that it's composed by a sum of squared values, thus $L^{reg} \ge 0$
  
  Now, we have Ridge, which would sum them $L = L^{pred} + L^{reg}$, which will ask the model to "balance" the use of the parameters with the predictions.
  
  However, considering the case where we multiply them, we have $L = L^{pred} \cdot L^{reg}$, and knowing that the two terms are greater or equal then zero, we have $L \ge 0$.
  
  Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.
  
  How can we make $L = L^{pred} \cdot L^{reg} = 0$?  
  Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how $L^{reg}$ can be easily be zeroed, just by putting the parameters of your model ($\beta$ in your formula) to 0
  
  So, multiplying the loss by a regularization based on norms (at least with $p\ge1$) will just drive the model to learn the 0-function, by setting all the weights to 0
  
  > **Comments:**
  > **username_0**: Yes, seems very obvious in retrospect, thanks. (A fix may be to do something like objective = RSS + (RSS*p-norm))  
  > **username_1**: @username_0 yes, most of the things in optimization I would say that look obvious in retrospect, but I knew the answer because it was a thing I also though long time ago, so i already went down the rabbit hole (and wrt the fix, idk, that's equivalent to $(1 + norm)*RSS$ and yes it might work, but idk which property would that have)  
  
  ---
  
  