<p>Well let's consider one, Ridge regression.<br />
We have 2 terms:</p>
<ul>
<li>the regression loss <span class="math-container">$L^{pred} = \sum(f(x) - y)^2$</span>, which we can see that it is a sum of squared values, thus <span class="math-container">$L^{pred} \ge 0$</span></li>
<li>the regularization loss <span class="math-container">$L^{reg} = \sum w_i^2$</span>, which we can also see that it's composed by a sum of squared values, thus <span class="math-container">$L^{reg} \ge 0$</span></li>
</ul>
<p>Now, we have Ridge, which would sum them <span class="math-container">$L = L^{pred} + L^{reg}$</span>, which will ask the model to &quot;balance&quot; the use of the parameters with the predictions.</p>
<p>However, considering the case where we multiply them, we have <span class="math-container">$L = L^{pred} \cdot L^{reg}$</span>, and knowing that the two terms are greater or equal then zero, we have <span class="math-container">$L \ge 0$</span>.</p>
<p>Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.</p>
<p>How can we make <span class="math-container">$L = L^{pred} \cdot L^{reg} = 0$</span>?<br />
Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how <span class="math-container">$L^{reg}$</span> can be easily be zeroed, just by putting the parameters of your model (<span class="math-container">$\beta$</span> in your formula) to 0</p>
<p>So, multiplying the loss by a regularization based on norms (at least with <span class="math-container">$p\ge1$</span>) will just drive the model to learn the 0-function, by setting all the weights to 0</p>

CC BY-SA 4.0
2023-11-20T15:02:50.650Z
Yes, seems very obvious in retrospect, thanks. (A fix may be to do something like objective = RSS + (RSS*p-norm))
BigMistake
CC BY-SA 4.0
2023-11-20T15:23:51.197Z
@BigMistake yes, most of the things in optimization I would say that look obvious in retrospect, but I knew the answer because it was a thing I also though long time ago, so i already went down the rabbit hole (and wrt the fix, idk, that's equivalent to $(1 + norm)*RSS$ and yes it might work, but idk which property would that have)
Alberto
CC BY-SA 4.0
2023-11-19T11:49:10.313Z
2023-11-19T11:49:10.313Z
Alberto
<p>The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term <em>added</em> to a penalty term based on a norm of the coefficients.</p>
<p>What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?</p>
<p>For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:</p>
<ul>
<li>Lasso (L1 norm): <span class="math-container">$$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta_j| \right)$$</span></li>
<li>Ridge (L2 norm): <span class="math-container">$$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta_j^2 \right)$$</span></li>
</ul>
<p>Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?</p>

CC BY-SA 4.0
2023-11-20T12:11:37.480Z
Sorry, just wanted to add a small point on the answer I gave... there exists a similar thing to what you are proposing in the optimization community https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares, but the function that you are multiplying must be something non trivial (like regularization).... maybe you are interested in, so wanted to let you know
Alberto
<regression><regularization><normalisation><l2-regularization><l1-regularization>
What are the consequences when we multiply, instead of add, a penalty term?
BigMistake
# What are the consequences when we multiply, instead of add, a penalty term?

(Asked by: username_0 on 2023-11-19)

The typical objective function in regression problems like Lasso or Ridge includes a Residual Sum of Squares (RSS) term *added* to a penalty term based on a norm of the coefficients.

What are the consequences of modifying the objective function by multiplying (instead of adding) the Residual Sum of Squares (RSS) term with a p-norm?

For example, considering the Lasso (L1 norm) and Ridge (L2 norm), we have:

* Lasso (L1 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum |\beta\_j| \right)$$
* Ridge (L2 norm): $$\text{Objective} = \text{RSS} \times \left( \lambda \sum \beta\_j^2 \right)$$

Would this model change any behavior characteristics of the solution, like optimization process, sparsity or robustness, from the standard Lasso or Ridge? I.e., what are the known implications? Is there a name for this, or practical uses?

**Tags:** regression, regularization, normalisation, l2-regularization, l1-regularization

> **Comments:**
> **username_1**: Sorry, just wanted to add a small point on the answer I gave... there exists a similar thing to what you are proposing in the optimization community https://en.wikipedia.org/wiki/Iteratively_reweighted_least_squares, but the function that you are multiplying must be something non trivial (like regularization).... maybe you are interested in, so wanted to let you know  

---

## 1 answers

### Answer by username_1 on 2023-11-19. Score: 2 | Accepted answer

Well let's consider one, Ridge regression.  
We have 2 terms:

* the regression loss $L^{pred} = \sum(f(x) - y)^2$, which we can see that it is a sum of squared values, thus $L^{pred} \ge 0$
* the regularization loss $L^{reg} = \sum w\_i^2$, which we can also see that it's composed by a sum of squared values, thus $L^{reg} \ge 0$

Now, we have Ridge, which would sum them $L = L^{pred} + L^{reg}$, which will ask the model to "balance" the use of the parameters with the predictions.

However, considering the case where we multiply them, we have $L = L^{pred} \cdot L^{reg}$, and knowing that the two terms are greater or equal then zero, we have $L \ge 0$.

Now, since we want to minimize the loss, and the loss at most is 0, then having a 0 loss will be a minima of such loss.

How can we make $L = L^{pred} \cdot L^{reg} = 0$?  
Well, we just need one of the two terms to be zero for a multiplication being zero, and we can notice how $L^{reg}$ can be easily be zeroed, just by putting the parameters of your model ($\beta$ in your formula) to 0

So, multiplying the loss by a regularization based on norms (at least with $p\ge1$) will just drive the model to learn the 0-function, by setting all the weights to 0

> **Comments:**
> **username_0**: Yes, seems very obvious in retrospect, thanks. (A fix may be to do something like objective = RSS + (RSS*p-norm))  
> **username_1**: @username_0 yes, most of the things in optimization I would say that look obvious in retrospect, but I knew the answer because it was a thing I also though long time ago, so i already went down the rabbit hole (and wrt the fix, idk, that's equivalent to $(1 + norm)*RSS$ and yes it might work, but idk which property would that have)  

---

