Answers.Body: <p>Regularization is different than constraints. The problem with an L1 regularization term would be
  <span class="math-container">\begin{align*} 
  \underset{\theta}{\min} &amp; \quad f(\theta) + c\lVert \theta \rVert_1
  \end{align*}</span>
  whereas the same problem with an L1 constraint would be
  <span class="math-container">\begin{align*} 
  \underset{\theta}{\min} &amp; \quad f(\theta)  \\
  \text{such that } &amp; \quad \lVert \theta \rVert_1 \leq M \\
  &amp; \quad  
  \end{align*}</span>
  where <span class="math-container">$c$</span> and <span class="math-container">$M$</span> are constants which needs to be specified. These are different problems, with different values of <span class="math-container">$\theta$</span> for solutions, and need to be solved differently (in particular, the constrained problem is significantly harder to solve).</p>
  <p>Now, these two problems are related, and we would expect them to have similar solutions, provided <span class="math-container">$c$</span> and <span class="math-container">$M$</span> are chosen appropriately. <a href="https://stats.stackexchange.com/questions/485302/l1-regularization-vs-constraint">You might want to also read this answer on stats exchange</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-02-17T23:22:38.277Z
Answers.LastActivityDate: 2022-02-17T23:22:38.277Z
Answers.UserDisplayName: Taw
Body: <p>I have the following question on &quot;Regularization vs. Constrained Optimization&quot; :</p>
  <p>In the context of statistical modelling, we are often taught about &quot;Regularization&quot; as a method of dealing with the &quot;Bias-Variance Tradeoff&quot; (i.e. stabilizing the inconsistent performance of complicated models). When a L1-Norm or L2-Norm Penalty Term is added to the estimation function (corresponding to the statistical model) being optimized, some of the model parameters will either &quot;shrink&quot; in size towards 0, thus producing a &quot;sparser&quot; model that is more likely to retain its &quot;low bias&quot; but possible reduce its &quot;high variance&quot;:</p>
  <p><a href="https://i.sstatic.net/wLm3Q.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/wLm3Q.png" alt="enter image description here" /></a></p>
  <p>I have often heard of functions containing these L1-Norm and L2-Norm &quot;Penalty Terms&quot; being referred to as &quot;optimization constraints&quot; (i.e. the &quot;feasible region&quot; from which valid choices of model parameters can belong to has now been &quot;altered&quot; due to these &quot;norm penalty constraints&quot;):</p>
  <p><a href="https://i.sstatic.net/QMs1h.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/QMs1h.png" alt="enter image description here" /></a></p>
  <p><strong>My Question:</strong> When we estimate some statistical model's parameters and the estimation equation contains some &quot;regularization penalty term,&quot; would it be incorrect to refer to this as an example of &quot;constrained optimization&quot;?</p>
  <p>Is regularized optimization in Machine Learning and Statistical Modelling fundamentally any different (with the exception of usually being more difficult and solved using approximate stochastic iterative methods) from Constrained Optimization in Linear Programming?</p>
  <p><a href="https://i.sstatic.net/NvGEz.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NvGEz.png" alt="enter image description here" /></a></p>
  <p><strong>References:</strong></p>
  <ul>
  <li><a href="https://en.wikipedia.org/wiki/Regularization_(mathematics)" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Regularization_(mathematics)</a></li>
  <li><a href="http://ab-initio.mit.edu/wiki/index.php?title=NLopt_Tutorial" rel="nofollow noreferrer">http://ab-initio.mit.edu/wiki/index.php?title=NLopt_Tutorial</a></li>
  </ul>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-02-14T23:52:57.653Z
Comments.Text: Yes, it can be considered as a type of 'soft' constraint. For example, with these norm regularizations, you bias the learning process towards regions where parameter values are smaller.
Comments.UserDisplayName: SpiderRico
Tags: <neural-networks><regularization>
Title: Can we Consider Regularization as a "Constraint"?
UserDisplayName: stats_noob
ThreadText: # Can we Consider Regularization as a "Constraint"?
  
  (Asked by: username_0 on 2022-02-14)
  
  I have the following question on "Regularization vs. Constrained Optimization" :
  
  In the context of statistical modelling, we are often taught about "Regularization" as a method of dealing with the "Bias-Variance Tradeoff" (i.e. stabilizing the inconsistent performance of complicated models). When a L1-Norm or L2-Norm Penalty Term is added to the estimation function (corresponding to the statistical model) being optimized, some of the model parameters will either "shrink" in size towards 0, thus producing a "sparser" model that is more likely to retain its "low bias" but possible reduce its "high variance":
  
  [![enter image description here](https://i.sstatic.net/wLm3Q.png)](https://i.sstatic.net/wLm3Q.png)
  
  I have often heard of functions containing these L1-Norm and L2-Norm "Penalty Terms" being referred to as "optimization constraints" (i.e. the "feasible region" from which valid choices of model parameters can belong to has now been "altered" due to these "norm penalty constraints"):
  
  [![enter image description here](https://i.sstatic.net/QMs1h.png)](https://i.sstatic.net/QMs1h.png)
  
  **My Question:** When we estimate some statistical model's parameters and the estimation equation contains some "regularization penalty term," would it be incorrect to refer to this as an example of "constrained optimization"?
  
  Is regularized optimization in Machine Learning and Statistical Modelling fundamentally any different (with the exception of usually being more difficult and solved using approximate stochastic iterative methods) from Constrained Optimization in Linear Programming?
  
  [![enter image description here](https://i.sstatic.net/NvGEz.png)](https://i.sstatic.net/NvGEz.png)
  
  **References:**
  
  * <https://en.wikipedia.org/wiki/Regularization_(mathematics)>
  * <http://ab-initio.mit.edu/wiki/index.php?title=NLopt_Tutorial>
  
  **Tags:** neural-networks, regularization
  > **Comments:**
  > **username_1**: Yes, it can be considered as a type of 'soft' constraint. For example, with these norm regularizations, you bias the learning process towards regions where parameter values are smaller.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2022-02-17. Score: 1
  
  Regularization is different than constraints. The problem with an L1 regularization term would be
  \begin{align\*}
  \underset{\theta}{\min} & \quad f(\theta) + c\lVert \theta \rVert\_1
  \end{align\*}
  whereas the same problem with an L1 constraint would be
  \begin{align\*}
  \underset{\theta}{\min} & \quad f(\theta) \\
  \text{such that } & \quad \lVert \theta \rVert\_1 \leq M \\
  & \quad
  \end{align\*}
  where $c$ and $M$ are constants which needs to be specified. These are different problems, with different values of $\theta$ for solutions, and need to be solved differently (in particular, the constrained problem is significantly harder to solve).
  
  Now, these two problems are related, and we would expect them to have similar solutions, provided $c$ and $M$ are chosen appropriately. [You might want to also read this answer on stats exchange](https://stats.stackexchange.com/questions/485302/l1-regularization-vs-constraint)
  
  
  ---
  
  