Answers.Body: <p>In <a href="https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/l1" rel="nofollow noreferrer">L1 regularization</a>, the penalty term you compute for every parameter is a function of the absolute value of a given weight (times some regularization factor).
  Thus, irrespective of whether a weight is positive or negative (due to the absolute value) and irrespective of how large the weight is, there will be a penalty incurred as long as weight is unequal 0. So, the only way how a training procedure can considerably reduce the L1 regularization penalty is by driving all (unnecessary) weights towards 0, which results in a sparse representation.</p>
  <p>Of course, the <a href="https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/l2" rel="nofollow noreferrer">L2 regularization</a> will also only be strictly 0 when all weights are 0. However, in L2, the contribution of a weight to the L2 penalty is proportional to the <em><strong>squared</strong></em> value of the weight. Therefore, a weight whose absolute value is smaller than 1, i.e. <span class="math-container">$abs(weight) &lt; 1$</span>, will be much less punished by L2 than it would be by L1, which means that L2 puts less emphasis on driving all weights towards exactly 0. This is because squaring a some value in (0,1) will result in a value of lower magnitude than taking the un-squared value itself: <span class="math-container">$x^2 &lt; x\ for\ all\ x\ with\ abs(x) &lt; 1$</span>.</p>
  <p>So, while both regularization terms end up being 0 only when weights are 0, the L1 term penalizes small weights with <span class="math-container">$abs(x) &lt; 1$</span> much more strongly than L2 does, thereby driving the weight more strongly towards 0 than L2 does.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-02T10:48:36.283Z
Answers.LastActivityDate: 2020-07-02T17:12:04.663Z
Answers.LastEditDate: 2020-07-02T17:12:04.663Z
Answers.UserDisplayName: Daniel B.
Body: <blockquote>
  <p>In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero.</p>
  </blockquote>
  <p>What's the reason for the above statement - could someone explain it mathematically, and/or provide some intuition (maybe geometric)?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-29T22:25:14.213Z
Comments.Text: This is a relatively old question, but, for reproducibility, could you please provide a link to the article/paper where you took this excerpt that you're quoting from?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-01-30T12:17:39.027Z
Comments.Text: Oops, at this point, I don't remember. I'll take care next time!
Comments.UserDisplayName: stoic-santiago
Tags: <machine-learning><regularization><l2-regularization><l1-regularization>
Title: Why does L1 regularization yield sparse features?
UserDisplayName: stoic-santiago
ThreadText: # Why does L1 regularization yield sparse features?
  
  (Asked by: username_0 on 2020-07-02)
  
  > In contrast to L2 regularization, L1 regularization usually yields sparse feature vectors and most feature weights are zero.
  
  What's the reason for the above statement - could someone explain it mathematically, and/or provide some intuition (maybe geometric)?
  
  **Tags:** machine-learning, regularization, l2-regularization, l1-regularization
  > **Comments:**
  > **username_1**: This is a relatively old question, but, for reproducibility, could you please provide a link to the article/paper where you took this excerpt that you're quoting from?  
  > **username_0**: Oops, at this point, I don't remember. I'll take care next time!  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-07-02. Score: 3 | Accepted answer
  
  In [L1 regularization](https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/l1), the penalty term you compute for every parameter is a function of the absolute value of a given weight (times some regularization factor).
  Thus, irrespective of whether a weight is positive or negative (due to the absolute value) and irrespective of how large the weight is, there will be a penalty incurred as long as weight is unequal 0. So, the only way how a training procedure can considerably reduce the L1 regularization penalty is by driving all (unnecessary) weights towards 0, which results in a sparse representation.
  
  Of course, the [L2 regularization](https://www.tensorflow.org/api_docs/python/tf/keras/regularizers/l2) will also only be strictly 0 when all weights are 0. However, in L2, the contribution of a weight to the L2 penalty is proportional to the ***squared*** value of the weight. Therefore, a weight whose absolute value is smaller than 1, i.e. $abs(weight) < 1$, will be much less punished by L2 than it would be by L1, which means that L2 puts less emphasis on driving all weights towards exactly 0. This is because squaring a some value in (0,1) will result in a value of lower magnitude than taking the un-squared value itself: $x^2 < x\ for\ all\ x\ with\ abs(x) < 1$.
  
  So, while both regularization terms end up being 0 only when weights are 0, the L1 term penalizes small weights with $abs(x) < 1$ much more strongly than L2 does, thereby driving the weight more strongly towards 0 than L2 does.
  
  
  ---
  
  