Answers.Body: <p>We will describe the input to the network as a vector, called features vector. Each component of this vector is usually related to some &quot;real world&quot; information, by example &quot;age of the person&quot;, &quot;number of atoms&quot;, &quot;...&quot;.</p>
  <p>In very usual situations, a specific component of the input vector will have near than always the same value. This is more usual in binary components or components that has a small set of possible values.</p>
  <p>However, usually <strong>in the cases where this component has a value different from the most usual one, this component is very important and informative</strong>.</p>
  <p>These values of this kind of components are called <em>infrequent features</em>.</p>
  <p>(example: &quot;rains?&quot; is in my city 99.9% of time &quot;false&quot;. However, when it is true, it is a key factor to all questions about the behavior of the population).</p>
  <p>The problem with these features: as unusual values are infrequent, the net has few chances to learn from them, and some learning algorithms could fail to give them the weight that they must (taken into account that, as has been said, these components are very important when they take a value different to the most frequent one).</p>
  <p>Some adaptive learning rate algorithms as AdaGrad tries to solve this issue.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-23T12:15:27.190Z
Answers.Text: Thank you very much for your answer, it is very clear :)
Answers.UserDisplayName: ava_punksmash
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-23T12:27:34.193Z
Answers.Text: OK thank you for the advice ! Edit: for now my vote is counted but not taken into account because i am a new user, but i do appreciate your help
Answers.UserDisplayName: ava_punksmash
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-10-23T11:53:51.970Z
Answers.LastActivityDate: 2020-10-23T15:28:50.167Z
Answers.LastEditDate: 2020-10-23T15:28:50.167Z
Answers.UserDisplayName: pasaba por aqui
Body: <p>I am reading this blog post: <a href="https://ruder.io/optimizing-gradient-descent/index.html" rel="nofollow noreferrer">https://ruder.io/optimizing-gradient-descent/index.html</a>. In the section about AdaGrad, it says:</p>
  <blockquote>
  <p>It adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.</p>
  </blockquote>
  <p>But I am not sure about the meaning of <em>infrequent features</em>: is it that the value of a given feature changes rarely?</p>
  
Tags: <deep-learning><terminology><optimizers>
Title: What do we mean by "infrequent features"?
UserDisplayName: ava_punksmash
ThreadText: # What do we mean by "infrequent features"?
  
  (Asked by: username_0 on 2020-10-23)
  
  I am reading this blog post: <https://ruder.io/optimizing-gradient-descent/index.html>. In the section about AdaGrad, it says:
  
  > It adapts the learning rate to the parameters, performing smaller updates (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.
  
  But I am not sure about the meaning of *infrequent features*: is it that the value of a given feature changes rarely?
  
  **Tags:** deep-learning, terminology, optimizers
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-10-23. Score: 2 | Accepted answer
  
  We will describe the input to the network as a vector, called features vector. Each component of this vector is usually related to some "real world" information, by example "age of the person", "number of atoms", "...".
  
  In very usual situations, a specific component of the input vector will have near than always the same value. This is more usual in binary components or components that has a small set of possible values.
  
  However, usually **in the cases where this component has a value different from the most usual one, this component is very important and informative**.
  
  These values of this kind of components are called *infrequent features*.
  
  (example: "rains?" is in my city 99.9% of time "false". However, when it is true, it is a key factor to all questions about the behavior of the population).
  
  The problem with these features: as unusual values are infrequent, the net has few chances to learn from them, and some learning algorithms could fail to give them the weight that they must (taken into account that, as has been said, these components are very important when they take a value different to the most frequent one).
  
  Some adaptive learning rate algorithms as AdaGrad tries to solve this issue.
  
  > **Comments:**
  > **username_0**: Thank you very much for your answer, it is very clear :)  
  > **username_0**: OK thank you for the advice ! Edit: for now my vote is counted but not taken into account because i am a new user, but i do appreciate your help  
  
  ---
  
  