Answers.Body: <p>I found a somewhat more accessible introduction here: </p>
  
  <p><a href="https://medium.com/konvergen/an-introduction-to-adagrad-f130ae871827" rel="nofollow noreferrer">https://medium.com/konvergen/an-introduction-to-adagrad-f130ae871827</a></p>
  
  <p>Let me start from the last part of your question. The frequency of a parameter is in G_t, which is the accumulated sum of squared gradients from all the time steps up to step t. If the gradient vanishes in many of the previous steps, then you divide the learning rate with a smaller number for that parameter. </p>
  
  <p>And for the first part, you want the parameter that is more frequent to have a smaller learning rate as it is updated on more iterations compared to a parameter which is updated only a small number of times. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-20T12:08:59.427Z
Answers.Text: For the first part: Maybe its just the fact that SGD doesnt update the neurons associated with the infrequent features much because they provide relatively small improvement (low gradient) and thus that must be compensated for with a higher learning rate? So basically, how does the frequency of feature influence frequency of adjustment (and not only the gradient and thus adjustment itself) if adjustments don't occur after every example, but after a mini-batch of examples where an example with the infrequent feature (for the sake of simplicity) has occured?
Answers.UserDisplayName: DaddyMike
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-20T13:23:53.143Z
Answers.Text: A good point for which I have no answer. I could not find anything about this in the original paper by Duchi et al, or the deep learning book of Goodfellow. I suppose the parameters have to be sparse enough (compared to mini batch size) for the algorithm to work effectively. – serali 9 mins ago    Delete
Answers.UserDisplayName: serali
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-19T16:50:55.043Z
Answers.LastActivityDate: 2019-08-20T13:23:00.393Z
Answers.LastEditDate: 2019-08-20T13:23:00.393Z
Answers.UserDisplayName: serali
Body: <blockquote>
    <p>It (Adagrad) adapts the learning rate to the parameters, performing smaller updates
    (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.</p>
  </blockquote>
  
  <p>From <a href="http://ruder.io/optimizing-gradient-descent/index.html#adagrad" rel="noreferrer">Sebastian Ruder's Blog</a></p>
  
  <p>If a parameter is associated with an infrequent feature then yes, it is more important to focus on properly adjusting that parameter since it is more decisive in classification problems. But how does making the learning rate higher in this situation help?</p>
  
  <p>If it only changes the size of the movement in the dimension of the parameter (makes it larger) wouldn't that make things even more imprecise? Since the network depends more on those infrequent features, shouldn't adjusting those parameters be done more precisely instead of just faster? The more decisive parameters should have a higher "slope", thus why should they also have high learning rates? I must be missing something, but what is it?</p>
  
  <p>Further, in the article, the formula for parameter adjustments with Adagrad is given. Where exactly in that formula do you find the information about the frequency of a parameter? There must be a relationship between the gradients of a parameter and the frequency of features associated with it because it's the gradients that play an important role in the formula. What is that relationship?</p>
  
  <p><strong>TLDR:</strong> I don't understand both the purpose and formula behind Adagrad. What is an intuitive explanation of it that also provides an answer to the questions above, or shows why they are irrelevant?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-19T03:17:59.637Z
Comments.Text: I'm fairly sure lecture 3 of cs231n on youtube provides a good explanation at some point. I'm just not sure if this is the specific lecture or not.
Comments.UserDisplayName: Recessive
Tags: <machine-learning><gradient-descent><hyperparameter-optimization>
Title: An intuitive explanation of Adagrad, its purpose and its formula
UserDisplayName: DaddyMike
ThreadText: # An intuitive explanation of Adagrad, its purpose and its formula
  
  (Asked by: username_0 on 2019-08-16)
  
  > It (Adagrad) adapts the learning rate to the parameters, performing smaller updates
  > (i.e. low learning rates) for parameters associated with frequently occurring features, and larger updates (i.e. high learning rates) for parameters associated with infrequent features.
  
  From [Sebastian Ruder's Blog](http://ruder.io/optimizing-gradient-descent/index.html#adagrad)
  
  If a parameter is associated with an infrequent feature then yes, it is more important to focus on properly adjusting that parameter since it is more decisive in classification problems. But how does making the learning rate higher in this situation help?
  
  If it only changes the size of the movement in the dimension of the parameter (makes it larger) wouldn't that make things even more imprecise? Since the network depends more on those infrequent features, shouldn't adjusting those parameters be done more precisely instead of just faster? The more decisive parameters should have a higher "slope", thus why should they also have high learning rates? I must be missing something, but what is it?
  
  Further, in the article, the formula for parameter adjustments with Adagrad is given. Where exactly in that formula do you find the information about the frequency of a parameter? There must be a relationship between the gradients of a parameter and the frequency of features associated with it because it's the gradients that play an important role in the formula. What is that relationship?
  
  **TLDR:** I don't understand both the purpose and formula behind Adagrad. What is an intuitive explanation of it that also provides an answer to the questions above, or shows why they are irrelevant?
  
  **Tags:** machine-learning, gradient-descent, hyperparameter-optimization
  
  > **Comments:**
  > **username_1**: I'm fairly sure lecture 3 of cs231n on youtube provides a good explanation at some point. I'm just not sure if this is the specific lecture or not.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-08-19. Score: 1
  
  I found a somewhat more accessible introduction here:
  
  <https://medium.com/konvergen/an-introduction-to-adagrad-f130ae871827>
  
  Let me start from the last part of your question. The frequency of a parameter is in G\_t, which is the accumulated sum of squared gradients from all the time steps up to step t. If the gradient vanishes in many of the previous steps, then you divide the learning rate with a smaller number for that parameter.
  
  And for the first part, you want the parameter that is more frequent to have a smaller learning rate as it is updated on more iterations compared to a parameter which is updated only a small number of times.
  
  > **Comments:**
  > **username_0**: For the first part: Maybe its just the fact that SGD doesnt update the neurons associated with the infrequent features much because they provide relatively small improvement (low gradient) and thus that must be compensated for with a higher learning rate? So basically, how does the frequency of feature influence frequency of adjustment (and not only the gradient and thus adjustment itself) if adjustments don't occur after every example, but after a mini-batch of examples where an example with the infrequent feature (for the sake of simplicity) has occured?  
  > **username_2**: A good point for which I have no answer. I could not find anything about this in the original paper by Duchi et al, or the deep learning book of Goodfellow. I suppose the parameters have to be sparse enough (compared to mini batch size) for the algorithm to work effectively. – username_2 9 mins ago    Delete  
  
  ---
  
  