Body: <p>This topic has been introduced in &quot;<em>Pattern Recognition and Machine Learning, Bishop, 2006</em>&quot;, section 5.4.1. I am a bit confused about this method and I have two questions.</p>
  <ol>
  <li><p>Why this method has attracted attention or has been developed?
  First, we want to compute the Hessian fast, so we try to approximate it in O(W) time, where W is the number of parameters.
  And then, we see that this matrix most of the time is heavily non-diagonal.</p>
  </li>
  <li><p>My second question is how can we know whether we can approximate a Hessian or not? Is there a hint/clue in problems?</p>
  </li>
  </ol>
  <p>Thanks in advance!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-19T13:41:31.100Z
Comments.Text: Hello. Please, provide more context about this "Hessian Diagonal Approximation". Provide a link to or name of the paper/book/etc that describes this. Moreover, please, to improve the clarity of your post, put your **specific** question in the title.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-19T13:52:09.990Z
Comments.Text: @nbro Thanks for the guidance. I did my best. I hope it is clear enough now.
Comments.UserDisplayName: Hossein Aghamohammadi
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-19T22:56:53.980Z
Comments.Text: Ok, thanks. I think your question is a duplicate of [this](https://ai.stackexchange.com/q/25102/2444). Let me know if I am correct.
Comments.UserDisplayName: nbro
Tags: <neural-networks><machine-learning><optimization>
Title: Why is there a Hessian diagonal approximation? And when can we use it?
UserDisplayName: Hossein Aghamohammadi
ThreadText: # Why is there a Hessian diagonal approximation? And when can we use it?
  
  (Asked by: username_0 on 2021-12-19)
  
  This topic has been introduced in "*Pattern Recognition and Machine Learning, Bishop, 2006*", section 5.4.1. I am a bit confused about this method and I have two questions.
  
  1. Why this method has attracted attention or has been developed?
     First, we want to compute the Hessian fast, so we try to approximate it in O(W) time, where W is the number of parameters.
     And then, we see that this matrix most of the time is heavily non-diagonal.
  2. My second question is how can we know whether we can approximate a Hessian or not? Is there a hint/clue in problems?
  
  Thanks in advance!
  
  **Tags:** neural-networks, machine-learning, optimization
  > **Comments:**
  > **username_1**: Hello. Please, provide more context about this "Hessian Diagonal Approximation". Provide a link to or name of the paper/book/etc that describes this. Moreover, please, to improve the clarity of your post, put your **specific** question in the title.  
  > **username_0**: @nbro Thanks for the guidance. I did my best. I hope it is clear enough now.  
  > **username_1**: Ok, thanks. I think your question is a duplicate of [this](https://ai.stackexchange.com/q/25102/2444). Let me know if I am correct.  
  
  ---
  
  ## 0 answers
  
  