Answers.Body: <p>Gradient descent presumes a Taylor Series.  They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss.</p>
  <p>The learner as a universal function approximator means there can be many configurations that yield minimum loss, and there is usually no global &quot;best&quot;.</p>
  <p>One of the reasons for multiple traverses through the data using the optimizer is that the local Taylor series estimate is low order (frequently only first order), so the optimum is not apparent after a single pass.  When the weights change, the landscape changes and for a 25-million-parameter network, that change is happening in a 25-million dimensional space.</p>
  <p>The higher order terms of the gradient can accelerate the optimization, but they can be prohibitively expensive to compute or estimate.  The Hessian is nice, but has high overhead to compute and store.  Things like conjugate gradient were popular because they are fast and (very) rough estimates of the Hessian.</p>
  <p>If you have a cheap way to get decent higher-order derivatives, then there is value in that.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T09:06:07.480Z
Answers.Text: Conjugate gradient does not use an explicit Hessian, it is used for linear systems with symmetric system matrix, it only uses the evaluation procedure for the matrix-vector products, so can be used for large sparse systems where the matrix is never constructed as separate data structure. // Hessian approximations are constructed in quasi-Newton methods like BFGS. Often implemented in a way to avoid the full matrix, only storing the initial diagonal and the data for the rank-1-updates.
Answers.UserDisplayName: Lutz Lehmann
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-29T16:31:55.257Z
Answers.Text: @LutzLehmann - good points.  I was meaning approximation in the loose and quick sense, so less bad than not having anything at all, and nothing like to within an error bound.  If you have 25 million weights, then the hessian is a matrix of size (25M)^2.
Answers.UserDisplayName: EngrStudent
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-11-28T15:07:31.030Z
Answers.LastActivityDate: 2022-11-29T16:36:20.367Z
Answers.LastEditDate: 2022-11-29T16:36:20.367Z
Answers.UserDisplayName: EngrStudent
Body: <p>Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks.</p>
  <ol>
  <li><p><strong>First order gradient</strong>: It quantifies the rate of change of a function with respect to its inputs. It is useful in artificial intelligence, especially in gradient-based algorithms, to know about the <strong>direction in which the parameters need to be updated</strong>.</p>
  </li>
  <li><p><strong>Second-order gradient</strong>: It somehow quantifies the curvature of the function. It is used in artificial intelligence, <strong>to know whether the function has convex or concave portions</strong>.</p>
  </li>
  </ol>
  <p>In this context, I want to learn whether there is any significance for higher-order gradients in artificial intelligence? Note that higher-order refers to the order <span class="math-container">$\ge 3$</span>.</p>
  
Tags: <reference-request><objective-functions><gradient-descent><gradient>
Title: Is there any significance for higher order gradients in artificial intelligence?
UserDisplayName: hanugm
ThreadText: # Is there any significance for higher order gradients in artificial intelligence?
  
  (Asked by: username_0 on 2021-08-18)
  
  Although I don't know in detail, I am aware of the following facts regarding the use of gradients in some domains of artificial intelligence, especially in minimizing the training of neural networks.
  
  1. **First order gradient**: It quantifies the rate of change of a function with respect to its inputs. It is useful in artificial intelligence, especially in gradient-based algorithms, to know about the **direction in which the parameters need to be updated**.
  2. **Second-order gradient**: It somehow quantifies the curvature of the function. It is used in artificial intelligence, **to know whether the function has convex or concave portions**.
  
  In this context, I want to learn whether there is any significance for higher-order gradients in artificial intelligence? Note that higher-order refers to the order $\ge 3$.
  
  **Tags:** reference-request, objective-functions, gradient-descent, gradient
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-11-28. Score: 1
  
  Gradient descent presumes a Taylor Series. They estimate the loss given the inputs and target, then use the difference to move the system weights to produce a less-bad loss.
  
  The learner as a universal function approximator means there can be many configurations that yield minimum loss, and there is usually no global "best".
  
  One of the reasons for multiple traverses through the data using the optimizer is that the local Taylor series estimate is low order (frequently only first order), so the optimum is not apparent after a single pass. When the weights change, the landscape changes and for a 25-million-parameter network, that change is happening in a 25-million dimensional space.
  
  The higher order terms of the gradient can accelerate the optimization, but they can be prohibitively expensive to compute or estimate. The Hessian is nice, but has high overhead to compute and store. Things like conjugate gradient were popular because they are fast and (very) rough estimates of the Hessian.
  
  If you have a cheap way to get decent higher-order derivatives, then there is value in that.
  
  > **Comments:**
  > **username_2**: Conjugate gradient does not use an explicit Hessian, it is used for linear systems with symmetric system matrix, it only uses the evaluation procedure for the matrix-vector products, so can be used for large sparse systems where the matrix is never constructed as separate data structure. // Hessian approximations are constructed in quasi-Newton methods like BFGS. Often implemented in a way to avoid the full matrix, only storing the initial diagonal and the data for the rank-1-updates.  
  > **username_1**: @LutzLehmann - good points.  I was meaning approximation in the loose and quick sense, so less bad than not having anything at all, and nothing like to within an error bound.  If you have 25 million weights, then the hessian is a matrix of size (25M)^2.  
  
  ---
  
  