title: Thermodynamic Natural Gradient Descent
summary: Second-order training methods have better convergence properties than
  gradient descent but are rarely used in practice for large-scale training due
  to their computational overhead. This can be viewed as a hardware limitation
  (imposed by digital computers). Here we show that natural gradient descent
  (NGD), a second-order method, can have a similar computational complexity per
  iteration to a first-order method, when employing appropriate hardware. We
  present a new hybrid digital-analog algorithm for training neural networks that
  is equivalent to NGD in a certain parameter regime but avoids prohibitively
  costly linear system solves. Our algorithm exploits the thermodynamic
  properties of an analog system at equilibrium, and hence requires an analog
  thermodynamic computer. The training occurs in a hybrid digital-analog loop,
  where the gradient and Fisher information matrix (or any other positive
  semi-definite curvature matrix) are calculated at given time intervals while
  the analog dynamics take place. We numerically demonstrate the superiority of
  this approach over state-of-the-art digital first- and second-order training
  methods on classification tasks and language model fine-tuning tasks.
paper_authors._id: 66501d233fa1a30553ee2c03
paper_authors.name: Kaelan Donatella
paper_authors.status: extracted_confirmed
paper_authors.statusLastChangedAt: 2024-05-24T12:17:20.890Z
paper_authors._id: 665084f127935a4a528f53f8
paper_authors.avatarUrl: /avatars/fb05a807a1f5f35c143082adc39dae12.svg
paper_authors.fullname: Kaelan Donatella
paper_authors.type: user
paper_authors.user: KaelanDt
paper_authors._id: 66501d233fa1a30553ee2c04
paper_authors.name: Samuel Duffield
paper_authors._id: 66501d233fa1a30553ee2c05
paper_authors.name: Maxwell Aifer
paper_authors._id: 66501d233fa1a30553ee2c06
paper_authors.name: Denis Melanson
paper_authors._id: 66501d233fa1a30553ee2c07
paper_authors.name: Gavin Crooks
paper_authors._id: 66501d233fa1a30553ee2c08
paper_authors.name: Patrick J. Coles
paper_title: Thermodynamic Natural Gradient Descent
paper_summary: Second-order training methods have better convergence properties than
  gradient descent but are rarely used in practice for large-scale training due
  to their computational overhead. This can be viewed as a hardware limitation
  (imposed by digital computers). Here we show that natural gradient descent
  (NGD), a second-order method, can have a similar computational complexity per
  iteration to a first-order method, when employing appropriate hardware. We
  present a new hybrid digital-analog algorithm for training neural networks that
  is equivalent to NGD in a certain parameter regime but avoids prohibitively
  costly linear system solves. Our algorithm exploits the thermodynamic
  properties of an analog system at equilibrium, and hence requires an analog
  thermodynamic computer. The training occurs in a hybrid digital-analog loop,
  where the gradient and Fisher information matrix (or any other positive
  semi-definite curvature matrix) are calculated at given time intervals while
  the analog dynamics take place. We numerically demonstrate the superiority of
  this approach over state-of-the-art digital first- and second-order training
  methods on classification tasks and language model fine-tuning tasks.
paper_ai_summary: A new hybrid digital-analog algorithm efficiently implements natural gradient descent, achieving superior performance in training neural networks compared to existing digital methods.
paper_ai_keywords: natural gradient descent NGD Fisher information matrix curvature matrix thermodynamic properties analog thermodynamic computer second-order training methods gradient descent hybrid digital-analog loop