Answers.Body: <p>I guess a first distinction should be made between deep learning as a whole or deep learning as architecture.
  I think the paragraph you quote refers to <strong>solving systems of linear equations</strong> as a simple operation involved in deep learning generically. And this is definitely the case, when training a deep model we're always solving systems of linear equations, think about the way weights and biases are applied to an input:</p>
  <p><span class="math-container">$W*x + b$</span></p>
  <p>this is in itself a system of linear equations.
  Then, moving away from training and model architectures, in deep learning there is still a massive use of dimensionality reduction techniques involved in pre or post processing of the data like SVD or PCA that also consists in solving systems of linear equations (an we could add any matrix factorization technique, relevant especially in the early methods for word embedding training before the advent of transformers).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-23T08:56:29.133Z
Answers.LastActivityDate: 2021-08-23T08:56:29.133Z
Answers.UserDisplayName: Edoardo Guerriero
Answers.Body: <p>Well the paragraph you quote says &quot;Machine learning algorithms ...&quot;, so it's not restricted to deep learning.</p>
  <p>When considering Machine Learning in general there is the case of Gaussian Process Regression where to fit the models a matrix needs to be inverted (which is equivalent to  solving a  linear system of equations), check equations 2.7 and 2.8 of <a href="https://gaussianprocess.org/gpml/chapters/RW2.pdf" rel="nofollow noreferrer">chapter 2 of the standard GPR book</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-25T14:03:10.840Z
Answers.LastActivityDate: 2022-10-25T14:03:10.840Z
Answers.UserDisplayName: Ken Grimes
Body: <p>Consider the following from <a href="https://www.deeplearningbook.org/contents/numerical.html" rel="nofollow noreferrer">Numerical Computation</a> chapter of Deep Learning book</p>
  <blockquote>
  <p>Machine learning algorithms usually require a high amount of numerical computation. This typically refers to algorithms that solve
  mathematical problems by methods that update estimates of the solution
  via an iterative process, rather than analytically deriving a formula
  to provide a symbolic expression for the correct solution. <strong>Common
  operations include optimization (ﬁnding the value of an argument that
  minimizes or maximizes a function) and solving systems of linear
  equations.</strong> Even just evaluating a mathematical function on a digital
  computer can be diﬃcult when the function involves real numbers, which
  cannot be represented precisely using a ﬁnite amount of memory.</p>
  </blockquote>
  <p>The paragraph clearly mentions that solving system of linear equations is a common operation in machine learning. I just know that solving system of linear equations is useful in reinforcement learning and some basic algorithms of machine learning including regression.</p>
  <p>Is solving system of linear equations useful anywhere in deep learning?</p>
  <p>I think that we use them nowhere since optimization is the only algorithm generally used in deep learning.</p>
  
Tags: <deep-learning><linear-algebra>
Title: Do solving system of linear equations required anywhere in contemporarty deep learning?
UserDisplayName: hanugm
ThreadText: # Do solving system of linear equations required anywhere in contemporarty deep learning?
  
  (Asked by: username_0 on 2021-08-23)
  
  Consider the following from [Numerical Computation](https://www.deeplearningbook.org/contents/numerical.html) chapter of Deep Learning book
  
  > Machine learning algorithms usually require a high amount of numerical computation. This typically refers to algorithms that solve
  > mathematical problems by methods that update estimates of the solution
  > via an iterative process, rather than analytically deriving a formula
  > to provide a symbolic expression for the correct solution. **Common
  > operations include optimization (ﬁnding the value of an argument that
  > minimizes or maximizes a function) and solving systems of linear
  > equations.** Even just evaluating a mathematical function on a digital
  > computer can be diﬃcult when the function involves real numbers, which
  > cannot be represented precisely using a ﬁnite amount of memory.
  
  The paragraph clearly mentions that solving system of linear equations is a common operation in machine learning. I just know that solving system of linear equations is useful in reinforcement learning and some basic algorithms of machine learning including regression.
  
  Is solving system of linear equations useful anywhere in deep learning?
  
  I think that we use them nowhere since optimization is the only algorithm generally used in deep learning.
  
  **Tags:** deep-learning, linear-algebra
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-23. Score: 1
  
  I guess a first distinction should be made between deep learning as a whole or deep learning as architecture.
  I think the paragraph you quote refers to **solving systems of linear equations** as a simple operation involved in deep learning generically. And this is definitely the case, when training a deep model we're always solving systems of linear equations, think about the way weights and biases are applied to an input:
  
  $W\*x + b$
  
  this is in itself a system of linear equations.
  Then, moving away from training and model architectures, in deep learning there is still a massive use of dimensionality reduction techniques involved in pre or post processing of the data like SVD or PCA that also consists in solving systems of linear equations (an we could add any matrix factorization technique, relevant especially in the early methods for word embedding training before the advent of transformers).
  
  
  ---
  
  ### Answer by username_2 on 2022-10-25. Score: 0
  
  Well the paragraph you quote says "Machine learning algorithms ...", so it's not restricted to deep learning.
  
  When considering Machine Learning in general there is the case of Gaussian Process Regression where to fit the models a matrix needs to be inverted (which is equivalent to solving a linear system of equations), check equations 2.7 and 2.8 of [chapter 2 of the standard GPR book](https://gaussianprocess.org/gpml/chapters/RW2.pdf)
  
  
  ---
  
  