Answers.Body: <p>To give some practical advice, it is important to understand parts of calculus. This is mainly <a href="https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b" rel="nofollow noreferrer">because Backpropagation is a leaky abstraction in modern libraries</a>. In a nutshell, there is a lot  which can go wrong (exploding or vanishing gradient for example) and you will need knowledge about gradient descent to handle it.</p>
  <p>I highly recommend <a href="https://www.youtube.com/watch?v=i94OvYb6noo" rel="nofollow noreferrer">Andrej Karpathys Lecture on it</a>. He gives a easy to understand and intuitive explanation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-04T18:31:53.203Z
Answers.LastActivityDate: 2021-09-04T18:31:53.203Z
Answers.UserDisplayName: tnfru
Answers.Body: <p>Neural networks are essentially just repeated matrix multiplications and applications of an <em>activation function</em>, so you really don't need a great deal of linear algebra to construct a simple neural network — if you understand how to multiply matrices, that's probably sufficient.</p>
  <p>The harder bit is the training process which is typically done through backpropagation. You need a bit of calculus, but differential geometry is overkill. There are some interesting topics in differential geometry for machine learning, but it's far beyond what is needed to implement backpropagation.</p>
  <p>To understand backpropagation you just need to know about the gradient of a function and what this means intuitively; you should also have a good knowledge of the chain rule. That's really all you need, and any course on &quot;multivariable calculus&quot; or something similar would give you more than enough to get started.</p>
  <p>Of course, it never hurts to know more, but fortunately neural networks are simple
  enough that you don't need to struggle for years before you can implement a basic neural network; try to get started as soon as you have the basics, and learn the rest as you go.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-06T09:21:01.960Z
Answers.LastActivityDate: 2021-08-06T09:21:01.960Z
Answers.UserDisplayName: htl
Body: <p>I've been studying <a href="https://en.wikipedia.org/wiki/Geometry" rel="nofollow noreferrer">geometry</a> and <a href="https://en.wikipedia.org/wiki/Linear_algebra" rel="nofollow noreferrer">linear algebra</a> for months with the goal to build neural networks. But now I'm reading that perceptrons require fitting curves, and curves are not expressed as linear functions. So, I might need to study <a href="https://en.wikipedia.org/wiki/Differential_geometry" rel="nofollow noreferrer">differential geometry</a> and <a href="https://en.wikipedia.org/wiki/Calculus" rel="nofollow noreferrer">calculus</a> for building good fitting curves in perceptrons.</p>
  <p>I already know how to code and was hoping to get my hands dirty by coding a few neural networks. But should I study calculus and differential geometry before coding?</p>
  <p>From <a href="https://www.youtube.com/watch?v=CkqGw15GlqA&amp;ab_channel=MatthewSalomone" rel="nofollow noreferrer">this video</a>, I understand that the least squares approximation can be used to fit a curve through a set of points, so maybe linear algebra is enough for building good neural networks?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T07:44:15.037Z
Comments.Text: What level are you hoping to build neural networks at? You only need a brief overview of calculus and statistics if you want to use a toolkit like Keras or PyTorch. If you want to understand all the details, and build everything yourself in C++ or similar, then your need more.
Comments.UserDisplayName: Neil Slater
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T08:01:59.510Z
Comments.Text: Building it from scratch. Not using any external library. I've already studied statistics and probability. Ok so I need Calculus.
Comments.UserDisplayName: user438454534
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-06T10:17:37.970Z
Comments.Text: As a side note, "perceptrons" and "neural networks" may not be the same thing. People usually use the term perceptron to refer to a very simple neural network that has no hidden layer. Maybe you meant the term "multi-layer perceptron" (MLP).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-07T13:48:20.123Z
Comments.Text: @nbro FYI: I asked [a new question](https://ai.stackexchange.com/q/30066/18652) based on your comment.
Comments.UserDisplayName: R.M.
Tags: <neural-networks><calculus><education>
Title: Are calculus and differential geometry required for building neural networks?
UserDisplayName: user438454534
ThreadText: # Are calculus and differential geometry required for building neural networks?
  
  (Asked by: username_0 on 2021-08-06)
  
  I've been studying [geometry](https://en.wikipedia.org/wiki/Geometry) and [linear algebra](https://en.wikipedia.org/wiki/Linear_algebra) for months with the goal to build neural networks. But now I'm reading that perceptrons require fitting curves, and curves are not expressed as linear functions. So, I might need to study [differential geometry](https://en.wikipedia.org/wiki/Differential_geometry) and [calculus](https://en.wikipedia.org/wiki/Calculus) for building good fitting curves in perceptrons.
  
  I already know how to code and was hoping to get my hands dirty by coding a few neural networks. But should I study calculus and differential geometry before coding?
  
  From [this video](https://www.youtube.com/watch?v=CkqGw15GlqA&ab_channel=MatthewSalomone), I understand that the least squares approximation can be used to fit a curve through a set of points, so maybe linear algebra is enough for building good neural networks?
  
  **Tags:** neural-networks, calculus, education
  > **Comments:**
  > **username_1**: What level are you hoping to build neural networks at? You only need a brief overview of calculus and statistics if you want to use a toolkit like Keras or PyTorch. If you want to understand all the details, and build everything yourself in C++ or similar, then your need more.  
  > **username_0**: Building it from scratch. Not using any external library. I've already studied statistics and probability. Ok so I need Calculus.  
  > **username_2**: As a side note, "perceptrons" and "neural networks" may not be the same thing. People usually use the term perceptron to refer to a very simple neural network that has no hidden layer. Maybe you meant the term "multi-layer perceptron" (MLP).  
  > **username_3**: @nbro FYI: I asked [a new question](https://ai.stackexchange.com/q/30066/18652) based on your comment.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_5 on 2021-08-06. Score: 13 | Accepted answer
  
  Neural networks are essentially just repeated matrix multiplications and applications of an *activation function*, so you really don't need a great deal of linear algebra to construct a simple neural network — if you understand how to multiply matrices, that's probably sufficient.
  
  The harder bit is the training process which is typically done through backpropagation. You need a bit of calculus, but differential geometry is overkill. There are some interesting topics in differential geometry for machine learning, but it's far beyond what is needed to implement backpropagation.
  
  To understand backpropagation you just need to know about the gradient of a function and what this means intuitively; you should also have a good knowledge of the chain rule. That's really all you need, and any course on "multivariable calculus" or something similar would give you more than enough to get started.
  
  Of course, it never hurts to know more, but fortunately neural networks are simple
  enough that you don't need to struggle for years before you can implement a basic neural network; try to get started as soon as you have the basics, and learn the rest as you go.
  
  
  ---
  
  ### Answer by username_4 on 2021-09-04. Score: 1
  
  To give some practical advice, it is important to understand parts of calculus. This is mainly [because Backpropagation is a leaky abstraction in modern libraries](https://karpathy.medium.com/yes-you-should-understand-backprop-e2f06eab496b). In a nutshell, there is a lot which can go wrong (exploding or vanishing gradient for example) and you will need knowledge about gradient descent to handle it.
  
  I highly recommend [Andrej Karpathys Lecture on it](https://www.youtube.com/watch?v=i94OvYb6noo). He gives a easy to understand and intuitive explanation.
  
  
  ---
  
  