Answers.Body: <p>I think, once you are covered with the common stuff, you can probably go on and study all kinds of neural network variants. </p>
  
  <hr>
  
  <p>The common stuff:</p>
  
  <p>a) An undergraduate level Linear Algebra course -- covering matrix calculus. You might find <a href="https://stats.stackexchange.com/questions/21346/reference-book-for-linear-algebra-applied-to-statistics">this</a> useful.</p>
  
  <p>b) An undergraduate level study in statistical inference. Concepts from this topic will come up most of the time and you might have hard time getting around even though you understand the rest of the math. I would recommend <a href="https://fsalamri.files.wordpress.com/2015/02/casella_berger_statistical_inference1.pdf" rel="nofollow noreferrer">this</a>.</p>
  
  <p>c) A starter book on neural networks. Ex- <a href="http://www.inf.fu-berlin.de/inst/ag-ki/rojas_home/documents/1996/NeuralNetworks/neuron.pdf" rel="nofollow noreferrer">Neural networks</a> by Raul Rojas.</p>
  
  <hr>
  
  <p>After all these are covered you will certainly be ready for learning the variants of neural networks with ease. For LSTM I would recommend <a href="https://www.cs.toronto.edu/~graves/preprint.pdf" rel="nofollow noreferrer">Alex Graves</a>. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-10-05T12:36:49.023Z
Answers.LastActivityDate: 2019-10-05T12:36:49.023Z
Answers.UserDisplayName: naive
Answers.Body: <p>Would personally recommend deeplearning.ai's <a href="https://www.coursera.org/specializations/deep-learning" rel="nofollow noreferrer">course</a> to begin with. There may be more comprehensive or better MOOC's for covering basic MLP's, CNN, RNN's, tuning and training of neural networks but this is probably the most common one and the one that I can personally vouch for.</p>
  
  <p>After this I'd recommend you get a physical or pdf copy of <a href="http://www.deeplearningbook.org/" rel="nofollow noreferrer">Deep Learning by Goodfellow et al.</a> and use it as reference material for any new idea you'd want to learn. Personally would not recommend reading the whole book and its better as a reference material as it is quite comprehensive. </p>
  
  <p>This should essentially be able to give you enough knowledge to be able to cover almost any paper/material on deep learning. The Course mentioned (most courses) would cover LSTM's as they are quite an old idea (~1997 I think) and GAN's are well covered in the book mentioned (The author invented them) since they are more of a recent advancement (2014). </p>
  
  <p>Hope this was helpful! </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-11-06T12:06:55.977Z
Answers.LastActivityDate: 2019-11-06T12:06:55.977Z
Answers.UserDisplayName: ashenoy
Body: <p>I'm interested in learning about Neural Networks and implementing them. I'm particularly interested in GANs and LSTM networks.</p>
  
  <p>I understand perceptrons and basic Neural Network configuration (sigmoid activation, weights, hidden layers etc). But what topics do I need to learn in order, to get to the point where I can implement GAN or LSTM.</p>
  
  <p>I intend to make an implementation of each in C++ to prove to myself that I understand. I haven't got a particularly good math background, but I understand most math-things when they are explained.</p>
  
  <p>For example, I understand backpropagation, but I don't really understand it. I understand how reinforced learning is used with backpropagation, but not fully how you can have things like training without datasets (like tD-backgammon). I don't quite understand CNNs, especially why you might make a particular architecture.</p>
  
  <p>If for each "topic" there was a book or website or something for each it would be great.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T03:14:04.487Z
Comments.Text: For CNNs, use the cs231n stanford lectures on youtube - they're free. They teach back propagation very well in those lectures as well. As a recommendation, maybe try making your models in python, it will be a lot faster then c++, and the benefit of speed that c++ gives doesn't make sense in the context, as it will be very hard to implement a model as efficiently as Tensorflow is in c++. For an LSTM, I found information from this tutorial nice and concise https://medium.com/@aidangomez/let-s-do-this-f9b699de31d9, although the structure is drawn a bit differently.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T03:18:26.890Z
Comments.Text: Many people will try to draw the full back prop as a series of equations, like this: https://miro.medium.com/max/1920/1*PBu6tUmPMxBqxEpZHujfPA.jpeg (you need to copy paste the full link) but please keep in mind this is over-complicating a rather simple algorithm once you come to understand it. If you get the chain rule, and the basic multi-variable calculus rule that f'(a,b) = f'(a) + f'(b) at points where gradients meet, you can do it entirely visually with no need for huge long strings of equations, which is how it's *mostly* done in the example I gave earlier.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T03:21:02.640Z
Comments.Text: After understanding all of this for any model, you'll probably have the hardest time just finding the correct input and output dimensions of an architecture. A lot of articles either: don't understand the dimensions, or assume you already know them. CNNs are especially tricky, and I ended up programming one missing an entire dimension for filter depth (it still works fine, just learns a bit slower). It may take a while to learn any or all of this, but from experience it is very satisfying to finally see a model work and well worth the time.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T03:35:18.357Z
Comments.Text: Hey thanks @Recessive. Is there a progression that you can think of that makes sense? I'm working through http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent but was hoping to use more than one source
Comments.UserDisplayName: NeomerArcana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T03:41:09.337Z
Comments.Text: Try to get to a position where you have learnt how to code from scratch a standard perceptron neural network. That tutorial looks good, although the back prop equations look a bit confusing. You only really need to know one for a sigmoid perceptron, and thats: https://wikimedia.org/api/rest_v1/media/math/render/svg/991c8f020800ec1da130849e20a3a415613e9bdb where that represents the gradient at the end of each node going backwards (aka the delta) (from https://en.wikipedia.org/wiki/Backpropagation, keep in mind, sigmoid = logistic function).
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T03:43:33.470Z
Comments.Text: If you are able to follow this: https://en.wikipedia.org/wiki/Backpropagation#Derivation_for_a_single-layered_network you will completely understand a perceptron. Also, just as an overview of what a perceptron is doing (how it's doing it isn't explained well enough that you can do it on paper though), watch 3blue1brown's youtube series: https://www.youtube.com/watch?v=aircAruvnKk . So there's a few other sources that you should be able to work with.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T04:30:32.350Z
Comments.Text: I believe I've got a handle on a perceptron and how they're combined into a network. Working through back-prop now, but I think I still need to understand the prediction vs categorisation applications for back-prop (even for something simple like sigmoid). Is there may a single book I can read?
Comments.UserDisplayName: NeomerArcana
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-10-02T04:39:37.813Z
Comments.Text: Yep, there's a book here you can read: https://www.reddit.com/r/MachineLearning/comments/5z8110/d_a_super_harsh_guide_to_machine_learning/
Comments.UserDisplayName: Recessive
Tags: <neural-networks>
Title: What order should I learn about Neural Networks?
UserDisplayName: NeomerArcana
ThreadText: # What order should I learn about Neural Networks?
  
  (Asked by: username_0 on 2019-10-01)
  
  I'm interested in learning about Neural Networks and implementing them. I'm particularly interested in GANs and LSTM networks.
  
  I understand perceptrons and basic Neural Network configuration (sigmoid activation, weights, hidden layers etc). But what topics do I need to learn in order, to get to the point where I can implement GAN or LSTM.
  
  I intend to make an implementation of each in C++ to prove to myself that I understand. I haven't got a particularly good math background, but I understand most math-things when they are explained.
  
  For example, I understand backpropagation, but I don't really understand it. I understand how reinforced learning is used with backpropagation, but not fully how you can have things like training without datasets (like tD-backgammon). I don't quite understand CNNs, especially why you might make a particular architecture.
  
  If for each "topic" there was a book or website or something for each it would be great.
  
  **Tags:** neural-networks
  
  > **Comments:**
  > **username_1**: For CNNs, use the cs231n stanford lectures on youtube - they're free. They teach back propagation very well in those lectures as well. As a recommendation, maybe try making your models in python, it will be a lot faster then c++, and the benefit of speed that c++ gives doesn't make sense in the context, as it will be very hard to implement a model as efficiently as Tensorflow is in c++. For an LSTM, I found information from this tutorial nice and concise https://medium.com/@aidangomez/let-s-do-this-f9b699de31d9, although the structure is drawn a bit differently.  
  > **username_1**: Many people will try to draw the full back prop as a series of equations, like this: https://miro.medium.com/max/1920/1*PBu6tUmPMxBqxEpZHujfPA.jpeg (you need to copy paste the full link) but please keep in mind this is over-complicating a rather simple algorithm once you come to understand it. If you get the chain rule, and the basic multi-variable calculus rule that f'(a,b) = f'(a) + f'(b) at points where gradients meet, you can do it entirely visually with no need for huge long strings of equations, which is how it's *mostly* done in the example I gave earlier.  
  > **username_1**: After understanding all of this for any model, you'll probably have the hardest time just finding the correct input and output dimensions of an architecture. A lot of articles either: don't understand the dimensions, or assume you already know them. CNNs are especially tricky, and I ended up programming one missing an entire dimension for filter depth (it still works fine, just learns a bit slower). It may take a while to learn any or all of this, but from experience it is very satisfying to finally see a model work and well worth the time.  
  > **username_0**: Hey thanks @username_1. Is there a progression that you can think of that makes sense? I'm working through http://neuralnetworksanddeeplearning.com/chap1.html#learning_with_gradient_descent but was hoping to use more than one source  
  > **username_1**: Try to get to a position where you have learnt how to code from scratch a standard perceptron neural network. That tutorial looks good, although the back prop equations look a bit confusing. You only really need to know one for a sigmoid perceptron, and thats: https://wikimedia.org/api/rest_v1/media/math/render/svg/991c8f020800ec1da130849e20a3a415613e9bdb where that represents the gradient at the end of each node going backwards (aka the delta) (from https://en.wikipedia.org/wiki/Backpropagation, keep in mind, sigmoid = logistic function).  
  > **username_1**: If you are able to follow this: https://en.wikipedia.org/wiki/Backpropagation#Derivation_for_a_single-layered_network you will completely understand a perceptron. Also, just as an overview of what a perceptron is doing (how it's doing it isn't explained well enough that you can do it on paper though), watch 3blue1brown's youtube series: https://www.youtube.com/watch?v=aircAruvnKk . So there's a few other sources that you should be able to work with.  
  > **username_0**: I believe I've got a handle on a perceptron and how they're combined into a network. Working through back-prop now, but I think I still need to understand the prediction vs categorisation applications for back-prop (even for something simple like sigmoid). Is there may a single book I can read?  
  > **username_1**: Yep, there's a book here you can read: https://www.reddit.com/r/MachineLearning/comments/5z8110/d_a_super_harsh_guide_to_machine_learning/  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2019-10-05. Score: 1
  
  I think, once you are covered with the common stuff, you can probably go on and study all kinds of neural network variants.
  
  ---
  
  The common stuff:
  
  a) An undergraduate level Linear Algebra course -- covering matrix calculus. You might find [this](https://stats.stackexchange.com/questions/21346/reference-book-for-linear-algebra-applied-to-statistics) useful.
  
  b) An undergraduate level study in statistical inference. Concepts from this topic will come up most of the time and you might have hard time getting around even though you understand the rest of the math. I would recommend [this](https://fsalamri.files.wordpress.com/2015/02/casella_berger_statistical_inference1.pdf).
  
  c) A starter book on neural networks. Ex- [Neural networks](http://www.inf.fu-berlin.de/inst/ag-ki/rojas_home/documents/1996/NeuralNetworks/neuron.pdf) by Raul Rojas.
  
  ---
  
  After all these are covered you will certainly be ready for learning the variants of neural networks with ease. For LSTM I would recommend [Alex Graves](https://www.cs.toronto.edu/~graves/preprint.pdf).
  
  
  ---
  
  ### Answer by username_3 on 2019-11-06. Score: 0
  
  Would personally recommend deeplearning.ai's [course](https://www.coursera.org/specializations/deep-learning) to begin with. There may be more comprehensive or better MOOC's for covering basic MLP's, CNN, RNN's, tuning and training of neural networks but this is probably the most common one and the one that I can personally vouch for.
  
  After this I'd recommend you get a physical or pdf copy of [Deep Learning by Goodfellow et al.](http://www.deeplearningbook.org/) and use it as reference material for any new idea you'd want to learn. Personally would not recommend reading the whole book and its better as a reference material as it is quite comprehensive.
  
  This should essentially be able to give you enough knowledge to be able to cover almost any paper/material on deep learning. The Course mentioned (most courses) would cover LSTM's as they are quite an old idea (~1997 I think) and GAN's are well covered in the book mentioned (The author invented them) since they are more of a recent advancement (2014).
  
  Hope this was helpful!
  
  
  ---
  
  