Answers.Body: <blockquote>
    <p>I even don't know what exactly back-propagation is used for, if this would help.</p>
  </blockquote>
  
  <p>Programming backprop manually is one of the best exercises in machine learning. In fact, to do this you don't need a special package, like Tensorflor or Keras, just a general computing library like NumPy (and a whiteboard).</p>
  
  <p>Though I dare to say that VAE is not the best thing to implement, if you are not 100% familiar with backprop. It's pretty advanced technique and you risk to mix up fundamental (e.g. backprop) and specific ideas (e.g. deliberately crafted loss function) and slow down your understanding.</p>
  
  <p>I really think that writing a simple feed-forward neural net (including forward and backward passes) is a very good starting point, after which you can move on to <a href="https://jmetzen.github.io/2015-11-27/vae.html" rel="nofollow noreferrer">VAE details</a>, if you'd like.</p>
  
  <blockquote>
    <p>Any suggestions on where/how to start?</p>
  </blockquote>
  
  <p>If you feel confident with algebra and calculus, I highly recommend <a href="http://cs231n.github.io/" rel="nofollow noreferrer">this Stanford course</a>, especially its assignments. The course itself is about computer vision and convolutional neural networks, but the first assignments are very general and will give a lot of insight of what and how neural networks are doing.</p>
  
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-09-30T19:48:23.277Z
Answers.LastActivityDate: 2017-10-03T06:36:55.603Z
Answers.LastEditDate: 2017-10-03T06:36:55.603Z
Answers.UserDisplayName: Maxim
Body: <p>As a researcher, I am getting interested in deep learning (as everyone else:)), and I decided to start with the variational eutoencoders, since I am more interested in unsupervised than supervised learning. I have already read tutorials on the general idea of the mechanism of VAE, and what people usually suggest is to use TensorFlow, Keros, etc. to implement them. However, I do not just want to <strong>use</strong> machine learning, but also want to <strong>develop novel algorithms</strong> on top of the existing ones, so I want to deeply understand the mathematical details. And in my experience, in order to understand them, reading is not enough, you need to implement. Any suggestions on where/how to start? I have a general machine learning and programming background, but no experience on deep learning. I even don't know what exactly back-propagation is used for, if this would help.</p>
  
Tags: <neural-networks>
Title: A good way to understand the mathematical details of variational autoencoders through implementation?
UserDisplayName: user5054
ThreadText: # A good way to understand the mathematical details of variational autoencoders through implementation?
  
  (Asked by: username_0 on 2017-09-27)
  
  As a researcher, I am getting interested in deep learning (as everyone else:)), and I decided to start with the variational eutoencoders, since I am more interested in unsupervised than supervised learning. I have already read tutorials on the general idea of the mechanism of VAE, and what people usually suggest is to use TensorFlow, Keros, etc. to implement them. However, I do not just want to **use** machine learning, but also want to **develop novel algorithms** on top of the existing ones, so I want to deeply understand the mathematical details. And in my experience, in order to understand them, reading is not enough, you need to implement. Any suggestions on where/how to start? I have a general machine learning and programming background, but no experience on deep learning. I even don't know what exactly back-propagation is used for, if this would help.
  
  **Tags:** neural-networks
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2017-09-30. Score: 1 | Accepted answer
  
  > I even don't know what exactly back-propagation is used for, if this would help.
  
  Programming backprop manually is one of the best exercises in machine learning. In fact, to do this you don't need a special package, like Tensorflor or Keras, just a general computing library like NumPy (and a whiteboard).
  
  Though I dare to say that VAE is not the best thing to implement, if you are not 100% familiar with backprop. It's pretty advanced technique and you risk to mix up fundamental (e.g. backprop) and specific ideas (e.g. deliberately crafted loss function) and slow down your understanding.
  
  I really think that writing a simple feed-forward neural net (including forward and backward passes) is a very good starting point, after which you can move on to [VAE details](https://jmetzen.github.io/2015-11-27/vae.html), if you'd like.
  
  > Any suggestions on where/how to start?
  
  If you feel confident with algebra and calculus, I highly recommend [this Stanford course](http://cs231n.github.io/), especially its assignments. The course itself is about computer vision and convolutional neural networks, but the first assignments are very general and will give a lot of insight of what and how neural networks are doing.
  
  
  ---
  
  