Body: <p>I am trying to train a VAE for anomaly detection. I chose one architecture from <a href="https://github.com/1Konny/Beta-VAE" rel="nofollow noreferrer">this Github repository</a> and I adjusted the input and output to match what I need. In my case, the input (and hence the output) are a 12D vector. I tried several sizes for the latent space, but, for some reason, it's not training. From the beginning, the KL loss in almost zero (around 1e-10), while the reconstruction loss (MSE for Gaussian distribution) is around 1, and they basically vary around these values without learning anything further. </p>
  
  <p>Are there any general tips for troubleshooting a VAE (I never trained one before)?</p>
  
  <p>I am pretty sure that the code is right and the data for sure has a background and signal (the ratio is 10:1), so I am not really sure what I am doing wrong.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-15T06:23:19.620Z
Comments.Text: Hello bill welcome to AI:SE!  Maybe try checking on the gradients to see if they are vanishing? If you ' adjusted the input and output to match what I need' without adjusting the quantity of layers and ensuring that the gradients don't vanish or explode anywhere in the middle then that could be the issue.
Comments.UserDisplayName: Michael Hearn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-11-15T15:06:13.017Z
Comments.Text: @MichaelHearn thank you for the idea! I will look into it. However, my NN is 6-7 layers deep, so I doubt there is a major gradient problem (I trained a lot deeper ones without BatchNorm or anything special, without any problem).
Comments.UserDisplayName: Bill
Tags: <generative-model><autoencoders><implementation><variational-autoencoder>
Title: Are there any general tips for troubleshooting a VAE when apparently it is not learning?
UserDisplayName: Bill
ThreadText: # Are there any general tips for troubleshooting a VAE when apparently it is not learning?
  
  (Asked by: username_0 on 2019-11-15)
  
  I am trying to train a VAE for anomaly detection. I chose one architecture from [this Github repository](https://github.com/1Konny/Beta-VAE) and I adjusted the input and output to match what I need. In my case, the input (and hence the output) are a 12D vector. I tried several sizes for the latent space, but, for some reason, it's not training. From the beginning, the KL loss in almost zero (around 1e-10), while the reconstruction loss (MSE for Gaussian distribution) is around 1, and they basically vary around these values without learning anything further.
  
  Are there any general tips for troubleshooting a VAE (I never trained one before)?
  
  I am pretty sure that the code is right and the data for sure has a background and signal (the ratio is 10:1), so I am not really sure what I am doing wrong.
  
  **Tags:** generative-model, autoencoders, implementation, variational-autoencoder
  
  > **Comments:**
  > **username_1**: Hello username_0 welcome to AI:SE!  Maybe try checking on the gradients to see if they are vanishing? If you ' adjusted the input and output to match what I need' without adjusting the quantity of layers and ensuring that the gradients don't vanish or explode anywhere in the middle then that could be the issue.  
  > **username_0**: @username_1 thank you for the idea! I will look into it. However, my NN is 6-7 layers deep, so I doubt there is a major gradient problem (I trained a lot deeper ones without BatchNorm or anything special, without any problem).  
  
  ---
  
  ## 0 answers
  
  