Body: <p>It struck me that regular auto-encoders with batch-norm and dropout have quite similar properties to VAEs which made me wonder whether VAEs where really much better than this simpler alternative. Let me explain:</p>
  <ul>
  <li>BatchNorm: encourages activations to follow N(0,1), just like KL-divergence does to the output layer of the encoder</li>
  <li>Dropout: during training time creates random (Gaussian due to CLT) encoder output distribution, mean and variance are learnable indirectly via network weights and mean/variance override params of batch norm layer</li>
  </ul>
  <p>You might argue that KL-divergence adds more flexibility by allowing it to not exactly follow N(0,1) but batch norm allows learning to override this by default anyways.</p>
  <p>All that considered are there really any practical benefits to VAEs which cannot be provided by this simpler setup?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-22T15:20:09.893Z
Comments.Text: Have you seen any work that uses autoencoders with batch norm and dropout and claimed that this would be (somehow) equivalent to VAEs? If yes, it might be a good idea to provide that reference. I know that, Monte Carlo Dropout with neural networks can be viewed as Bayesian neural networks. This is the closest thing to what you're describing, but for regular neural networks, not autoencoders.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-22T16:06:11.313Z
Comments.Text: No I'm just reasoning from first principles here, I could be wrong. But if I am I'm curious why.
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-22T16:21:48.403Z
Comments.Text: @nbro
  I take that back I did just find a reference trying to do something similar to what I mentioned: https://vitalab.github.io/article/2021/08/20/VariationalToDeterministicAE.html
  
  They say they can use deterministic auto-encoders + injected noise for regularization as an alternative to VAEs. That is precisely what dropout is (though they don't use dropout explicitly).
Comments.UserDisplayName: profPlum
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-03-22T16:41:41.987Z
Comments.Text: De-noising auto-encoders also are essentially auto-encoders with dropout. But I don't know if anyone has directly compared them to VAEs. Still I sense they are similar...
  
   https://proceedings.neurips.cc/paper/2013/file/559cb990c9dffd8675f6bc2186971dc2-Paper.pdf
Comments.UserDisplayName: profPlum
Tags: <autoencoders><variational-autoencoder>
Title: VAEs vs Autoencoders with BatchNorm and Dropout?
UserDisplayName: profPlum
ThreadText: # VAEs vs Autoencoders with BatchNorm and Dropout?
  
  (Asked by: username_0 on 2022-03-22)
  
  It struck me that regular auto-encoders with batch-norm and dropout have quite similar properties to VAEs which made me wonder whether VAEs where really much better than this simpler alternative. Let me explain:
  
  * BatchNorm: encourages activations to follow N(0,1), just like KL-divergence does to the output layer of the encoder
  * Dropout: during training time creates random (Gaussian due to CLT) encoder output distribution, mean and variance are learnable indirectly via network weights and mean/variance override params of batch norm layer
  
  You might argue that KL-divergence adds more flexibility by allowing it to not exactly follow N(0,1) but batch norm allows learning to override this by default anyways.
  
  All that considered are there really any practical benefits to VAEs which cannot be provided by this simpler setup?
  
  **Tags:** autoencoders, variational-autoencoder
  
  > **Comments:**
  > **username_1**: Have you seen any work that uses autoencoders with batch norm and dropout and claimed that this would be (somehow) equivalent to VAEs? If yes, it might be a good idea to provide that reference. I know that, Monte Carlo Dropout with neural networks can be viewed as Bayesian neural networks. This is the closest thing to what you're describing, but for regular neural networks, not autoencoders.  
  > **username_0**: No I'm just reasoning from first principles here, I could be wrong. But if I am I'm curious why.  
  > **username_0**: @username_1 I take that back I did just find a reference trying to do something similar to what I mentioned: https://vitalab.github.io/article/2021/08/20/VariationalToDeterministicAE.html  They say they can use deterministic auto-encoders + injected noise for regularization as an alternative to VAEs. That is precisely what dropout is (though they don't use dropout explicitly).  
  > **username_0**: De-noising auto-encoders also are essentially auto-encoders with dropout. But I don't know if anyone has directly compared them to VAEs. Still I sense they are similar...   https://proceedings.neurips.cc/paper/2013/file/559cb990c9dffd8675f6bc2186971dc2-Paper.pdf  
  
  ---
  
  ## 0 answers
  
  