Answers.Body: <p>The fact that you can approximate any distribution is given by the definition ELBO, which is a lower bound in order to learn <span class="math-container">$p(x)$</span></p>
  <p>Theoretically speaking, if you are able to make that difference between <span class="math-container">$p(x)$</span> and ELBO <span class="math-container">$0$</span>, than you can say that you have actually approximated <span class="math-container">$p(x)$</span> correctly</p>
  <p>This usually doesn't happen, as it is an ideal/optimal solution, which is not what happens when you then introduce mathematical models, optimized with gradient descent, and moreover non convex</p>
  <p>However, you might want to take a look at optimal transport, so you might get a better idea how to transform an arbitrary complex distribution, to a simple one</p>
  <p>About the bottleneck, you can think that your NN is so powerful, that might learn a transformation between your more-complex-prior to a Gaussian, thus a bigger NN might learn the transformation <span class="math-container">$p_\text{data}\rightarrow p_\text{complex prior} \rightarrow p_\text{Gaussian}$</span></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T22:47:59.510Z
Answers.Text: I see, thank you for your fast response. So are you essentially saying that the neural network may learn how to "morph" the space of $x$ using a differentiable mapping into some Gaussian distribution that it can then leverage from that point onward? This was my attempt at summarizing your last point.
  
  Edit: thanks also for pointing me towards optimal transport.
Answers.UserDisplayName: Decaying Tails
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-08T08:38:26.443Z
Answers.Text: @Joel yes, you can phrase it like that... you can even think that each layer learns a step towards that simpler distribution... however, OT transforms distributions, NN transform a sample of that distribution
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-10T00:28:36.740Z
Answers.Text: I see @Alberto Sinigaglia, thank you. I am wondering, however, if the mapping of x to the parameterization of the latent space (usually Gaussian) is deterministic, then why can't we view the process as transforming the entire distribution itself?
Answers.UserDisplayName: Decaying Tails
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-10T00:35:29.440Z
Answers.Text: On second thoughts, I think you mean that the parameters of $p(z|x)$ itself are usually dependent on $x$ and so the NN is technically mapping the sample $x$ to a different distribution - am I interpreting you correctly?
Answers.UserDisplayName: Decaying Tails
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-10T11:34:42.100Z
Answers.Text: @Joel well, first, transforming samples to look like a distribution (VAE), it's much simpler that transforming in closed form the whole distribution (Opt Transport)... secondly, the fact that we are using a distribution in the latent space, it's not always necessary... you can achieve a pseudo-VAE with just a naive AE with a L2 regularization on the latent vectors
Answers.UserDisplayName: Alberto
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-07T22:30:14.017Z
Answers.LastActivityDate: 2023-08-07T22:30:14.017Z
Answers.UserDisplayName: Alberto
Body: <p>I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though (|)
  and (|)
  are Gaussian.</p>
  <p>I understand that the parameters are typically learned using neural networks =()
  and =()
  , where 
  and 
  denote arbitrary neural networks. Therefore, some ideas I had is perhaps the entire VAE is able to learn differentiable, change of variables mappings, which allows the VAE to ensure that, although 
  and 
  both characterize Gaussian distributions, they do so using some transformation of the original variables 
  and/or 
  and so effectively correspond to some arbitrary distribution rather than a Gaussian in 
  or 
  .</p>
  <p>However, and I may just be unnecessarily picky here, but I can't quite convince myself why the Gaussian PDF itself, even with the change of variables mapping, will not act as a &quot;bottleneck&quot; in learning some rather complex non-Gaussian distributions. I was wondering if there is some universal approximation theorem in the case of VAEs, as there in feed-forward neural networks / RNNs etc.</p>
  
Tags: <machine-learning><deep-learning><autoencoders><variational-autoencoder><bayesian-neural-networks>
Title: Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?
UserDisplayName: Decaying Tails
ThreadText: # Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?
  
  (Asked by: username_0 on 2023-08-07)
  
  I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though (|)
  and (|)
  are Gaussian.
  
  I understand that the parameters are typically learned using neural networks =()
  and =()
  , where
  and
  denote arbitrary neural networks. Therefore, some ideas I had is perhaps the entire VAE is able to learn differentiable, change of variables mappings, which allows the VAE to ensure that, although
  and
  both characterize Gaussian distributions, they do so using some transformation of the original variables
  and/or
  and so effectively correspond to some arbitrary distribution rather than a Gaussian in
  or
  .
  
  However, and I may just be unnecessarily picky here, but I can't quite convince myself why the Gaussian PDF itself, even with the change of variables mapping, will not act as a "bottleneck" in learning some rather complex non-Gaussian distributions. I was wondering if there is some universal approximation theorem in the case of VAEs, as there in feed-forward neural networks / RNNs etc.
  
  **Tags:** machine-learning, deep-learning, autoencoders, variational-autoencoder, bayesian-neural-networks
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-08-07. Score: 2 | Accepted answer
  
  The fact that you can approximate any distribution is given by the definition ELBO, which is a lower bound in order to learn $p(x)$
  
  Theoretically speaking, if you are able to make that difference between $p(x)$ and ELBO $0$, than you can say that you have actually approximated $p(x)$ correctly
  
  This usually doesn't happen, as it is an ideal/optimal solution, which is not what happens when you then introduce mathematical models, optimized with gradient descent, and moreover non convex
  
  However, you might want to take a look at optimal transport, so you might get a better idea how to transform an arbitrary complex distribution, to a simple one
  
  About the bottleneck, you can think that your NN is so powerful, that might learn a transformation between your more-complex-prior to a Gaussian, thus a bigger NN might learn the transformation $p\_\text{data}\rightarrow p\_\text{complex prior} \rightarrow p\_\text{Gaussian}$
  
  > **Comments:**
  > **username_0**: I see, thank you for your fast response. So are you essentially saying that the neural network may learn how to "morph" the space of $x$ using a differentiable mapping into some Gaussian distribution that it can then leverage from that point onward? This was my attempt at summarizing your last point.  Edit: thanks also for pointing me towards optimal transport.  
  > **username_1**: @Joel yes, you can phrase it like that... you can even think that each layer learns a step towards that simpler distribution... however, OT transforms distributions, NN transform a sample of that distribution  
  > **username_0**: I see @Alberto Sinigaglia, thank you. I am wondering, however, if the mapping of x to the parameterization of the latent space (usually Gaussian) is deterministic, then why can't we view the process as transforming the entire distribution itself?  
  > **username_0**: On second thoughts, I think you mean that the parameters of $p(z|x)$ itself are usually dependent on $x$ and so the NN is technically mapping the sample $x$ to a different distribution - am I interpreting you correctly?  
  > **username_1**: @Joel well, first, transforming samples to look like a distribution (VAE), it's much simpler that transforming in closed form the whole distribution (Opt Transport)... secondly, the fact that we are using a distribution in the latent space, it's not always necessary... you can achieve a pseudo-VAE with just a naive AE with a L2 regularization on the latent vectors  
  
  ---
  
  