<p>The fact that you can approximate any distribution is given by the definition ELBO, which is a lower bound in order to learn <span class="math-container">$p(x)$</span></p>
<p>Theoretically speaking, if you are able to make that difference between <span class="math-container">$p(x)$</span> and ELBO <span class="math-container">$0$</span>, than you can say that you have actually approximated <span class="math-container">$p(x)$</span> correctly</p>
<p>This usually doesn't happen, as it is an ideal/optimal solution, which is not what happens when you then introduce mathematical models, optimized with gradient descent, and moreover non convex</p>
<p>However, you might want to take a look at optimal transport, so you might get a better idea how to transform an arbitrary complex distribution, to a simple one</p>
<p>About the bottleneck, you can think that your NN is so powerful, that might learn a transformation between your more-complex-prior to a Gaussian, thus a bigger NN might learn the transformation <span class="math-container">$p_\text{data}\rightarrow p_\text{complex prior} \rightarrow p_\text{Gaussian}$</span></p>

CC BY-SA 4.0
2023-08-07T22:47:59.510Z
I see, thank you for your fast response. So are you essentially saying that the neural network may learn how to "morph" the space of $x$ using a differentiable mapping into some Gaussian distribution that it can then leverage from that point onward? This was my attempt at summarizing your last point.

Edit: thanks also for pointing me towards optimal transport.
Decaying Tails
CC BY-SA 4.0
2023-08-08T08:38:26.443Z
@Joel yes, you can phrase it like that... you can even think that each layer learns a step towards that simpler distribution... however, OT transforms distributions, NN transform a sample of that distribution
Alberto
CC BY-SA 4.0
2023-08-10T00:28:36.740Z
I see @Alberto Sinigaglia, thank you. I am wondering, however, if the mapping of x to the parameterization of the latent space (usually Gaussian) is deterministic, then why can't we view the process as transforming the entire distribution itself?
Decaying Tails
CC BY-SA 4.0
2023-08-10T00:35:29.440Z
On second thoughts, I think you mean that the parameters of $p(z|x)$ itself are usually dependent on $x$ and so the NN is technically mapping the sample $x$ to a different distribution - am I interpreting you correctly?
Decaying Tails
CC BY-SA 4.0
2023-08-10T11:34:42.100Z
@Joel well, first, transforming samples to look like a distribution (VAE), it's much simpler that transforming in closed form the whole distribution (Opt Transport)... secondly, the fact that we are using a distribution in the latent space, it's not always necessary... you can achieve a pseudo-VAE with just a naive AE with a L2 regularization on the latent vectors
Alberto
CC BY-SA 4.0
2023-08-07T22:30:14.017Z
2023-08-07T22:30:14.017Z
Alberto
<p>I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though (|)
and (|)
are Gaussian.</p>
<p>I understand that the parameters are typically learned using neural networks =()
and =()
, where 
and 
denote arbitrary neural networks. Therefore, some ideas I had is perhaps the entire VAE is able to learn differentiable, change of variables mappings, which allows the VAE to ensure that, although 
and 
both characterize Gaussian distributions, they do so using some transformation of the original variables 
and/or 
and so effectively correspond to some arbitrary distribution rather than a Gaussian in 
or 
.</p>
<p>However, and I may just be unnecessarily picky here, but I can't quite convince myself why the Gaussian PDF itself, even with the change of variables mapping, will not act as a &quot;bottleneck&quot; in learning some rather complex non-Gaussian distributions. I was wondering if there is some universal approximation theorem in the case of VAEs, as there in feed-forward neural networks / RNNs etc.</p>

<machine-learning><deep-learning><autoencoders><variational-autoencoder><bayesian-neural-networks>
Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?
Decaying Tails
# Why can Variational Autoencoders (VAEs) approximate arbitrary distributions?

(Asked by: username_0 on 2023-08-07)

I am trying to reason to myself why is it that VAEs can approximate arbitrary probability distributions even though (|)
and (|)
are Gaussian.

I understand that the parameters are typically learned using neural networks =()
and =()
, where
and
denote arbitrary neural networks. Therefore, some ideas I had is perhaps the entire VAE is able to learn differentiable, change of variables mappings, which allows the VAE to ensure that, although
and
both characterize Gaussian distributions, they do so using some transformation of the original variables
and/or
and so effectively correspond to some arbitrary distribution rather than a Gaussian in
or
.

However, and I may just be unnecessarily picky here, but I can't quite convince myself why the Gaussian PDF itself, even with the change of variables mapping, will not act as a "bottleneck" in learning some rather complex non-Gaussian distributions. I was wondering if there is some universal approximation theorem in the case of VAEs, as there in feed-forward neural networks / RNNs etc.

**Tags:** machine-learning, deep-learning, autoencoders, variational-autoencoder, bayesian-neural-networks


---

## 1 answers

### Answer by username_1 on 2023-08-07. Score: 2 | Accepted answer

The fact that you can approximate any distribution is given by the definition ELBO, which is a lower bound in order to learn $p(x)$

Theoretically speaking, if you are able to make that difference between $p(x)$ and ELBO $0$, than you can say that you have actually approximated $p(x)$ correctly

This usually doesn't happen, as it is an ideal/optimal solution, which is not what happens when you then introduce mathematical models, optimized with gradient descent, and moreover non convex

However, you might want to take a look at optimal transport, so you might get a better idea how to transform an arbitrary complex distribution, to a simple one

About the bottleneck, you can think that your NN is so powerful, that might learn a transformation between your more-complex-prior to a Gaussian, thus a bigger NN might learn the transformation $p\_\text{data}\rightarrow p\_\text{complex prior} \rightarrow p\_\text{Gaussian}$

> **Comments:**
> **username_0**: I see, thank you for your fast response. So are you essentially saying that the neural network may learn how to "morph" the space of $x$ using a differentiable mapping into some Gaussian distribution that it can then leverage from that point onward? This was my attempt at summarizing your last point.  Edit: thanks also for pointing me towards optimal transport.  
> **username_1**: @Joel yes, you can phrase it like that... you can even think that each layer learns a step towards that simpler distribution... however, OT transforms distributions, NN transform a sample of that distribution  
> **username_0**: I see @username_1 Sinigaglia, thank you. I am wondering, however, if the mapping of x to the parameterization of the latent space (usually Gaussian) is deterministic, then why can't we view the process as transforming the entire distribution itself?  
> **username_0**: On second thoughts, I think you mean that the parameters of $p(z|x)$ itself are usually dependent on $x$ and so the NN is technically mapping the sample $x$ to a different distribution - am I interpreting you correctly?  
> **username_1**: @Joel well, first, transforming samples to look like a distribution (VAE), it's much simpler that transforming in closed form the whole distribution (Opt Transport)... secondly, the fact that we are using a distribution in the latent space, it's not always necessary... you can achieve a pseudo-VAE with just a naive AE with a L2 regularization on the latent vectors  

---

