Answers.Body: <p>If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize the change in distribution under the application of an autoencoder.</p>
  <p>Let <span class="math-container">$g:\mathcal{X}\rightarrow\mathcal{Z}$</span> be the encoder, and <span class="math-container">$f:\mathcal{Z}\rightarrow\mathcal{X}$</span> be the decoder. <span class="math-container">$\mathcal{X}$</span> corresponds to the input space, whereas <span class="math-container">$\mathcal{Z}$</span>  corresponds to the latent space. Let <span class="math-container">$P$</span> be the true distribution of your data, from which you have an i.i.d. sample <span class="math-container">$S = \{\mathbf{x}_{i}\}_{i=1}^{n}$</span>. You can approximate <span class="math-container">$P$</span> using your sample,</p>
  <p><span class="math-container">$$\hat{P}(\mathbf{x}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{x}-\mathbf{x}_{i})$$</span></p>
  <p>this is called an empirical approximation of <span class="math-container">$P$</span>. Now, if you want to study the distribution of <span class="math-container">$\mathbf{z}_{i} = g(\mathbf{x}_{i})$</span>, you can write a new distribution,</p>
  <p><span class="math-container">$$\hat{Q}(\mathbf{z}) = \dfrac{1}{n}\sum_{i=1}^{n}\delta(\mathbf{z}-\mathbf{z}_{i})$$</span></p>
  <p>which is an approximation of <span class="math-container">$Q = g_{\sharp}P$</span>, where <span class="math-container">$g_{\sharp}$</span> is the <a href="https://en.wikipedia.org/wiki/Pushforward_measure" rel="nofollow noreferrer">pushforward</a> of <span class="math-container">$P$</span> by <span class="math-container">$g$</span>. Note that this does not tell us much about the properties of <span class="math-container">$Q$</span>, since we did not make enough assumptions. In some frameworks (e.g. <a href="https://en.wikipedia.org/wiki/Transportation_theory_(mathematics)" rel="nofollow noreferrer">Optimal Transport</a>), this formulation is very useful. If you want to read more on an probabilistic/transport analysis of the distributions in an autoencoder you can take a look on [1].</p>
  <h2>References</h2>
  <p>[1] Sonoda, S., &amp; Murata, N. (2019). <a href="https://jmlr.org/papers/volume20/16-243/16-243.pdf" rel="nofollow noreferrer">Transport analysis of infinitely deep neural network</a>. The Journal of Machine Learning Research, 20(1), 31-82.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-07-11T07:20:32.630Z
Answers.LastActivityDate: 2022-07-11T07:20:32.630Z
Answers.UserDisplayName: Eduardo Montesuma
Body: <p>Is there any result on the distribution of autoencoder embeddings?</p>
  <p>For example, the following image (taken from <a href="https://medium.com/dataseries/convolutional-autoencoder-in-pytorch-on-mnist-dataset-d65145c132ac#97ad" rel="nofollow noreferrer">this article</a>) visualizes the latent space with t-SNE. As you can see, images from the same class (in this case MNIST digits) roughly form a cluster. Is there any theory explaining why that is the case?</p>
  <p><a href="https://i.sstatic.net/Htier.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/Htier.png" alt="enter image description here" /></a></p>
  <p>Note that I am talking about autoencoders, not variational autoencoders, whose embedding by definition has a multi-variate Gaussian distribution.</p>
  
Tags: <neural-networks><autoencoders><generative-model><probability-distribution><embeddings>
Title: What is the distribution of autoencoder embeddings?
UserDisplayName: nalzok
ThreadText: # What is the distribution of autoencoder embeddings?
  
  (Asked by: username_0 on 2022-07-11)
  
  Is there any result on the distribution of autoencoder embeddings?
  
  For example, the following image (taken from [this article](https://medium.com/dataseries/convolutional-autoencoder-in-pytorch-on-mnist-dataset-d65145c132ac#97ad)) visualizes the latent space with t-SNE. As you can see, images from the same class (in this case MNIST digits) roughly form a cluster. Is there any theory explaining why that is the case?
  
  [![enter image description here](https://i.sstatic.net/Htier.png)](https://i.sstatic.net/Htier.png)
  
  Note that I am talking about autoencoders, not variational autoencoders, whose embedding by definition has a multi-variate Gaussian distribution.
  
  **Tags:** neural-networks, autoencoders, generative-model, probability-distribution, embeddings
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-07-11. Score: 1
  
  If you don't make assumptions about your input distribution, and the form of your network, it's very difficult to express the embedding distribution in closed-form. Here's an idea on how to formalize the change in distribution under the application of an autoencoder.
  
  Let $g:\mathcal{X}\rightarrow\mathcal{Z}$ be the encoder, and $f:\mathcal{Z}\rightarrow\mathcal{X}$ be the decoder. $\mathcal{X}$ corresponds to the input space, whereas $\mathcal{Z}$ corresponds to the latent space. Let $P$ be the true distribution of your data, from which you have an i.i.d. sample $S = \{\mathbf{x}\_{i}\}\_{i=1}^{n}$. You can approximate $P$ using your sample,
  
  $$\hat{P}(\mathbf{x}) = \dfrac{1}{n}\sum\_{i=1}^{n}\delta(\mathbf{x}-\mathbf{x}\_{i})$$
  
  this is called an empirical approximation of $P$. Now, if you want to study the distribution of $\mathbf{z}\_{i} = g(\mathbf{x}\_{i})$, you can write a new distribution,
  
  $$\hat{Q}(\mathbf{z}) = \dfrac{1}{n}\sum\_{i=1}^{n}\delta(\mathbf{z}-\mathbf{z}\_{i})$$
  
  which is an approximation of $Q = g\_{\sharp}P$, where $g\_{\sharp}$ is the [pushforward](https://en.wikipedia.org/wiki/Pushforward_measure) of $P$ by $g$. Note that this does not tell us much about the properties of $Q$, since we did not make enough assumptions. In some frameworks (e.g. [Optimal Transport](https://en.wikipedia.org/wiki/Transportation_theory_(mathematics))), this formulation is very useful. If you want to read more on an probabilistic/transport analysis of the distributions in an autoencoder you can take a look on [1].
  
  ## References
  
  [1] Sonoda, S., & Murata, N. (2019). [Transport analysis of infinitely deep neural network](https://jmlr.org/papers/volume20/16-243/16-243.pdf). The Journal of Machine Learning Research, 20(1), 31-82.
  
  
  ---
  
  