Answers.Body: <p>VAE's try and model the <strong>distribution</strong> of your data. So it's not going to learn " images composed of random noise at each pixel" <em>per se</em> (though, if overfitting, it could remember them). But it would be very capable of learning the simple noise <strong>distribution</strong> from which you sampled your random pixels</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:09:26.670Z
Answers.Text: The question is not well-posed, so you should have waited for clarification before answering.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:11:53.967Z
Answers.Text: I think the question is well posed, you seemed not to have understood it the way I did, is all :) the user seems to wonder if there's something to learn from noise and I believe my answer answered just that
Answers.UserDisplayName: ted
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:14:19.083Z
Answers.Text: Yes, I admit that I missed one part. But it's still not clear whether the user wants to know about 1. images whose pixels have been generated from $N(0, 1)$ or 2. images to which we have added Gaussian noise. These are two different things! In one case, we are just corrupting the image, in the other case, we are really generating "Gaussian images".
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:25:41.543Z
Answers.Text: I really understand the question as being about 1 => "composed of random noise at each pixel" ; but only they can actually answer that, true
Answers.UserDisplayName: ted
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:33:43.087Z
Answers.Text: Indeed, I meant 1), images are generated where each pixel is Gaussian noise, before anything to do with the VAE. Sorry @nbro for not being clear!
Answers.UserDisplayName: user1234544
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:39:13.417Z
Answers.Text: @ted if it were to learn the simple noise (standard normal) distribution, would it still be able to decompress/decode from the latent space accurately to the original image? Because of the introduction of random noise, reparameterization trick, wouldn't the latent space effectively look like a completely different one of those images (with pixels still from the standard normal)?
Answers.UserDisplayName: user1234544
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:50:55.600Z
Answers.Text: I think you're confused by epsilon, it's a sampling procedure not a noise added to the latent representation or anything, it's very different from the generating process of your data
Answers.UserDisplayName: ted
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T01:42:14.433Z
Answers.Text: @ted what if the latent space were the same size as the input, would it learn the distribution better than if it had to compress under these circumstances?
Answers.UserDisplayName: user1234544
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-09T01:45:40.933Z
Answers.Text: @ted Well, the pixels of the "random images" are also sampled, so I understand the confusion of the OP.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T23:05:27.897Z
Answers.LastActivityDate: 2020-04-08T23:05:27.897Z
Answers.UserDisplayName: ted
Body: <p>Can a variational auto-encoder (VAE) learn images whose pixels have been generated from a Gaussian distribution (e.g. <span class="math-container">$N(0, 1)$</span>), i.e. each pixel is a sample from <span class="math-container">$N(0, 1)$</span>?</p>
  
  <p>My gut feeling says no, because the VAE adds additional noise <span class="math-container">$\epsilon$</span> to the original image in the latent space, and if all images and pixels are random from the same distribution it would be impossible do decode/reconstruct into the particular input image. However, VAEs are a bit of a mystery to me internally. Any help would be appreciated!</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-08T22:05:04.803Z
Comments.Text: Hi and welcome to AI SE! The $\epsilon$ that you're talking about probably refers to the sample drawn from $N(0,1)$ during the forward pass of the encoder. This is the so-called "re-parametrization trick", i.e. you do not sample directly from the latent space (i.e. you do not sample directly $z$, i.e. a latent representation of the image), but you sample $\epsilon$, then you combine it with the mean and variance of the distribution of the latent space to produce $z$. Why? Because this apparently makes training more stable (i.e. it's a variance reduction technique).
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-08T22:22:19.567Z
Comments.Text: @nbro I see, thanks for the explanation! So in this case, would it be that the VAE can actually learn these images composed of random pixels?
Comments.UserDisplayName: user1234544
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-08T23:05:50.440Z
Comments.Text: It's a bit ambiguous what you mean by "images composed of random noise at each pixel (each drawn from, say N(0, 1))"? Do you mean that 1. you add a sample from N(0, 1) to each pixel of the original image (i.e. you add Gaussian noise to some image, e.g. a face of a person), or 2. every pixel is drawn from $N(0, 1)$ (i.e. you generate the images really by drawing a sample from $N(0, 1)$)? I believe it's option two. Anyway, by "random", one usually means draw from the uniform distribution.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-08T23:06:40.403Z
Comments.Text: Anyway, I don't think you can learn images whose pixels have been generated from a uniform distribution. In general, in those cases, there will be no pattern to learn (i.e. anything to compress). Have a look at information entropy. However, there are denoising auto-encoders. These can reconstruct the original inputs (i.e. not the noisy ones), but the noisy inputs are "just" noisy  (i.e. there are still patterns in those noisy images), so they are not uniformly randomly generated.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-08T23:36:18.917Z
Comments.Text: @nbro yes, sorry, it is 2, every pixel is random Gaussian noise. From what you're saying, it can't learn a pattern because there is none? What if we were to use a latent space of size equal to the input space size, would it be able to learn then?
Comments.UserDisplayName: user1234544
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-08T23:43:51.673Z
Comments.Text: I was talking about the case where pixels are generated from a uniform distribution, not from a normal distribution. Anyway, note that N(0, 1) doesn't restrict the values of the pixels to be in the range [0, 1].
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-09T23:28:34.460Z
Comments.Text: Why did you delete this post and the other? I can delete it again, but I just wanted to understand why.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-10T03:08:14.080Z
Comments.Text: @nbro Question wasn't well-posed and the other one was absolutely all over the place as you pointed out, as a result I guess it got overlooked by people who may have been able to answer and didn't get much interaction at all. I think I'm going to get my ideas sorted, and post a more precise and thought-out question tomorrow that should be easier to answer and understand for people. I'll see you then!
Comments.UserDisplayName: user1234544
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-10T03:32:11.633Z
Comments.Text: This question is actually interesting, but feel free to delete it again if you want. Your other post was also interesting, although it needed clarification initially and you were asking too many things. Try to ask more specific questions next time and ask only one question per post. If you have multiple questions, ask each of them in its separate post.  Also, note that you may not receive an answer, but it doesn't mean your question is not interesting. Our community is still small, but, hopefully, we will grow and provide more help.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-10T18:35:18.113Z
Comments.Text: @nbro sounds good, let's keep this one then.
Comments.UserDisplayName: user1234544
Tags: <deep-learning><autoencoders><probability-distribution><variational-autoencoder>
Title: Can a variational auto-encoder learn images composed of random noise at each pixel (each drawn from the same distribution)?
UserDisplayName: user1234544
ThreadText: # Can a variational auto-encoder learn images composed of random noise at each pixel (each drawn from the same distribution)?
  
  (Asked by: username_0 on 2020-04-08)
  
  Can a variational auto-encoder (VAE) learn images whose pixels have been generated from a Gaussian distribution (e.g. $N(0, 1)$), i.e. each pixel is a sample from $N(0, 1)$?
  
  My gut feeling says no, because the VAE adds additional noise $\epsilon$ to the original image in the latent space, and if all images and pixels are random from the same distribution it would be impossible do decode/reconstruct into the particular input image. However, VAEs are a bit of a mystery to me internally. Any help would be appreciated!
  
  **Tags:** deep-learning, autoencoders, probability-distribution, variational-autoencoder
  
  > **Comments:**
  > **username_1**: Hi and welcome to AI SE! The $\epsilon$ that you're talking about probably refers to the sample drawn from $N(0,1)$ during the forward pass of the encoder. This is the so-called "re-parametrization trick", i.e. you do not sample directly from the latent space (i.e. you do not sample directly $z$, i.e. a latent representation of the image), but you sample $\epsilon$, then you combine it with the mean and variance of the distribution of the latent space to produce $z$. Why? Because this apparently makes training more stable (i.e. it's a variance reduction technique).  
  > **username_0**: @username_1 I see, thanks for the explanation! So in this case, would it be that the VAE can actually learn these images composed of random pixels?  
  > **username_1**: It's a bit ambiguous what you mean by "images composed of random noise at each pixel (each drawn from, say N(0, 1))"? Do you mean that 1. you add a sample from N(0, 1) to each pixel of the original image (i.e. you add Gaussian noise to some image, e.g. a face of a person), or 2. every pixel is drawn from $N(0, 1)$ (i.e. you generate the images really by drawing a sample from $N(0, 1)$)? I believe it's option two. Anyway, by "random", one usually means draw from the uniform distribution.  
  > **username_1**: Anyway, I don't think you can learn images whose pixels have been generated from a uniform distribution. In general, in those cases, there will be no pattern to learn (i.e. anything to compress). Have a look at information entropy. However, there are denoising auto-encoders. These can reconstruct the original inputs (i.e. not the noisy ones), but the noisy inputs are "just" noisy  (i.e. there are still patterns in those noisy images), so they are not uniformly randomly generated.  
  > **username_0**: @username_1 yes, sorry, it is 2, every pixel is random Gaussian noise. From what you're saying, it can't learn a pattern because there is none? What if we were to use a latent space of size equal to the input space size, would it be able to learn then?  
  > **username_1**: I was talking about the case where pixels are generated from a uniform distribution, not from a normal distribution. Anyway, note that N(0, 1) doesn't restrict the values of the pixels to be in the range [0, 1].  
  > **username_1**: Why did you delete this post and the other? I can delete it again, but I just wanted to understand why.  
  > **username_0**: @username_1 Question wasn't well-posed and the other one was absolutely all over the place as you pointed out, as a result I guess it got overlooked by people who may have been able to answer and didn't get much interaction at all. I think I'm going to get my ideas sorted, and post a more precise and thought-out question tomorrow that should be easier to answer and understand for people. I'll see you then!  
  > **username_1**: This question is actually interesting, but feel free to delete it again if you want. Your other post was also interesting, although it needed clarification initially and you were asking too many things. Try to ask more specific questions next time and ask only one question per post. If you have multiple questions, ask each of them in its separate post.  Also, note that you may not receive an answer, but it doesn't mean your question is not interesting. Our community is still small, but, hopefully, we will grow and provide more help.  
  > **username_0**: @username_1 sounds good, let's keep this one then.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-08. Score: 1 | Accepted answer
  
  VAE's try and model the **distribution** of your data. So it's not going to learn " images composed of random noise at each pixel" *per se* (though, if overfitting, it could remember them). But it would be very capable of learning the simple noise **distribution** from which you sampled your random pixels
  
  > **Comments:**
  > **username_1**: The question is not well-posed, so you should have waited for clarification before answering.  
  > **username_2**: I think the question is well posed, you seemed not to have understood it the way I did, is all :) the user seems to wonder if there's something to learn from noise and I believe my answer answered just that  
  > **username_1**: Yes, I admit that I missed one part. But it's still not clear whether the user wants to know about 1. images whose pixels have been generated from $N(0, 1)$ or 2. images to which we have added Gaussian noise. These are two different things! In one case, we are just corrupting the image, in the other case, we are really generating "Gaussian images".  
  > **username_2**: I really understand the question as being about 1 => "composed of random noise at each pixel" ; but only they can actually answer that, true  
  > **username_0**: Indeed, I meant 1), images are generated where each pixel is Gaussian noise, before anything to do with the VAE. Sorry @username_1 for not being clear!  
  > **username_0**: @username_2 if it were to learn the simple noise (standard normal) distribution, would it still be able to decompress/decode from the latent space accurately to the original image? Because of the introduction of random noise, reparameterization trick, wouldn't the latent space effectively look like a completely different one of those images (with pixels still from the standard normal)?  
  > **username_2**: I think you're confused by epsilon, it's a sampling procedure not a noise added to the latent representation or anything, it's very different from the generating process of your data  
  > **username_0**: @username_2 what if the latent space were the same size as the input, would it learn the distribution better than if it had to compress under these circumstances?  
  > **username_1**: @username_2 Well, the pixels of the "random images" are also sampled, so I understand the confusion of the OP.  
  
  ---
  
  