Answers.Body: <p>The only disadvantage and difference between these generative models and the method you describe, is the input. You describe inputting tags, where as for a GAN, or VAE, the generation segment of the model takes in some representation of a probability distribution. For a GAN, it's mostly random noise, and for a VAE it is some latent space (see nbros answer).</p>
  <p>Your described method prevents the network from properly learning fluid generation. If you have a discrete input, the network will attempt to perform a sort of classification on the input, rather then generation, and so when you try to generate a new image, you will most likely get the image equivalent of gibberish.</p>
  <p>In fact, that's why a standard auto-encoder (not variational) doesn't work very well for generation. You would think that you could feed in your own custom input into the latent space:<a href="https://i.sstatic.net/NZ8ze.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/NZ8ze.png" alt="Example" /></a></p>
  <p>But if you tried this, you would end up telling the network to try and generate something from a latent space it can't interpret.</p>
  <p>Hence where the &quot;variational&quot; part of the VAE comes in. This helps the network learn to generate from a <em>continuous</em> distribution, so no matter what input you use, the network will be able to interpret it and give an appropriate output.</p>
  <p>As for a GAN, it is simply fed random noise at each training step, so it too generates based on a continuous distribution.</p>
  <p>If you were to try and train your method of generation, I would predict that you would find an average of all images that share similar tags (say you have the tags of cat, dog and brown haired, if you input &quot;dog=1, cat=0, brown haired=1&quot; you would get an average of all brown haired dogs), but if you tried to input a combination of tags the network has not seen, as it has not learnt from a continuous distribution, the resultant image would not be anything like what you'd expect from those tags.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T13:15:36.513Z
Answers.Text: Thanks! I guess in the end the main problem with direct mapping is that the network never sees intermediate inputs. So it is unlikely that it will produce high quality images for them.
Answers.UserDisplayName: Avetik
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-12T05:46:43.387Z
Answers.LastActivityDate: 2019-12-12T05:46:43.387Z
Answers.LastEditDate: 2020-06-17T09:57:20.710Z
Answers.UserDisplayName: Recessive
Answers.Body: <p><em>I will only focus on the VAE because I am more familiar with it, but the explanations may also apply to the GAN and other generative models.</em></p>
  
  <p>In the case of the VAE, you train a neural network not only to generate images but to represent them <strong>compactly</strong> in a so-called <strong>latent space</strong>, so you train the VAE to do <strong>dimensionality reduction</strong>. More precisely, the VAE attempts to learn a probability distribution with smaller dimensionality than the dimensionality of the training data but that hopefully represents the training data. Consequently, the model is forced to learn the essential features of the probability distribution that generated the training data.</p>
  
  <p>The VAE is a <a href="https://en.wikipedia.org/wiki/Generative_model" rel="nofollow noreferrer">generative model</a> rather than a <a href="https://en.wikipedia.org/wiki/Discriminative_model" rel="nofollow noreferrer">discriminative model</a>. In the case of the neural network that maps inputs to images, you will not be learning a latent probability distribution (unless you formulate your model in such a way), from which you could sample, but you would just be mapping, in a supervised way (that is, you would need a labelled training dataset) and deterministically, the inputs to the images. In the VAE, there's some form of stochasticity, while training (and testing) the model.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T18:18:59.553Z
Answers.Text: I will probably improve this answer later!
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-11T18:08:50.980Z
Answers.LastActivityDate: 2019-12-11T18:14:33.003Z
Answers.LastEditDate: 2019-12-11T18:14:33.003Z
Answers.UserDisplayName: nbro
Body: <p>Say a simple neural network's input is a collection of tags (encoded in some way), and the output is an image that corresponds to those tags. Say this network consists of some dense layers and some reverse (transpose) convolution layers.</p>
  
  <p>What is the disadvantage of this network, that directs people to invent fairly complicated things like GANs or VAEs?</p>
  
Tags: <convolutional-neural-networks><generative-adversarial-networks><image-generation><variational-autoencoder>
Title: What makes GAN or VAE better at image generation than NN that directly maps inputs to images
UserDisplayName: Avetik
ThreadText: # What makes GAN or VAE better at image generation than NN that directly maps inputs to images
  
  (Asked by: username_0 on 2019-12-11)
  
  Say a simple neural network's input is a collection of tags (encoded in some way), and the output is an image that corresponds to those tags. Say this network consists of some dense layers and some reverse (transpose) convolution layers.
  
  What is the disadvantage of this network, that directs people to invent fairly complicated things like GANs or VAEs?
  
  **Tags:** convolutional-neural-networks, generative-adversarial-networks, image-generation, variational-autoencoder
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-12-12. Score: 2 | Accepted answer
  
  The only disadvantage and difference between these generative models and the method you describe, is the input. You describe inputting tags, where as for a GAN, or VAE, the generation segment of the model takes in some representation of a probability distribution. For a GAN, it's mostly random noise, and for a VAE it is some latent space (see nbros answer).
  
  Your described method prevents the network from properly learning fluid generation. If you have a discrete input, the network will attempt to perform a sort of classification on the input, rather then generation, and so when you try to generate a new image, you will most likely get the image equivalent of gibberish.
  
  In fact, that's why a standard auto-encoder (not variational) doesn't work very well for generation. You would think that you could feed in your own custom input into the latent space:[![Example](https://i.sstatic.net/NZ8ze.png)](https://i.sstatic.net/NZ8ze.png)
  
  But if you tried this, you would end up telling the network to try and generate something from a latent space it can't interpret.
  
  Hence where the "variational" part of the VAE comes in. This helps the network learn to generate from a *continuous* distribution, so no matter what input you use, the network will be able to interpret it and give an appropriate output.
  
  As for a GAN, it is simply fed random noise at each training step, so it too generates based on a continuous distribution.
  
  If you were to try and train your method of generation, I would predict that you would find an average of all images that share similar tags (say you have the tags of cat, dog and brown haired, if you input "dog=1, cat=0, brown haired=1" you would get an average of all brown haired dogs), but if you tried to input a combination of tags the network has not seen, as it has not learnt from a continuous distribution, the resultant image would not be anything like what you'd expect from those tags.
  
  > **Comments:**
  > **username_0**: Thanks! I guess in the end the main problem with direct mapping is that the network never sees intermediate inputs. So it is unlikely that it will produce high quality images for them.  
  
  ---
  
  ### Answer by username_2 on 2019-12-11. Score: 2
  
  *I will only focus on the VAE because I am more familiar with it, but the explanations may also apply to the GAN and other generative models.*
  
  In the case of the VAE, you train a neural network not only to generate images but to represent them **compactly** in a so-called **latent space**, so you train the VAE to do **dimensionality reduction**. More precisely, the VAE attempts to learn a probability distribution with smaller dimensionality than the dimensionality of the training data but that hopefully represents the training data. Consequently, the model is forced to learn the essential features of the probability distribution that generated the training data.
  
  The VAE is a [generative model](https://en.wikipedia.org/wiki/Generative_model) rather than a [discriminative model](https://en.wikipedia.org/wiki/Discriminative_model). In the case of the neural network that maps inputs to images, you will not be learning a latent probability distribution (unless you formulate your model in such a way), from which you could sample, but you would just be mapping, in a supervised way (that is, you would need a labelled training dataset) and deterministically, the inputs to the images. In the VAE, there's some form of stochasticity, while training (and testing) the model.
  
  > **Comments:**
  > **username_2**: I will probably improve this answer later!  
  
  ---
  
  