Answers.Body: <p>The thing is, the decoder samples from a latent mu and sigma, so you cant sample from a raw image directly. But if you’re trying to put a random image into the encoder of a trained VAE to match it to some sample image (via reconstruction loss), then your random input image will converge to the target sample.</p>
  
  <p>This will work when the following VAE architecture constraints are satisfied:</p>
  
  <ol>
  <li><p>The target sample is contained in the previously used training distribution.</p></li>
  <li><p>The parameters of the VAE are frozen after training.</p></li>
  <li><p>The input image values are “backpropagate-able”. (Interpret the input image as optimizable parameters.)</p></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-04T02:51:32.070Z
Answers.Text: Is it possible to leave the 1000 or so input image pixels unchanged? Is there a code sample I can look at? Thanks for the response. I am happy this is at least possible.
Answers.UserDisplayName: John Watts
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-04T17:37:43.397Z
Answers.Text: Np! With this specific architecture example its not possible to leave the input pixels unchanged, since they are the values that eventually will converge to some output target. Maybe I'm misunderstanding the question though. `main.py` in [this](https://github.com/arikanev/VAE_exps) repo, is sample code that trains a VAE on mnist, then freezes the encoder+decoder weights, then generates a random input that is updated to converge to a target from the mnist distribution.
Answers.UserDisplayName: Ari K
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-03T18:51:39.467Z
Answers.LastActivityDate: 2018-05-03T18:59:09.837Z
Answers.LastEditDate: 2018-05-03T18:59:09.837Z
Answers.UserDisplayName: Ari K
Answers.Body: <p>You could use VAE as previously answered though it will not work well in practice.<br>
  I think denoising auto-encoder is suitable for your problem because during training, the input is corrupted stochastically, thus it must learn to guess the distribution of the missing information (reconstruct the clean original input)<br>
  We could argue that VAE is better than DAE at modeling p(x) because of the randomness introduced at the latent space layer while DAE like algorithm keeps putting noise starting from the input layer.<br>
  suppose your data is concentrated on this 1-D curved manifold, what VAE could do is just pick some random value and output p(X|Z) which is Gaussian by the way, while DAE would learn to map a corrupted data point x˜ back to the original data point x.</p>
  
  <p><a href="https://i.sstatic.net/axc8Z.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/axc8Z.png" alt="enter image description here"></a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2018-05-04T07:32:14.870Z
Answers.LastActivityDate: 2018-05-04T07:32:14.870Z
Answers.UserDisplayName: Fadi Bakoura
Body: <p>Is it possible to use a VAE to reconstruct an image starting from an initial image instead of using <code>K.random_normal</code>, as shown in the “sampling” function of <a href="https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py" rel="nofollow noreferrer">this example</a>?</p>
  <p>I have used a sample image with the VAE encoder to get <code>z_mean</code> and <code>z_logvar</code>.</p>
  <p>I have been given 1000 pixels in an otherwise blank image (with nothing in it).</p>
  <p>Now, I want to reconstruct the sample image using the decoder with a given constraint that the 1000 pixels in the otherwise blank image remain the same. The remaining pixels can be reconstructed so they are as close to the initial sample image as possible. In other words, my starting point for the decoder is a blank image with some pixels that don’t change.</p>
  <p>How can I modify the decoder to generate an image based on this constraint? Is it possible? Are there variations of VAE that might make this possible? So we can predict the latent variables by starting from an initial point(s)?</p>
  
Tags: <neural-networks><deep-learning><computer-vision><variational-autoencoder><constrained-optimization>
Title: How to use a VAE to reconstruct an image starting from an initial image instead of starting from a random vector?
UserDisplayName: John Watts
ThreadText: # How to use a VAE to reconstruct an image starting from an initial image instead of starting from a random vector?
  
  (Asked by: username_0 on 2018-05-03)
  
  Is it possible to use a VAE to reconstruct an image starting from an initial image instead of using `K.random_normal`, as shown in the “sampling” function of [this example](https://github.com/keras-team/keras/blob/master/examples/variational_autoencoder.py)?
  
  I have used a sample image with the VAE encoder to get `z_mean` and `z_logvar`.
  
  I have been given 1000 pixels in an otherwise blank image (with nothing in it).
  
  Now, I want to reconstruct the sample image using the decoder with a given constraint that the 1000 pixels in the otherwise blank image remain the same. The remaining pixels can be reconstructed so they are as close to the initial sample image as possible. In other words, my starting point for the decoder is a blank image with some pixels that don’t change.
  
  How can I modify the decoder to generate an image based on this constraint? Is it possible? Are there variations of VAE that might make this possible? So we can predict the latent variables by starting from an initial point(s)?
  
  **Tags:** neural-networks, deep-learning, computer-vision, variational-autoencoder, constrained-optimization
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2018-05-03. Score: 1
  
  The thing is, the decoder samples from a latent mu and sigma, so you cant sample from a raw image directly. But if you’re trying to put a random image into the encoder of a trained VAE to match it to some sample image (via reconstruction loss), then your random input image will converge to the target sample.
  
  This will work when the following VAE architecture constraints are satisfied:
  
  1. The target sample is contained in the previously used training distribution.
  2. The parameters of the VAE are frozen after training.
  3. The input image values are “backpropagate-able”. (Interpret the input image as optimizable parameters.)
  
  > **Comments:**
  > **username_0**: Is it possible to leave the 1000 or so input image pixels unchanged? Is there a code sample I can look at? Thanks for the response. I am happy this is at least possible.  
  > **username_1**: Np! With this specific architecture example its not possible to leave the input pixels unchanged, since they are the values that eventually will converge to some output target. Maybe I'm misunderstanding the question though. `main.py` in [this](https://github.com/arikanev/VAE_exps) repo, is sample code that trains a VAE on mnist, then freezes the encoder+decoder weights, then generates a random input that is updated to converge to a target from the mnist distribution.  
  
  ---
  
  ### Answer by username_2 on 2018-05-04. Score: 0
  
  You could use VAE as previously answered though it will not work well in practice.  
  I think denoising auto-encoder is suitable for your problem because during training, the input is corrupted stochastically, thus it must learn to guess the distribution of the missing information (reconstruct the clean original input)  
  We could argue that VAE is better than DAE at modeling p(x) because of the randomness introduced at the latent space layer while DAE like algorithm keeps putting noise starting from the input layer.  
  suppose your data is concentrated on this 1-D curved manifold, what VAE could do is just pick some random value and output p(X|Z) which is Gaussian by the way, while DAE would learn to map a corrupted data point x˜ back to the original data point x.
  
  [![enter image description here](https://i.sstatic.net/axc8Z.png)](https://i.sstatic.net/axc8Z.png)
  
  
  ---
  
  