Answers.Body: <p>Normally in synthetic data generation using VAEs, you do not use the encoder anymore upon inference. Instead, generating samples after training is usually done by simply sampling random noise from 'a distribution' (i.e. a normal distribution) in the dimensions of the latent space and decoding it using the decoder. Hence, generating samples would be usually done as follows:</p>
  <pre><code>noise = tf.random.normal(shape=(100, latent_dim))
  generated_samples = decoder(noise)
  </code></pre>
  <p>if you use training samples in generating your new data, it is not representative of how good your generative model is as it then bases the output on the training data.</p>
  <p><strong>Answer</strong> However, if you do want to sample using the training data to get the means and standard deviations, then your second method is a more appropriate method than your first method. You need the standard deviation, and you only get the <code>log(std)</code> from the model. Taking the exponent gives you the standard deviation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T11:00:38.903Z
Answers.Text: Why is the second method more appropriate ? Could you please elaborate
Answers.UserDisplayName: quant
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T11:15:10.597Z
Answers.Text: Because you need to convert your `log_sigma` which is equal in function to `log(std)`, to `sigma` (equal to `std`). You can do so by taking the exponent.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T11:26:09.337Z
Answers.Text: In the first method there is no need to take the exponent because it calculates directly the `std` from $z$, so in terms of "rescaling" these methods are equivalent. The question refers to "should I use the `z_log_sigma` directly (after rescaling it using the `exp` ofc) or should I infer the std from $z$ ?
Answers.UserDisplayName: quant
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T12:57:15.270Z
Answers.Text: My bad, I misread your code. The more i look at it the more confused i get too be honest. So first a question: are you trying to generate only one class? or multiple classes? Having the same mean and standard deviation for sampling noise for all classes is not logical. For sampling one class, you would have to use a different mean and standard deviation than for sampling another class
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T14:11:14.817Z
Answers.Text: I have one class
Answers.UserDisplayName: quant
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T14:31:57.237Z
Answers.Text: How much do the calculations differ between the two options? In my expectation, they shouldnt differ much? Though you might want to calculate the std sample wise and take the mean, instead of calculating the std over the whole batch.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T14:50:48.827Z
Answers.Text: They dont differ much indeed, as expected. Still, it is a matter of completeness
Answers.UserDisplayName: quant
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-23T10:52:49.647Z
Answers.LastActivityDate: 2023-01-23T10:52:49.647Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>I have built the following function which takes as input some data and runs a VAE on them:</p>
  <pre><code>def VAE(data, original_dim, latent_dim, test_size, epochs):
      
      x_train, x_test = train_test_split(data, test_size=test_size, random_state=42)
      
      # Define the VAE architecture
      #Encoder
      encoder_inputs = tf.keras.Input(shape=(original_dim,))
      x = layers.Dense(64, activation='relu')(encoder_inputs)
      x = layers.Dense(32, activation='relu')(x)
      x = layers.Dense(8, activation='relu')(x)
  
      #--- Custom Latent Space Layer
      z_mean = layers.Dense(units=latent_dim, name='Z-Mean', activation='linear')(x)
      z_log_sigma = layers.Dense(units=latent_dim, name='Z-Log-Sigma', activation='linear')(x)
      z = layers.Lambda(sampling, name='Z-Sampling-Layer')([z_mean, z_log_sigma, latent_dim]) # Z sampling layer
  
      # Instantiate the encoder
      encoder = tf.keras.Model(encoder_inputs, [z_mean, z_log_sigma, z], name='encoder')
  
      #Decoder
      latent_inputs = tf.keras.Input(shape=(latent_dim,))
      x = layers.Dense(8, activation='relu')(latent_inputs)
      x = layers.Dense(32, activation='relu')(x)
      x = layers.Dense(64, activation='relu')(x)
      decoder_outputs = layers.Dense(1, activation='relu')(x)
  
      # Instantiate the decoder
      decoder = tf.keras.Model(latent_inputs, decoder_outputs, name='decoder')
  
      # Define outputs from a VAE model by specifying how the encoder-decoder models are linked
      # Instantiate a VAE model
      vae = tf.keras.Model(inputs=encoder_inputs, outputs=decoder(encoder(encoder_inputs)[2]), name='vae')
      
      # Reconstruction loss compares inputs and outputs and tries to minimise the difference
      r_loss = original_dim * tf.keras.losses.mse(encoder_inputs, decoder(encoder(encoder_inputs)[2]))  # use MSE
  
      # KL divergence loss compares the encoded latent distribution Z with standard Normal distribution and penalizes if it's too different
      kl_loss = -0.5 * K.mean(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)
  
      #VAE total loss
      vae_loss = K.mean(r_loss + kl_loss)
  
      # Add loss to the model and compile it
      vae.add_loss(vae_loss)
      vae.compile(optimizer='adam')
      
      # train the model
      vae.fit(x_train, x_train, epochs=epochs, validation_data=(x_test, x_test))
  </code></pre>
  <p>where</p>
  <pre><code>def sampling(args):
      
      z_mean, z_log_sigma, latent_dim = args
      epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0., stddev=1., seed=42)
      return z_mean + K.exp(z_log_sigma) * epsilon
  </code></pre>
  <p>My question is, if I want to generate new data, by using the above VAE how can I achieve that ?</p>
  <p>If I want to sample 100 new data, should I use this</p>
  <pre><code>   latent_mean = tf.math.reduce_mean(encoder(x_train)[2], axis=0) 
     latent_std = tf.math.reduce_std(encoder(x_train)[2], axis=0)
     latent_sample = tf.random.normal(shape=(100, latent_dim), mean=latent_mean, 
                                      stddev=latent_std)
     generated_data = decoder(latent_sample)
  </code></pre>
  <p>or</p>
  <pre><code>   latent_mean = tf.math.reduce_mean(encoder(x_train)[0], axis=0) 
     latent_std = tf.math.reduce_mean(tf.math.exp(encoder(x_train))[1], axis=0)
     latent_sample = tf.random.normal(shape=(100, latent_dim), mean=latent_mean, 
                                      stddev=latent_std)
     generated_data = decoder(latent_sample)
  </code></pre>
  <p>?</p>
  <p>Basically should I infer <code>z_mean</code> and <code>z_log_sigma</code> from the <code>z</code> or should I use <code>z_mean</code> and <code>z_log_sigma</code> directly ? What is the difference ?</p>
  <p>Moreover, I have seen that everytime <code>tf.random.normal</code> is used to generate new data from the latent space. Why not use lognormal for instance ? Is it because of the KL divergence ?</p>
  <p>The end-goal is the distribution of the <code>generated_data</code> to be as close as possible to the distribution of the original <code>data</code>.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-23T10:42:04.300Z
Comments.Text: Why would you use log-normal? that's not the distribution you've learnt the parameters for, as far as I am aware.
Comments.UserDisplayName: David
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-23T10:46:41.303Z
Comments.Text: @DavidIreland As far as I understand, you can generate anything, since whatever you generate it will later pass from the decoder, and passing from the decoder does not necessarily mean that the distribution from which you sampled from will be preserved, right ? Or am I missing something ?
Comments.UserDisplayName: quant
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-23T11:41:40.093Z
Comments.Text: Once you have trained the VAE, you can discard the encoder and feed data directly in.  You seem to know this however in your case it means that the z_mean and z_log_variance can be discorded since you are only generating a sample latent vector.  This should be centered on zero, so you can simply generate vectors based on a normal curve.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-23T14:06:11.863Z
Comments.Text: @DavidHoelzer what is the purpose then of the z_mean and z_log_variance ?
Comments.UserDisplayName: quant
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T02:29:55.107Z
Comments.Text: The expectation is that those two layers model the distribution, which you are sampling with the reparameterization trick using your z layer.
Comments.UserDisplayName: David Hoelzer
Tags: <deep-learning><python><autoencoders><generative-model><variational-autoencoder>
Title: How to generate new data using VAE?
UserDisplayName: quant
ThreadText: # How to generate new data using VAE?
  
  (Asked by: username_0 on 2023-01-22)
  
  I have built the following function which takes as input some data and runs a VAE on them:
  
  ```
  def VAE(data, original_dim, latent_dim, test_size, epochs):
      
      x_train, x_test = train_test_split(data, test_size=test_size, random_state=42)
      
      # Define the VAE architecture
      #Encoder
      encoder_inputs = tf.keras.Input(shape=(original_dim,))
      x = layers.Dense(64, activation='relu')(encoder_inputs)
      x = layers.Dense(32, activation='relu')(x)
      x = layers.Dense(8, activation='relu')(x)
  
      #--- Custom Latent Space Layer
      z_mean = layers.Dense(units=latent_dim, name='Z-Mean', activation='linear')(x)
      z_log_sigma = layers.Dense(units=latent_dim, name='Z-Log-Sigma', activation='linear')(x)
      z = layers.Lambda(sampling, name='Z-Sampling-Layer')([z_mean, z_log_sigma, latent_dim]) # Z sampling layer
  
      # Instantiate the encoder
      encoder = tf.keras.Model(encoder_inputs, [z_mean, z_log_sigma, z], name='encoder')
  
      #Decoder
      latent_inputs = tf.keras.Input(shape=(latent_dim,))
      x = layers.Dense(8, activation='relu')(latent_inputs)
      x = layers.Dense(32, activation='relu')(x)
      x = layers.Dense(64, activation='relu')(x)
      decoder_outputs = layers.Dense(1, activation='relu')(x)
  
      # Instantiate the decoder
      decoder = tf.keras.Model(latent_inputs, decoder_outputs, name='decoder')
  
      # Define outputs from a VAE model by specifying how the encoder-decoder models are linked
      # Instantiate a VAE model
      vae = tf.keras.Model(inputs=encoder_inputs, outputs=decoder(encoder(encoder_inputs)[2]), name='vae')
      
      # Reconstruction loss compares inputs and outputs and tries to minimise the difference
      r_loss = original_dim * tf.keras.losses.mse(encoder_inputs, decoder(encoder(encoder_inputs)[2]))  # use MSE
  
      # KL divergence loss compares the encoded latent distribution Z with standard Normal distribution and penalizes if it's too different
      kl_loss = -0.5 * K.mean(1 + z_log_sigma - K.square(z_mean) - K.exp(z_log_sigma), axis=-1)
  
      #VAE total loss
      vae_loss = K.mean(r_loss + kl_loss)
  
      # Add loss to the model and compile it
      vae.add_loss(vae_loss)
      vae.compile(optimizer='adam')
      
      # train the model
      vae.fit(x_train, x_train, epochs=epochs, validation_data=(x_test, x_test))
  
  ```
  
  where
  
  ```
  def sampling(args):
      
      z_mean, z_log_sigma, latent_dim = args
      epsilon = K.random_normal(shape=(K.shape(z_mean)[0], latent_dim), mean=0., stddev=1., seed=42)
      return z_mean + K.exp(z_log_sigma) * epsilon
  
  ```
  
  My question is, if I want to generate new data, by using the above VAE how can I achieve that ?
  
  If I want to sample 100 new data, should I use this
  
  ```
     latent_mean = tf.math.reduce_mean(encoder(x_train)[2], axis=0) 
     latent_std = tf.math.reduce_std(encoder(x_train)[2], axis=0)
     latent_sample = tf.random.normal(shape=(100, latent_dim), mean=latent_mean, 
                                      stddev=latent_std)
     generated_data = decoder(latent_sample)
  
  ```
  
  or
  
  ```
     latent_mean = tf.math.reduce_mean(encoder(x_train)[0], axis=0) 
     latent_std = tf.math.reduce_mean(tf.math.exp(encoder(x_train))[1], axis=0)
     latent_sample = tf.random.normal(shape=(100, latent_dim), mean=latent_mean, 
                                      stddev=latent_std)
     generated_data = decoder(latent_sample)
  
  ```
  
  ?
  
  Basically should I infer `z_mean` and `z_log_sigma` from the `z` or should I use `z_mean` and `z_log_sigma` directly ? What is the difference ?
  
  Moreover, I have seen that everytime `tf.random.normal` is used to generate new data from the latent space. Why not use lognormal for instance ? Is it because of the KL divergence ?
  
  The end-goal is the distribution of the `generated_data` to be as close as possible to the distribution of the original `data`.
  
  **Tags:** deep-learning, python, autoencoders, generative-model, variational-autoencoder
  
  > **Comments:**
  > **username_1**: Why would you use log-normal? that's not the distribution you've learnt the parameters for, as far as I am aware.  
  > **username_0**: @DavidIreland As far as I understand, you can generate anything, since whatever you generate it will later pass from the decoder, and passing from the decoder does not necessarily mean that the distribution from which you sampled from will be preserved, right ? Or am I missing something ?  
  > **username_2**: Once you have trained the VAE, you can discard the encoder and feed data directly in.  You seem to know this however in your case it means that the z_mean and z_log_variance can be discorded since you are only generating a sample latent vector.  This should be centered on zero, so you can simply generate vectors based on a normal curve.  
  > **username_0**: @username_2 what is the purpose then of the z_mean and z_log_variance ?  
  > **username_2**: The expectation is that those two layers model the distribution, which you are sampling with the reparameterization trick using your z layer.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_3 on 2023-01-23. Score: 2
  
  Normally in synthetic data generation using VAEs, you do not use the encoder anymore upon inference. Instead, generating samples after training is usually done by simply sampling random noise from 'a distribution' (i.e. a normal distribution) in the dimensions of the latent space and decoding it using the decoder. Hence, generating samples would be usually done as follows:
  
  ```
  noise = tf.random.normal(shape=(100, latent_dim))
  generated_samples = decoder(noise)
  
  ```
  
  if you use training samples in generating your new data, it is not representative of how good your generative model is as it then bases the output on the training data.
  
  **Answer** However, if you do want to sample using the training data to get the means and standard deviations, then your second method is a more appropriate method than your first method. You need the standard deviation, and you only get the `log(std)` from the model. Taking the exponent gives you the standard deviation.
  
  > **Comments:**
  > **username_0**: Why is the second method more appropriate ? Could you please elaborate  
  > **username_3**: Because you need to convert your `log_sigma` which is equal in function to `log(std)`, to `sigma` (equal to `std`). You can do so by taking the exponent.  
  > **username_0**: In the first method there is no need to take the exponent because it calculates directly the `std` from $z$, so in terms of "rescaling" these methods are equivalent. The question refers to "should I use the `z_log_sigma` directly (after rescaling it using the `exp` ofc) or should I infer the std from $z$ ?  
  > **username_3**: My bad, I misread your code. The more i look at it the more confused i get too be honest. So first a question: are you trying to generate only one class? or multiple classes? Having the same mean and standard deviation for sampling noise for all classes is not logical. For sampling one class, you would have to use a different mean and standard deviation than for sampling another class  
  > **username_0**: I have one class  
  > **username_3**: How much do the calculations differ between the two options? In my expectation, they shouldnt differ much? Though you might want to calculate the std sample wise and take the mean, instead of calculating the std over the whole batch.  
  > **username_0**: They dont differ much indeed, as expected. Still, it is a matter of completeness  
  
  ---
  
  