Answers.Body: <p>I have found a mistake that can explain poor performance.</p>
  <p>In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.</p>
  <p>Removing the normalization dramatically increases the performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-01T06:31:08.100Z
Answers.LastActivityDate: 2021-09-01T06:31:08.100Z
Answers.UserDisplayName: Alex
Body: <p>I am trying to implement a toy VAE project.
  My goal is to use a VAE to model the moon dataset from scikit-learn, with an extra constant (but noisy) z-dimension.</p>
  <p>To this end I use an approximate posterior with the form of a beta distribution and a uniform prior in a 1D latent space, because essentially the data is 1D. The decoder is a NN-parameterized gaussian.</p>
  <p>I cannot get it to work using the simple ELBO.</p>
  <p>I tried so far :</p>
  <ul>
  <li>Increasing the number of monte carlo samples in the SGVB</li>
  <li>Various deterministic pretrainings which tend to raise nans</li>
  <li>Increasing the width or depth of the networks</li>
  <li>Gradient clipping</li>
  <li>learning rate annealing</li>
  <li>Remove the noise in the data and perform Batch gradient descent instead of mini-batch</li>
  <li>...</li>
  </ul>
  <p>I use layers of residual blocks with Tanh nonlinearities, whose outputs are <span class="math-container">$\log \alpha$</span> and <span class="math-container">$\log \beta$</span> for the encoder, <span class="math-container">$\mu$</span> and <span class="math-container">$\log \sigma$</span> for the decoder.</p>
  <p>I am starting to wonder whether the distribution is actually hard to model, because I ran out of bugs to fix and strategies to improve training.</p>
  <p>Are some low dimensional distributions known to be hard to model this way ?</p>
  <p>Additionally, what obvious or non obvious mistakes could I have made ?</p>
  <p><strong>ADDENDA</strong></p>
  <p>Code to generate the data:</p>
  <pre><code># Adapted from sklearn.dataset.make_moons
  
  def make_moons(n_samples=100, noise=None):
      generator = default_rng()
  
      n_samples_out = n_samples // 2
      n_samples_in = n_samples - n_samples_out
  
      outer_circ_x = np.cos(np.linspace(0, np.pi, n_samples_out))
      outer_circ_y = np.sin(np.linspace(0, np.pi, n_samples_out))
      inner_circ_x = 1 - np.cos(np.linspace(0, np.pi, n_samples_in))
      inner_circ_y = 1 - np.sin(np.linspace(0, np.pi, n_samples_in)) - .5
  
      X = np.vstack([np.append(outer_circ_x, inner_circ_x),
                     np.append(outer_circ_y, inner_circ_y),
                     np.zeros(n_samples)]).T
      y = np.hstack([np.zeros(n_samples_out, dtype=np.intp),
                     np.ones(n_samples_in, dtype=np.intp)])
  
      if noise is not None:
          X += generator.multivariate_normal(np.zeros(3), np.diag([noise, noise, noise])**2, size=n_samples)
  
      return X, y
  
  # create dataset
  moon_coordinates, moon_labels = make_moons(n_samples=500, noise=.01)
  moon_coordinates = moon_coordinates.astype(np.float32)
  moon_labels = moon_labels.astype(np.float32)
  
  # normalize dataset
  moon_coordinates = (moon_coordinates-moon_coordinates.mean(axis=0))/np.std(moon_coordinates, axis=0)
  </code></pre>
  <p><strong>UPDATE</strong></p>
  <p>I have found a mistake that can explain poor performance.</p>
  <p>In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.</p>
  <p>Removing the normalization dramatically increases the performance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T15:45:03.343Z
Comments.Text: Could you please link the Moon Dataset? Is this what you are referring to? https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html
Comments.UserDisplayName: Andre Goulart
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T17:19:26.327Z
Comments.Text: Yes. I have updated the question to provide the code.
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-05T00:08:53.780Z
Comments.Text: I am trying to model this using an MSE loss, in that case I do not have to standardize it. However, my model is not learning beyond a point with VAE. My Reconstruction loss staturates and somehow when sampling after training, the output is not capturing the shape of moons dataset and instead a blob. Can you share some info on the architecture of your model.
Comments.UserDisplayName: hsuya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-31T07:19:55.880Z
Comments.Text: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space?
Comments.UserDisplayName: pedagogue1
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-01T07:43:51.260Z
Comments.Text: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space? Please don't delete my comment I am new here and looking to add points.
Comments.UserDisplayName: pedagogue1
Tags: <variational-autoencoder><bayesian-deep-learning><evidence-lower-bound>
Title: Are some low dimensional distributions known to be hard to model with VAEs?
UserDisplayName: Alex
ThreadText: # Are some low dimensional distributions known to be hard to model with VAEs?
  
  (Asked by: username_0 on 2021-08-31)
  
  I am trying to implement a toy VAE project.
  My goal is to use a VAE to model the moon dataset from scikit-learn, with an extra constant (but noisy) z-dimension.
  To this end I use an approximate posterior with the form of a beta distribution and a uniform prior in a 1D latent space, because essentially the data is 1D. The decoder is a NN-parameterized gaussian.
  I cannot get it to work using the simple ELBO.
  I tried so far :
  
  Increasing the number of monte carlo samples in the SGVB
  Various deterministic pretrainings which tend to raise nans
  Increasing the width or depth of the networks
  Gradient clipping
  learning rate annealing
  Remove the noise in the data and perform Batch gradient descent instead of mini-batch
  ...
  
  I use layers of residual blocks with Tanh nonlinearities, whose outputs are $\log \alpha$ and $\log \beta$ for the encoder, $\mu$ and $\log \sigma$ for the decoder.
  I am starting to wonder whether the distribution is actually hard to model, because I ran out of bugs to fix and strategies to improve training.
  Are some low dimensional distributions known to be hard to model this way ?
  Additionally, what obvious or non obvious mistakes could I have made ?
  ADDENDA
  Code to generate the data:
  # Adapted from sklearn.dataset.make_moons
  
  def make_moons(n_samples=100, noise=None):
      generator = default_rng()
  
      n_samples_out = n_samples // 2
      n_samples_in = n_samples - n_samples_out
  
      outer_circ_x = np.cos(np.linspace(0, np.pi, n_samples_out))
      outer_circ_y = np.sin(np.linspace(0, np.pi, n_samples_out))
      inner_circ_x = 1 - np.cos(np.linspace(0, np.pi, n_samples_in))
      inner_circ_y = 1 - np.sin(np.linspace(0, np.pi, n_samples_in)) - .5
  
      X = np.vstack([np.append(outer_circ_x, inner_circ_x),
                     np.append(outer_circ_y, inner_circ_y),
                     np.zeros(n_samples)]).T
      y = np.hstack([np.zeros(n_samples_out, dtype=np.intp),
                     np.ones(n_samples_in, dtype=np.intp)])
  
      if noise is not None:
          X += generator.multivariate_normal(np.zeros(3), np.diag([noise, noise, noise])**2, size=n_samples)
  
      return X, y
  
  # create dataset
  moon_coordinates, moon_labels = make_moons(n_samples=500, noise=.01)
  moon_coordinates = moon_coordinates.astype(np.float32)
  moon_labels = moon_labels.astype(np.float32)
  
  # normalize dataset
  moon_coordinates = (moon_coordinates-moon_coordinates.mean(axis=0))/np.std(moon_coordinates, axis=0)
  
  UPDATE
  I have found a mistake that can explain poor performance.
  In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.
  Removing the normalization dramatically increases the performance.
  
  **Tags:** variational-autoencoder, bayesian-deep-learning, evidence-lower-bound
  > **Comments:**
  > **username_1**: Could you please link the Moon Dataset? Is this what you are referring to? https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html  
  > **username_0**: Yes. I have updated the question to provide the code.  
  > **username_2**: I am trying to model this using an MSE loss, in that case I do not have to standardize it. However, my model is not learning beyond a point with VAE. My Reconstruction loss staturates and somehow when sampling after training, the output is not capturing the shape of moons dataset and instead a blob. Can you share some info on the architecture of your model.  
  > **username_3**: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space?  
  > **username_3**: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space? Please don't delete my comment I am new here and looking to add points.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-09-01. Score: 0 | Accepted answer
  
  I have found a mistake that can explain poor performance.
  
  In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.
  
  Removing the normalization dramatically increases the performance.
  
  
  ---
  
  