Answers.Body: <p>I have found a mistake that can explain poor performance.</p>
  <p>In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.</p>
  <p>Removing the normalization dramatically increases the performance.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-09-01T06:31:08.100Z
Answers.LastActivityDate: 2021-09-01T06:31:08.100Z
Answers.UserDisplayName: Alex
Body: <p>I am trying to implement a toy VAE project.
  My goal is to use a VAE to model the moon dataset from scikit-learn, with an extra constant (but noisy) z-dimension.</p>
  <p>To this end I use an approximate posterior with the form of a beta distribution and a uniform prior in a 1D latent space, because essentially the data is 1D. The decoder is a NN-parameterized gaussian.</p>
  <p>I cannot get it to work using the simple ELBO.</p>
  <p>I tried so far :</p>
  <ul>
  <li>Increasing the number of monte carlo samples in the SGVB</li>
  <li>Various deterministic pretrainings which tend to raise nans</li>
  <li>Increasing the width or depth of the networks</li>
  <li>Gradient clipping</li>
  <li>learning rate annealing</li>
  <li>Remove the noise in the data and perform Batch gradient descent instead of mini-batch</li>
  <li>...</li>
  </ul>
  <p>I use layers of residual blocks with Tanh nonlinearities, whose outputs are <span class="math-container">$\log \alpha$</span> and <span class="math-container">$\log \beta$</span> for the encoder, <span class="math-container">$\mu$</span> and <span class="math-container">$\log \sigma$</span> for the decoder.</p>
  <p>I am starting to wonder whether the distribution is actually hard to model, because I ran out of bugs to fix and strategies to improve training.</p>
  <p>Are some low dimensional distributions known to be hard to model this way ?</p>
  <p>Additionally, what obvious or non obvious mistakes could I have made ?</p>
  <p><strong>ADDENDA</strong></p>
  <p>Code to generate the data:</p>
  <pre><code># Adapted from sklearn.dataset.make_moons
  
  def make_moons(n_samples=100, noise=None):
      generator = default_rng()
  
      n_samples_out = n_samples // 2
      n_samples_in = n_samples - n_samples_out
  
      outer_circ_x = np.cos(np.linspace(0, np.pi, n_samples_out))
      outer_circ_y = np.sin(np.linspace(0, np.pi, n_samples_out))
      inner_circ_x = 1 - np.cos(np.linspace(0, np.pi, n_samples_in))
      inner_circ_y = 1 - np.sin(np.linspace(0, np.pi, n_samples_in)) - .5
  
      X = np.vstack([np.append(outer_circ_x, inner_circ_x),
                     np.append(outer_circ_y, inner_circ_y),
                     np.zeros(n_samples)]).T
      y = np.hstack([np.zeros(n_samples_out, dtype=np.intp),
                     np.ones(n_samples_in, dtype=np.intp)])
  
      if noise is not None:
          X += generator.multivariate_normal(np.zeros(3), np.diag([noise, noise, noise])**2, size=n_samples)
  
      return X, y
  
  # create dataset
  moon_coordinates, moon_labels = make_moons(n_samples=500, noise=.01)
  moon_coordinates = moon_coordinates.astype(np.float32)
  moon_labels = moon_labels.astype(np.float32)
  
  # normalize dataset
  moon_coordinates = (moon_coordinates-moon_coordinates.mean(axis=0))/np.std(moon_coordinates, axis=0)
  </code></pre>
  <p><strong>UPDATE</strong></p>
  <p>I have found a mistake that can explain poor performance.</p>
  <p>In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.</p>
  <p>Removing the normalization dramatically increases the performance.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T15:45:03.343Z
Comments.Text: Could you please link the Moon Dataset? Is this what you are referring to? https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html
Comments.UserDisplayName: Andre Goulart
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-08-31T17:19:26.327Z
Comments.Text: Yes. I have updated the question to provide the code.
Comments.UserDisplayName: Alex
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-05T00:08:53.780Z
Comments.Text: I am trying to model this using an MSE loss, in that case I do not have to standardize it. However, my model is not learning beyond a point with VAE. My Reconstruction loss staturates and somehow when sampling after training, the output is not capturing the shape of moons dataset and instead a blob. Can you share some info on the architecture of your model.
Comments.UserDisplayName: hsuya
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-31T07:19:55.880Z
Comments.Text: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space?
Comments.UserDisplayName: pedagogue1
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-04-01T07:43:51.260Z
Comments.Text: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space? Please don't delete my comment I am new here and looking to add points.
Comments.UserDisplayName: pedagogue1
Tags: <variational-autoencoder><bayesian-deep-learning><evidence-lower-bound>
Title: Are some low dimensional distributions known to be hard to model with VAEs?
UserDisplayName: Alex
ThreadText: # Are some low dimensional distributions known to be hard to model with VAEs?
  
  (Asked by: username_0 on 2021-08-31)
  
  I am trying to implement a toy VAE project.
  My goal is to use a VAE to model the moon dataset from scikit-learn, with an extra constant (but noisy) z-dimension.
  
  To this end I use an approximate posterior with the form of a beta distribution and a uniform prior in a 1D latent space, because essentially the data is 1D. The decoder is a NN-parameterized gaussian.
  
  I cannot get it to work using the simple ELBO.
  
  I tried so far :
  
  * Increasing the number of monte carlo samples in the SGVB
  * Various deterministic pretrainings which tend to raise nans
  * Increasing the width or depth of the networks
  * Gradient clipping
  * learning rate annealing
  * Remove the noise in the data and perform Batch gradient descent instead of mini-batch
  * ...
  
  I use layers of residual blocks with Tanh nonlinearities, whose outputs are $\log \alpha$ and $\log \beta$ for the encoder, $\mu$ and $\log \sigma$ for the decoder.
  
  I am starting to wonder whether the distribution is actually hard to model, because I ran out of bugs to fix and strategies to improve training.
  
  Are some low dimensional distributions known to be hard to model this way ?
  
  Additionally, what obvious or non obvious mistakes could I have made ?
  
  **ADDENDA**
  
  Code to generate the data:
  
  ```
  # Adapted from sklearn.dataset.make_moons
  
  def make_moons(n_samples=100, noise=None):
      generator = default_rng()
  
      n_samples_out = n_samples // 2
      n_samples_in = n_samples - n_samples_out
  
      outer_circ_x = np.cos(np.linspace(0, np.pi, n_samples_out))
      outer_circ_y = np.sin(np.linspace(0, np.pi, n_samples_out))
      inner_circ_x = 1 - np.cos(np.linspace(0, np.pi, n_samples_in))
      inner_circ_y = 1 - np.sin(np.linspace(0, np.pi, n_samples_in)) - .5
  
      X = np.vstack([np.append(outer_circ_x, inner_circ_x),
                     np.append(outer_circ_y, inner_circ_y),
                     np.zeros(n_samples)]).T
      y = np.hstack([np.zeros(n_samples_out, dtype=np.intp),
                     np.ones(n_samples_in, dtype=np.intp)])
  
      if noise is not None:
          X += generator.multivariate_normal(np.zeros(3), np.diag([noise, noise, noise])**2, size=n_samples)
  
      return X, y
  
  # create dataset
  moon_coordinates, moon_labels = make_moons(n_samples=500, noise=.01)
  moon_coordinates = moon_coordinates.astype(np.float32)
  moon_labels = moon_labels.astype(np.float32)
  
  # normalize dataset
  moon_coordinates = (moon_coordinates-moon_coordinates.mean(axis=0))/np.std(moon_coordinates, axis=0)
  
  ```
  
  **UPDATE**
  
  I have found a mistake that can explain poor performance.
  
  In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.
  
  Removing the normalization dramatically increases the performance.
  
  **Tags:** variational-autoencoder, bayesian-deep-learning, evidence-lower-bound
  
  > **Comments:**
  > **username_1**: Could you please link the Moon Dataset? Is this what you are referring to? https://scikit-learn.org/stable/modules/generated/sklearn.datasets.make_moons.html  
  > **username_0**: Yes. I have updated the question to provide the code.  
  > **username_2**: I am trying to model this using an MSE loss, in that case I do not have to standardize it. However, my model is not learning beyond a point with VAE. My Reconstruction loss staturates and somehow when sampling after training, the output is not capturing the shape of moons dataset and instead a blob. Can you share some info on the architecture of your model.  
  > **username_3**: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space?  
  > **username_3**: did you ever figure out a good VAE implementation of the noisy moon dataset? I am trying to embed the same noisy moon dataset into latent space so it can follow a multivariate normal distribution. I think VAE does exactly that. So I googled and came across your post. However, I am not sure the 3rd axis makes sense. Do you go from 2D initial space to a 2D latent space? Please don't delete my comment I am new here and looking to add points.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2021-09-01. Score: 0 | Accepted answer
  
  I have found a mistake that can explain poor performance.
  
  In my post I said that the data is basically 1D, yet when I create the dataset I normalize the standard deviation in every dimension. This increases the magnitude of the z noise, and all of a sudden the third dimension accounts for a lot of variance and my model tries to fit to this noise.
  
  Removing the normalization dramatically increases the performance.
  
  
  ---
  
  