Answers.Body: <p>In general, very simple datasets like MNIST and Fashion-MNIST can be encoded in just two dimensions, without sacrificing too much reconstruction quality.</p>
  <p>For more complex data, this is often neither feasible nor advisable: even if you employ a very powerful encoder and decoder, the bottleneck is just too small to capture the relevant features (or hidden factors) that sufficiently describe your data. Usually, it happens that a very low bottleneck dimensionality can only capture something that is very simple and, therefore, also not very useful, which makes the reconstructions looks like random stuff or even just spurious patterns that are somewhat weakly correlated with your inputs.</p>
  <p>I think the best way, is to apply <em>dimensionality reduction</em> methods (like UMAP or t-SNE) to a trained bottleneck to reduce its size to 2 or 3 dimensions such that you can inspect it nicely. As stated <a href="http://distill.pub/2016/misread-tsne" rel="nofollow noreferrer">here</a>, tSNE can behave quite arbitrarily due to its very random nature (and difficuly to tune the hyperparameters), and so you may end up visualizing some patters that, actually, are not in the data. For such reason I suggest using UMAP first, and recur to t-SNE only if you're not able to get good results.</p>
  <p>I can also suggest that thinks like l2-normalization (during training) of the embeddings may help the subsequent dimensionality reduction.</p>
  <p>Otherwise, a good way to measure similarity (or closeness) between samples is to train a Siamese network and then threshold the learned distance metric: have a look <a href="https://keras.io/examples/vision/siamese_contrastive/" rel="nofollow noreferrer">here</a>. You can also use the triplet loss, or simply use a pre-trained VGG network in which you extract the 4096-d embedding.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-12T16:02:57.483Z
Answers.LastActivityDate: 2023-11-12T16:02:57.483Z
Answers.UserDisplayName: Luca Anzalone
Body: <p>I want to do a project with a small size image dataset (the size is about 50*50). There's another similar dataset, and I want to prove that the datasets are different. I built a convolutional autoencoder with fully connected bottleneck because I want to use the latent space which produced by encoder to get a scatter plot. I think if the scatter points are not in the same area, then it can said the similarity between the two datasets is not that high.</p>
  <p>I tried many model architectures, and it just can’t perform well on validation data. I am wondering if there is any way to reduce the dimension to such a low level. Thanks!</p>
  <p>Here's my model:</p>
  <pre><code>    self.conv1 = nn.Conv2d(1, 32, (2,2), 1,padding=1)
      self.conv2 = nn.Conv2d(32, 64, (2,2), 1,padding=1)
      self.conv3 = nn.Conv2d(64, 128, (2,2), 1,padding=1)
      self.conv4 = nn.Conv2d(128,256, (2,2), 1,padding=1)
      self.conv5 = nn.Conv2d(256, 128, (2,2), 1,padding=1)
      self.conv6 = nn.Conv2d(128, 32, (2,2), 1,padding=1)
      self.conv7 = nn.Conv2d(32, 8, (2,2), 2,padding=1)
      self.Flatten = nn.Flatten()
      self.lin1 = nn.Linear(8*30*30,512)
      self.lin2 = nn.Linear(512,128)
      self.lin3 = nn.Linear(128,10)
      
      self.lin4 = nn.Linear(10,128)
      self.lin5 = nn.Linear(128,512)
      self.lin6 = nn.Linear(512,8*30*30)
      #reshape here
      self.convtrans1 = nn.ConvTranspose2d(8, 32, (2,2), 2,padding=1)
      self.convtrans2 = nn.ConvTranspose2d(32, 128, (2,2), 1,padding=1)
      self.convtrans3 = nn.ConvTranspose2d(128, 256, (2,2), 1,padding=1)
      self.convtrans4 = nn.ConvTranspose2d(256, 128, (2,2), 1,padding=1)
      self.convtrans5 = nn.ConvTranspose2d(128, 64, (2,2), 1,padding=1)
      self.convtrans6 = nn.ConvTranspose2d(64, 32, (2,2), 1,padding=1)
      self.convtrans7 = nn.ConvTranspose2d(32, 1, (2,2), 1,padding=1)
  </code></pre>
  
Tags: <convolutional-neural-networks><computer-vision><autoencoders>
Title: Is it possible to build a convolutional autoencoder with fully connected bottleneck with low dimension?
UserDisplayName: Kekai
ThreadText: # Is it possible to build a convolutional autoencoder with fully connected bottleneck with low dimension?
  
  (Asked by: username_0 on 2023-11-12)
  
  I want to do a project with a small size image dataset (the size is about 50\*50). There's another similar dataset, and I want to prove that the datasets are different. I built a convolutional autoencoder with fully connected bottleneck because I want to use the latent space which produced by encoder to get a scatter plot. I think if the scatter points are not in the same area, then it can said the similarity between the two datasets is not that high.
  
  I tried many model architectures, and it just can’t perform well on validation data. I am wondering if there is any way to reduce the dimension to such a low level. Thanks!
  
  Here's my model:
  
  ```
      self.conv1 = nn.Conv2d(1, 32, (2,2), 1,padding=1)
      self.conv2 = nn.Conv2d(32, 64, (2,2), 1,padding=1)
      self.conv3 = nn.Conv2d(64, 128, (2,2), 1,padding=1)
      self.conv4 = nn.Conv2d(128,256, (2,2), 1,padding=1)
      self.conv5 = nn.Conv2d(256, 128, (2,2), 1,padding=1)
      self.conv6 = nn.Conv2d(128, 32, (2,2), 1,padding=1)
      self.conv7 = nn.Conv2d(32, 8, (2,2), 2,padding=1)
      self.Flatten = nn.Flatten()
      self.lin1 = nn.Linear(8*30*30,512)
      self.lin2 = nn.Linear(512,128)
      self.lin3 = nn.Linear(128,10)
      
      self.lin4 = nn.Linear(10,128)
      self.lin5 = nn.Linear(128,512)
      self.lin6 = nn.Linear(512,8*30*30)
      #reshape here
      self.convtrans1 = nn.ConvTranspose2d(8, 32, (2,2), 2,padding=1)
      self.convtrans2 = nn.ConvTranspose2d(32, 128, (2,2), 1,padding=1)
      self.convtrans3 = nn.ConvTranspose2d(128, 256, (2,2), 1,padding=1)
      self.convtrans4 = nn.ConvTranspose2d(256, 128, (2,2), 1,padding=1)
      self.convtrans5 = nn.ConvTranspose2d(128, 64, (2,2), 1,padding=1)
      self.convtrans6 = nn.ConvTranspose2d(64, 32, (2,2), 1,padding=1)
      self.convtrans7 = nn.ConvTranspose2d(32, 1, (2,2), 1,padding=1)
  
  ```
  
  **Tags:** convolutional-neural-networks, computer-vision, autoencoders
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-12. Score: 2
  
  In general, very simple datasets like MNIST and Fashion-MNIST can be encoded in just two dimensions, without sacrificing too much reconstruction quality.
  
  For more complex data, this is often neither feasible nor advisable: even if you employ a very powerful encoder and decoder, the bottleneck is just too small to capture the relevant features (or hidden factors) that sufficiently describe your data. Usually, it happens that a very low bottleneck dimensionality can only capture something that is very simple and, therefore, also not very useful, which makes the reconstructions looks like random stuff or even just spurious patterns that are somewhat weakly correlated with your inputs.
  
  I think the best way, is to apply *dimensionality reduction* methods (like UMAP or t-SNE) to a trained bottleneck to reduce its size to 2 or 3 dimensions such that you can inspect it nicely. As stated [here](http://distill.pub/2016/misread-tsne), tSNE can behave quite arbitrarily due to its very random nature (and difficuly to tune the hyperparameters), and so you may end up visualizing some patters that, actually, are not in the data. For such reason I suggest using UMAP first, and recur to t-SNE only if you're not able to get good results.
  
  I can also suggest that thinks like l2-normalization (during training) of the embeddings may help the subsequent dimensionality reduction.
  
  Otherwise, a good way to measure similarity (or closeness) between samples is to train a Siamese network and then threshold the learned distance metric: have a look [here](https://keras.io/examples/vision/siamese_contrastive/). You can also use the triplet loss, or simply use a pre-trained VGG network in which you extract the 4096-d embedding.
  
  
  ---
  
  