Answers.Body: <p>I found out pytorch backpropagates to the beginning of your model and also trains the embeddings.
  The sources I linked do it the right way.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-26T17:20:46.630Z
Answers.LastActivityDate: 2023-06-26T17:20:46.630Z
Answers.UserDisplayName: Jaanis
Body: <p>I want to cluster mixed-type tabular data, for the categorial columns I want to use Categorial Embeddings and then an Autoencoder Network before clustering with KMeans or similar.</p>
  <p>Now, when I want to train my model, the Batch will be fed into the Embeddings Layers and then the Autoencoder, which will then reconstruct the Embeddings. My Question is, what value should be passed into the loss function? Since the Decoder reconstructs the embedded columns, I cant pass the original categorial values.</p>
  <p>Solutions I found <a href="https://github.com/chrislemke/autoembedder/blob/main/src/autoembedder/model.py" rel="nofollow noreferrer">here</a> and <a href="https://medium.com/kirey-group/autoembedder-training-embedding-layers-on-unsupervised-tasks-fc364c0f6eec" rel="nofollow noreferrer">here</a> propose to save the Input Batch as a in a variable after feeding it into the Embeddings Layer (but before the AE) and use that as the target for the loss function.</p>
  <p>I am a ML/DL beginner, but this sounds fishy to me, because the Embeddings will not be trained by gradient descent. I tested this approach on a small tabular dataset vs. just feeding the categorial data into the AE (no Embeddings) and found that using the first approach (saving embedded cols as variable) to moderatly degrade Clustering Accuracy and NMI Score (This is not representative - just a small test on a small dataset). <a href="https://github.com/jaanisfehling/attention-mixed-type-clustering/blob/main/embedding_heart_disease.ipynb" rel="nofollow noreferrer">Here</a> is my Notebook.</p>
  
Tags: <training><autoencoders><embeddings><clustering>
Title: How to learn Categorial Embeddings in Unsupervised Learning?
UserDisplayName: Jaanis
ThreadText: # How to learn Categorial Embeddings in Unsupervised Learning?
  
  (Asked by: username_0 on 2023-06-25)
  
  I want to cluster mixed-type tabular data, for the categorial columns I want to use Categorial Embeddings and then an Autoencoder Network before clustering with KMeans or similar.
  
  Now, when I want to train my model, the Batch will be fed into the Embeddings Layers and then the Autoencoder, which will then reconstruct the Embeddings. My Question is, what value should be passed into the loss function? Since the Decoder reconstructs the embedded columns, I cant pass the original categorial values.
  
  Solutions I found [here](https://github.com/chrislemke/autoembedder/blob/main/src/autoembedder/model.py) and [here](https://medium.com/kirey-group/autoembedder-training-embedding-layers-on-unsupervised-tasks-fc364c0f6eec) propose to save the Input Batch as a in a variable after feeding it into the Embeddings Layer (but before the AE) and use that as the target for the loss function.
  
  I am a ML/DL beginner, but this sounds fishy to me, because the Embeddings will not be trained by gradient descent. I tested this approach on a small tabular dataset vs. just feeding the categorial data into the AE (no Embeddings) and found that using the first approach (saving embedded cols as variable) to moderatly degrade Clustering Accuracy and NMI Score (This is not representative - just a small test on a small dataset). [Here](https://github.com/jaanisfehling/attention-mixed-type-clustering/blob/main/embedding_heart_disease.ipynb) is my Notebook.
  
  **Tags:** training, autoencoders, embeddings, clustering
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2023-06-26. Score: 0 | Accepted answer
  
  I found out pytorch backpropagates to the beginning of your model and also trains the embeddings.
  The sources I linked do it the right way.
  
  
  ---
  
  