Answers.Body: <p>We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also increases chances of overfitting.</p>
  <p>If you are running into overfitting while training your autoencoder tying your weights is an option. It is a regularization method. Not tying them is fine if you are getting any overfitting or want to solve overfitting by an alternative method.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-02-24T08:14:34.900Z
Answers.LastActivityDate: 2021-02-24T08:14:34.900Z
Answers.UserDisplayName: Abhishek Verma
Body: <p>To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?</p>
  <p>Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.</p>
  <p>So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-02-24T10:32:48.983Z
Comments.Text: What do you mean by "tying weights" or "tie the weights"?
Comments.UserDisplayName: nbro
Tags: <neural-networks><autoencoders><principal-component-analysis>
Title: Why Autoencoder Weights Are Not Always Tied
UserDisplayName: tushar
ThreadText: # Why Autoencoder Weights Are Not Always Tied
  
  (Asked by: username_0 on 2021-02-23)
  
  To me, tying weights in an autoencoder makes sense if we think of the auto encoder as doing PCA. Why in any situation would it make sense to not tie the weights? If we don't tie the weights, would it not try to learn something that is PCA anyway or rather something that might not be as optimal as PCA?
  
  Also, if weights are not tied, it doesn't make sense to me that the auto-encoder is invertible i.e. if the decoder is looking for an inverse operation because it's a mapping between spaces of different dimension which should not invertible.
  
  So, if the weights are not tied then why do we expect the decoder to learn anything meaningful i.e neither PCA nor an inverse operation?
  
  **Tags:** neural-networks, autoencoders, principal-component-analysis
  
  > **Comments:**
  > **username_1**: What do you mean by "tying weights" or "tie the weights"?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-02-24. Score: 1
  
  We expect the decoder to learn anything meaningful without tying the weights because the loss function is calculated between the input and reconstructed output and training will minimize that loss. The untied autoencoder's decoder will learn to transform the embedding back into the input. Not tying the weights gives more representational power and also increases chances of overfitting.
  
  If you are running into overfitting while training your autoencoder tying your weights is an option. It is a regularization method. Not tying them is fine if you are getting any overfitting or want to solve overfitting by an alternative method.
  
  
  ---
  
  