Answers.Body: <p><a href="https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf" rel="nofollow noreferrer"><strong>Sparse auto-encoders</strong> (SAEs)</a> are auto-encoders that impose constraints on the parameters so that they are <em>sparse</em> (i.e. zero or close to zero). This can be achieved in different ways. For example, you can train an auto-encoder with <a href="https://papers.nips.cc/paper/3313-sparse-deep-belief-net-model-for-visual-area-v2.pdf" rel="nofollow noreferrer">a loss function that includes a penalty term</a> (to constraint the parameters to be close to zero or zero) or you e.g. <a href="https://arxiv.org/pdf/1312.5663.pdf" rel="nofollow noreferrer">set the smallest activations to zero</a>.</p>
  
  <p><strong>Convolution auto-encoders (CAEs)</strong> are auto-encoders that use the convolution operation. So, they can be viewed as the auto-encoder version of convolutional neural networks. For this reason, they are particularly suited to compress and reconstruct images. The authors of the original paper, <a href="http://people.idsia.ch/~juergen/icann2011stack.pdf" rel="nofollow noreferrer">Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction</a>, train them with gradient descent and back-propagation by minimizing the mean squared error, so there's e.g. no penalty term, but you can probably combine SAEs with CAEs.</p>
  
  <p>Of course, you could say that CAEs are <em>sparse</em> with respect to the traditional auto-encoder (in the same way that you can say that CNNs are sparse with respect to fully connected neural networks), so, in this sense, CAEs are also sparse auto-encoders.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-05-21T13:15:05.607Z
Answers.LastActivityDate: 2020-05-21T13:15:05.607Z
Answers.UserDisplayName: nbro
Body: <p>What are the main differences and similarities between sparse autoencoders and convolution autoencoders?</p>
  
  <p>When should one be preferred over the other? What are their applications?</p>
  
  <p>(References are welcome. Somehow I was not able to find any comparisons of these autoencoders although I looked in a few textbooks and searched for material online. I was able to find the descriptions of each autoencoder separately, but what I am interested in is the comparison.)</p>
  
Tags: <neural-networks><comparison><autoencoders>
Title: What are the main differences between sparse autoencoders and convolution autoencoders?
UserDisplayName: Pluviophile
ThreadText: # What are the main differences between sparse autoencoders and convolution autoencoders?
  
  (Asked by: username_0 on 2020-05-21)
  
  What are the main differences and similarities between sparse autoencoders and convolution autoencoders?
  
  When should one be preferred over the other? What are their applications?
  
  (References are welcome. Somehow I was not able to find any comparisons of these autoencoders although I looked in a few textbooks and searched for material online. I was able to find the descriptions of each autoencoder separately, but what I am interested in is the comparison.)
  
  **Tags:** neural-networks, comparison, autoencoders
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-05-21. Score: 0 | Accepted answer
  
  [**Sparse auto-encoders** (SAEs)](https://web.stanford.edu/class/cs294a/sparseAutoencoder.pdf) are auto-encoders that impose constraints on the parameters so that they are *sparse* (i.e. zero or close to zero). This can be achieved in different ways. For example, you can train an auto-encoder with [a loss function that includes a penalty term](https://papers.nips.cc/paper/3313-sparse-deep-belief-net-model-for-visual-area-v2.pdf) (to constraint the parameters to be close to zero or zero) or you e.g. [set the smallest activations to zero](https://arxiv.org/pdf/1312.5663.pdf).
  
  **Convolution auto-encoders (CAEs)** are auto-encoders that use the convolution operation. So, they can be viewed as the auto-encoder version of convolutional neural networks. For this reason, they are particularly suited to compress and reconstruct images. The authors of the original paper, [Stacked Convolutional Auto-Encoders for Hierarchical Feature Extraction](http://people.idsia.ch/~juergen/icann2011stack.pdf), train them with gradient descent and back-propagation by minimizing the mean squared error, so there's e.g. no penalty term, but you can probably combine SAEs with CAEs.
  
  Of course, you could say that CAEs are *sparse* with respect to the traditional auto-encoder (in the same way that you can say that CNNs are sparse with respect to fully connected neural networks), so, in this sense, CAEs are also sparse auto-encoders.
  
  
  ---
  
  