Answers.Body: <p>A use case of autoencoders (in particular, of the decoder or generative model of the autoencoder) is to denoise the input. This type of autoencoders, called <em>denoising autoencoders</em>, take a partially corrupted input and they attempt to reconstruct the corresponding uncorrupted input. There are several applications of this model. For example, if you had a corrupted image, you could potentially recover the uncorrupted one using a denoising autoencoder.</p>
  
  <p>Autoencoders and PCA are related:</p>
  
  <blockquote>
    <p>an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principal component loading vectors, but that they are not identical to the loading vectors. </p>
  </blockquote>
  
  <p>For more info, have a look at the paper <a href="https://arxiv.org/pdf/1804.10253.pdf" rel="nofollow noreferrer">From Principal Subspaces to Principal Components with Linear Autoencoders</a> (2018), by Elad Plaut. See also <a href="https://stats.stackexchange.com/a/120096/82135">this answer</a>, which also explains the relation between PCA and autoencoders.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T15:53:23.880Z
Answers.LastActivityDate: 2019-03-23T15:59:38.497Z
Answers.LastEditDate: 2019-03-23T15:59:38.497Z
Answers.UserDisplayName: nbro
Answers.Body: <p>It is important to think about what sort of patterns in the data are being represented. </p>
  
  <p>Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human brain you'd realise that every element in this dataset can be described in terms of a single numeric parameter, which is that intensity value. This is something that PCA would work fine for, because each of the dimensions (we can think of each pixel as a different dimension) is perfectly linearly correlated. </p>
  
  <p>Suppose instead that you have a dataset of black and white 128x128px bitmap images of centred circles. As a human brain you'd quickly realise that every element in this dataset can be fully described by a single numeric parameter, which is the radius of the circle. That is a very impressive level of reduction from 16384 binary dimensions, and perhaps more importantly it's a semantically meaningful property of the data. However, PCA probably won't be able to find that pattern.</p>
  
  <p>Your question was "Why can't we simply use PCA, if the purpose is dimensionality reduction?" The simple answer is that PCA is the simplest tool for dimensionality reduction, but it can miss a lot of relationships that more powerful techniques such as autoencoders might find. </p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-24T00:26:08.930Z
Answers.LastActivityDate: 2019-03-24T00:26:08.930Z
Answers.UserDisplayName: Josiah
Answers.Body: <p>PCA is a linear method that creates a transformation that is capable of changing the vectors projections (changing axis)</p>
  
  <p>Since PCA looks for the direction of maximum variance it usually have high discriminativity <strong>BUT</strong> it does not guaranteed that the direction of most variance is the direction of most discriminativity.</p>
  
  <p>LDA is a linear method that creates a transformation that is capable of finding the direction that is most relevant to decide if a vector belong to class A or B.</p>
  
  <p>PCA and LDA have non-linear Kernel versions that might overcome their linear limitations.</p>
  
  <p>Autoencoders can perform dimensionality reduction with other kinds of loss function, can be non-linear and might perform better than PCA and LDA for a lot of cases.</p>
  
  <p>There is probably no best machine learning algorithm to do anything, sometimes Deep Learning and Neural Nets are overkill for simple problems and PCA and LDA might be tried before other, more complex, dimensionality reductions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T21:45:07.080Z
Answers.Text: What does LDA have to do with question?
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T21:46:31.093Z
Answers.Text: LDA can be used as dimensionality reduction. The original algorithm derives only one projection but you can use it to get lower ranking discriminative direction for more acurate modelling
Answers.UserDisplayName: Pedro Henrique Monforte
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T16:26:24.680Z
Answers.Text: PCA does not guarantee maximum variance either, since it is a greedy algorithm it only approximates the maximum.
Answers.UserDisplayName: Mathieu Bouville
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-31T18:46:01.483Z
Answers.Text: PCA has a closed solution, the main axis will always be de direction of maximum variance. It is not a greedy algorithm (but it can be implemented as one)
Answers.UserDisplayName: Pedro Henrique Monforte
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-03-23T21:29:29.433Z
Answers.LastActivityDate: 2019-03-23T21:29:29.433Z
Answers.UserDisplayName: Pedro Henrique Monforte
Answers.Body: <ol>
  <li>The decoder half is necessary in order to compute the loss function for training the network. Similar to how the 'adversary' is still necessary in a GAN even if you are only interested in the generative component.</li>
  <li>Autoencoders can learn non-linear embeddings of the data, and hence are more powerful than vanilla PCA.</li>
  <li>Autoencoders have <a href="https://en.wikipedia.org/wiki/Autoencoder#Applications" rel="nofollow noreferrer">applications</a> beyond dimensionality reduction:
  
  <ul>
  <li>Generating new data points, or perform interpolation (see VAE's)</li>
  <li>Create denoising filters (e.g. in image processing)</li>
  <li>Compress/decompress data</li>
  <li>Link prediction (e.g. in drug discovery)</li>
  </ul></li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-24T16:11:40.003Z
Answers.LastActivityDate: 2020-04-25T19:04:52.107Z
Answers.LastEditDate: 2020-04-25T19:04:52.107Z
Answers.UserDisplayName: brazofuerte
Body: <p><a href="https://en.wikipedia.org/wiki/Autoencoder" rel="noreferrer">Autoencoders</a> are neural networks that learn a compressed representation of the input in order to later reconstruct it, so they can be used for dimensionality reduction. They are composed of an encoder and a decoder (which can be separate neural networks). Dimensionality reduction can be useful in order to deal with or attenuate the issues related to the curse of dimensionality, where data becomes sparse and it is more difficult to obtain "statistical significance". So, autoencoders (and algorithms like PCA) can be used to deal with the curse of dimensionality. </p>
  
  <p>Why do we care about dimensionality reduction specifically using autoencoders? Why can't we simply use PCA, if the purpose is dimensionality reduction? </p>
  
  <p>Why do we need to decompress the latent representation of the input if we just want to perform dimensionality reduction, or why do we need the decoder part in an autoencoder? What are the use cases? In general, why do we need to compress the input to later decompress it? Wouldn't it be better to just use the original input (to start with)?</p>
  
Tags: <machine-learning><autoencoders><dimensionality-reduction><curse-of-dimensionality>
Title: What are the purposes of autoencoders?
UserDisplayName: nbro
ThreadText: # What are the purposes of autoencoders?
  
  (Asked by: username_0 on 2019-03-23)
  
  [Autoencoders](https://en.wikipedia.org/wiki/Autoencoder) are neural networks that learn a compressed representation of the input in order to later reconstruct it, so they can be used for dimensionality reduction. They are composed of an encoder and a decoder (which can be separate neural networks). Dimensionality reduction can be useful in order to deal with or attenuate the issues related to the curse of dimensionality, where data becomes sparse and it is more difficult to obtain "statistical significance". So, autoencoders (and algorithms like PCA) can be used to deal with the curse of dimensionality.
  
  Why do we care about dimensionality reduction specifically using autoencoders? Why can't we simply use PCA, if the purpose is dimensionality reduction?
  
  Why do we need to decompress the latent representation of the input if we just want to perform dimensionality reduction, or why do we need the decoder part in an autoencoder? What are the use cases? In general, why do we need to compress the input to later decompress it? Wouldn't it be better to just use the original input (to start with)?
  
  **Tags:** machine-learning, autoencoders, dimensionality-reduction, curse-of-dimensionality
  
  ---
  
  ## 4 answers
  
  ### Answer by username_1 on 2019-03-24. Score: 7
  
  It is important to think about what sort of patterns in the data are being represented.
  
  Suppose that you have a dataset of greyscale images, such that every image is a uniform intensity. As a human brain you'd realise that every element in this dataset can be described in terms of a single numeric parameter, which is that intensity value. This is something that PCA would work fine for, because each of the dimensions (we can think of each pixel as a different dimension) is perfectly linearly correlated.
  
  Suppose instead that you have a dataset of black and white 128x128px bitmap images of centred circles. As a human brain you'd quickly realise that every element in this dataset can be fully described by a single numeric parameter, which is the radius of the circle. That is a very impressive level of reduction from 16384 binary dimensions, and perhaps more importantly it's a semantically meaningful property of the data. However, PCA probably won't be able to find that pattern.
  
  Your question was "Why can't we simply use PCA, if the purpose is dimensionality reduction?" The simple answer is that PCA is the simplest tool for dimensionality reduction, but it can miss a lot of relationships that more powerful techniques such as autoencoders might find.
  
  
  ---
  
  ### Answer by username_0 on 2019-03-23. Score: 3
  
  A use case of autoencoders (in particular, of the decoder or generative model of the autoencoder) is to denoise the input. This type of autoencoders, called *denoising autoencoders*, take a partially corrupted input and they attempt to reconstruct the corresponding uncorrupted input. There are several applications of this model. For example, if you had a corrupted image, you could potentially recover the uncorrupted one using a denoising autoencoder.
  
  Autoencoders and PCA are related:
  
  > an autoencoder with a single fully-connected hidden layer, a linear activation function and a squared error cost function trains weights that span the same subspace as the one spanned by the principal component loading vectors, but that they are not identical to the loading vectors.
  
  For more info, have a look at the paper [From Principal Subspaces to Principal Components with Linear Autoencoders](https://arxiv.org/pdf/1804.10253.pdf) (2018), by Elad Plaut. See also [this answer](https://stats.stackexchange.com/a/120096/82135), which also explains the relation between PCA and autoencoders.
  
  
  ---
  
  ### Answer by username_2 on 2019-03-23. Score: 2
  
  PCA is a linear method that creates a transformation that is capable of changing the vectors projections (changing axis)
  
  Since PCA looks for the direction of maximum variance it usually have high discriminativity **BUT** it does not guaranteed that the direction of most variance is the direction of most discriminativity.
  
  LDA is a linear method that creates a transformation that is capable of finding the direction that is most relevant to decide if a vector belong to class A or B.
  
  PCA and LDA have non-linear Kernel versions that might overcome their linear limitations.
  
  Autoencoders can perform dimensionality reduction with other kinds of loss function, can be non-linear and might perform better than PCA and LDA for a lot of cases.
  
  There is probably no best machine learning algorithm to do anything, sometimes Deep Learning and Neural Nets are overkill for simple problems and PCA and LDA might be tried before other, more complex, dimensionality reductions.
  
  > **Comments:**
  > **username_0**: What does LDA have to do with question?  
  > **username_2**: LDA can be used as dimensionality reduction. The original algorithm derives only one projection but you can use it to get lower ranking discriminative direction for more acurate modelling  
  > **username_3**: PCA does not guarantee maximum variance either, since it is a greedy algorithm it only approximates the maximum.  
  > **username_2**: PCA has a closed solution, the main axis will always be de direction of maximum variance. It is not a greedy algorithm (but it can be implemented as one)  
  
  ---
  
  ### Answer by username_4 on 2020-04-24. Score: 1
  
  1. The decoder half is necessary in order to compute the loss function for training the network. Similar to how the 'adversary' is still necessary in a GAN even if you are only interested in the generative component.
  2. Autoencoders can learn non-linear embeddings of the data, and hence are more powerful than vanilla PCA.
  3. Autoencoders have [applications](https://en.wikipedia.org/wiki/Autoencoder#Applications) beyond dimensionality reduction:
     * Generating new data points, or perform interpolation (see VAE's)
     * Create denoising filters (e.g. in image processing)
     * Compress/decompress data
     * Link prediction (e.g. in drug discovery)
  
  
  ---
  
  