Answers.Body: <p>The quote hangs on the definitions of &quot;space&quot; and &quot;manifold&quot;. An example of a space might be the familiar 3D euclidean vector space, and an example of a manifold would be the surface of sphere within this space. You can also consider the surface of a sphere as a space, and a ring drawn on it as yet another manifold. Whether you think about about something as a space or manifold depends on context.</p>
  <p>From the quote, the dimension of the image space is <span class="math-container">$n$</span>.</p>
  <p>The dimension of the autoencoder representation/embedding is <span class="math-container">$k$</span>, and that is considered to be in &quot;latent space&quot; (the space from which all outputs will be mapped from) when you are looking at the compressed image vector at the encode/decode interface. You can consider moving within this space for instance by modifying the embedding vector directly, or compressing some new input image.</p>
  <p>The decoder maps out those <span class="math-container">$k$</span> dimensions as a manifold in the image space. So it outputs a structure of <span class="math-container">$n$</span> dimensions, but only creates values that are in the manifold that it can represent by expanding from the <span class="math-container">$k$</span>-dimensional embedding at the encoder/decoder interface. The expansion is always from a single point to another single point, no matter how much values can be multiplied, and the output shape twisted through the image space using non-linearities, it never <em>fills</em> the image space, instead it creates a <span class="math-container">$k$</span>-dimensional surface within it. You cannot move in the <span class="math-container">$n$</span>-dimensional image space arbitrarily and always end up in a position that the decoder can output.</p>
  <p>The shape of the manifold - how those <span class="math-container">$k$</span> dimensions embed into the <span class="math-container">$n$</span> dimensional image space - is learned. However, the fact that the decoding produces a <span class="math-container">$k$</span>-dimensional manifold in the larger space is a direct consequence of the architecture. Even an untrained decoder will do this, although the resulting manifold is of little interest.</p>
  <p>Another way of understanding this discrepancy is that the compression is lossy (information is discarded), but the decompression is precise (new information is not created, nor is image space sampled from stochastically). So the decompression cannot fully &quot;fill&quot; the image space.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-09T01:14:30.253Z
Answers.Text: Thank you for your prompt answer and clarification, I understand now!
Answers.UserDisplayName: Quersi
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-08T17:20:35.627Z
Answers.LastActivityDate: 2024-02-08T20:22:25.583Z
Answers.LastEditDate: 2024-02-08T20:22:25.583Z
Answers.UserDisplayName: Neil Slater
Body: <p>This <a href="https://arxiv.org/pdf/2106.10151.pdf" rel="nofollow noreferrer">quite famous paper</a> states page 3 that:</p>
  <blockquote>
  <p>The (well-known) fact which underlies the new conceptual framework is that all the natural images are located on or near some low-dimensional manifold (as shown by a huge number of previous papers, from Ruderman [1994] to Pope et al. [2021]). We can approximate this manifold by using a high-quality autoencoding DNN (Bourlard and Kamp [1988], Wang et al. [2014]) which first <strong>compresses the given image into a k-dimensional latent space, and then decompresses this latent space into a k-dimensional image manifold</strong> in the n-dimensional input space.</p>
  </blockquote>
  <p>Does this make sense? Shouldn't decompression happen in a larger dimension? I always though autoencoders '<em>shrank</em>' the dimension of the input then expanded it back to the original shape, but here it seems to imply the compressed image remains of the same dimension as its encoded form somehow.</p>
  
Tags: <deep-learning><autoencoders><adversarial-ml>
Title: Is "The Dimpled Manifold Hypothesis" correct to say this about autoencoders?
UserDisplayName: Quersi
ThreadText: # Is "The Dimpled Manifold Hypothesis" correct to say this about autoencoders?
  
  (Asked by: username_0 on 2024-02-08)
  
  This [quite famous paper](https://arxiv.org/pdf/2106.10151.pdf) states page 3 that:
  
  > The (well-known) fact which underlies the new conceptual framework is that all the natural images are located on or near some low-dimensional manifold (as shown by a huge number of previous papers, from Ruderman [1994] to Pope et al. [2021]). We can approximate this manifold by using a high-quality autoencoding DNN (Bourlard and Kamp [1988], Wang et al. [2014]) which first **compresses the given image into a k-dimensional latent space, and then decompresses this latent space into a k-dimensional image manifold** in the n-dimensional input space.
  
  Does this make sense? Shouldn't decompression happen in a larger dimension? I always though autoencoders '*shrank*' the dimension of the input then expanded it back to the original shape, but here it seems to imply the compressed image remains of the same dimension as its encoded form somehow.
  
  **Tags:** deep-learning, autoencoders, adversarial-ml
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2024-02-08. Score: 2 | Accepted answer
  
  The quote hangs on the definitions of "space" and "manifold". An example of a space might be the familiar 3D euclidean vector space, and an example of a manifold would be the surface of sphere within this space. You can also consider the surface of a sphere as a space, and a ring drawn on it as yet another manifold. Whether you think about about something as a space or manifold depends on context.
  
  From the quote, the dimension of the image space is $n$.
  
  The dimension of the autoencoder representation/embedding is $k$, and that is considered to be in "latent space" (the space from which all outputs will be mapped from) when you are looking at the compressed image vector at the encode/decode interface. You can consider moving within this space for instance by modifying the embedding vector directly, or compressing some new input image.
  
  The decoder maps out those $k$ dimensions as a manifold in the image space. So it outputs a structure of $n$ dimensions, but only creates values that are in the manifold that it can represent by expanding from the $k$-dimensional embedding at the encoder/decoder interface. The expansion is always from a single point to another single point, no matter how much values can be multiplied, and the output shape twisted through the image space using non-linearities, it never *fills* the image space, instead it creates a $k$-dimensional surface within it. You cannot move in the $n$-dimensional image space arbitrarily and always end up in a position that the decoder can output.
  
  The shape of the manifold - how those $k$ dimensions embed into the $n$ dimensional image space - is learned. However, the fact that the decoding produces a $k$-dimensional manifold in the larger space is a direct consequence of the architecture. Even an untrained decoder will do this, although the resulting manifold is of little interest.
  
  Another way of understanding this discrepancy is that the compression is lossy (information is discarded), but the decompression is precise (new information is not created, nor is image space sampled from stochastically). So the decompression cannot fully "fill" the image space.
  
  > **Comments:**
  > **username_0**: Thank you for your prompt answer and clarification, I understand now!  
  
  ---
  
  