Answers.Body: <p>To answer this rather succinctly, an encoder is a function mapping some input to some different space. An example of this is what the brain does. We have to process the sensory input that the environment gives us in order for it to be storable.</p>
  
  <p>An autoencoder's job, on the other hand, is to learn a representation(encoding). An autoencoder will have the same number of output nodes as there are inputs for the purposes of reconstructing the inputs instead of trying to predict the Y target. Autoencoders are usually used in reducing output dimensions in high dimensional data sets.</p>
  
  <p>Hope I answered your question!</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T10:03:15.257Z
Answers.Text: Why would you need to reconstruct the inputs if you already have them? I think you should answer to this question too.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 3.0
Answers.CreationDate: 2017-10-12T20:36:32.803Z
Answers.LastActivityDate: 2017-10-12T20:36:32.803Z
Answers.UserDisplayName: hisairnessag3
Answers.Body: <p>As an addition to NicolaBernini's answer. Here is a full listing which should work with a Python 3 installation that includes Tensorflow:</p>
  <pre><code>&quot;&quot;&quot;MNIST autoencoder&quot;&quot;&quot;
  
  from tensorflow.python.keras.layers import  Input, Dense, Flatten, Reshape
  from tensorflow.python.keras.models import Model 
  keras.datasets import mnist
  import matplotlib.pyplot as plt
  from matplotlib.pyplot import figure
  
  &quot;&quot;&quot;## Load the MNIST dataset&quot;&quot;&quot;
  
  (x_train, y_train), (x_test, y_test) = mnist.load_data()
  
  &quot;&quot;&quot;## Define the autoencoder model&quot;&quot;&quot;
  
  ## MNIST 28x28 Input 
  image_shape = (28,28)
  
  ## Latent Space 
  space_compressed_size = 25 
  
  in_img = Input(shape=image_shape)
  img = Flatten()(in_img)
  encoder = Dense(space_compressed_size, activation='elu')(img)
  decoder = Dense(28*28, activation='elu')(encoder)
  reshaped = Reshape(image_shape)(decoder)
  autoencoder = Model(in_img, reshaped)
  autoencoder.compile(optimizer='adam', loss='mean_squared_error')
  
  &quot;&quot;&quot;## Train the autoencoder&quot;&quot;&quot;
  
  history = autoencoder.fit(x_train, x_train, epochs=10, shuffle=True, validation_data=(x_test, x_test))
  
  &quot;&quot;&quot;## Plot the training curves&quot;&quot;&quot;
  
  plt.plot(history.history['loss'])
  plt.plot(history.history['val_loss'])
  plt.legend(['loss', 'val_loss'])
  plt.show()
  
  &quot;&quot;&quot;## Generate some output images given some input images. This will allow us to see the quality of the reconstruction for the current value of ```space_compressed_size```&quot;&quot;&quot;
  
  rebuilt_images = autoencoder.predict([x_test[0:10]])
  
  &quot;&quot;&quot;## Plot the reconstructed images and compare them to the originals&quot;&quot;&quot;
  
  
  figure(num=None, figsize=(8, 32), dpi=80, facecolor='w', edgecolor='k')
  plot_ref = 0
  
  for i in range(len(rebuilt_images)):
  
    plot_ref += 1
    plt.subplot(len(rebuilt_images), 3, plot_ref)
    
    if i==0:
      plt.title(&quot;Reconstruction&quot;)
    
    plt.imshow(rebuilt_images[i].reshape((28,28)), cmap=&quot;gray&quot;)
    
    plot_ref += 1
    plt.subplot(len(rebuilt_images), 3, plot_ref)
    
    if i==0:
      plt.title(&quot;Original&quot;)
    
    plt.imshow(x_test[i].reshape((28,28)), cmap=&quot;gray&quot;)
    
    plot_ref += 1
    plt.subplot(len(rebuilt_images), 3, plot_ref)
    
    if i==0:
      plt.title(&quot;Error&quot;)
  
    plt.imshow(abs(rebuilt_images[i] - x_test[i]).reshape((28,28)), cmap=&quot;gray&quot;)
  
  plt.show(block=True)
  </code></pre>
  <p>I have changed the loss function of the training optimiser to &quot;mean_squared_error&quot; to capture the grayscale output of the images.
  Change the value of
  <code>space_compressed_size</code>
  to see how that effects the quality of the image reconstructions.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T14:24:09.233Z
Answers.LastActivityDate: 2022-05-08T18:17:49.210Z
Answers.LastEditDate: 2022-05-08T18:17:49.210Z
Answers.UserDisplayName: DrMcCleod
Answers.Body: <h1>Theory</h1>
  
  <h2>Encoder</h2>
  
  <ul>
  <li>In general, an Encoder is a mapping <span class="math-container">$f : X \rightarrow Y $</span> with <span class="math-container">$X$</span> Input Space and <span class="math-container">$Y$</span> Code Space </li>
  <li>In case of Neural Networks, it is a <strong>Generative Model</strong> hence a function which is able to compute a <strong>Representation</strong> out of some input (like GAN)</li>
  </ul>
  
  <p>The point is: how would you train such an encoder network ? </p>
  
  <ul>
  <li>The general answer is: it depends on what you want your <strong>code</strong> to be and ultimately depends on what kind of problem the NN has to solve, so let's pick one  </li>
  </ul>
  
  <h2>Signal Compression</h2>
  
  <p>The goal is to learn a compressed representation for your input that allows to reconstruct the original input minimizing the loss of information </p>
  
  <p>In this case hence you want the dimensionality of <span class="math-container">$Y$</span> to be lower than the dimensionality <span class="math-container">$X$</span> which in the NN case means the code space will be represented by less neurons than the input space </p>
  
  <h2>Autoencoder</h2>
  
  <p>Focusing on the Signal Compression problem, what we want to build is a system which is able to </p>
  
  <ul>
  <li><p>take a given signal with size <code>N</code> bytes </p></li>
  <li><p>compress it into another signal with size <code>M&lt;N</code> bytes </p></li>
  <li><p>reconstruct the original signal, starting from the compressed representation, as good as possible </p></li>
  </ul>
  
  <p>To be able to achiebve this goal, we need basically 2 components </p>
  
  <ul>
  <li><p>an Encoder which compresses its input, performing the <span class="math-container">$f : X \rightarrow Y$</span> mapping </p></li>
  <li><p>a Decoder which decompresses its input, performing the <span class="math-container">$f: Y \rightarrow X$</span> mapping </p></li>
  </ul>
  
  <p>We can approach this problem with the Neural Network Framework, defining an Encoder NN and a Decoder NN and training them </p>
  
  <p>It is important to observe this kind of problem can be effectively approached with the convenient learning strategy of <strong>unsupervised learning</strong> : there is no need to spend any <em>human work</em> (expensive) to build a supervision signal as the original input can be used for this purpose </p>
  
  <p>This means we have to build a NN which operates essentially between 2 spaces </p>
  
  <ul>
  <li><p>the <span class="math-container">$X$</span> Input Space </p></li>
  <li><p>the <span class="math-container">$Y$</span> Latent or Compressed Space </p></li>
  </ul>
  
  <p>The general idea behind the training is to make a certain input go along the encoder + decoder pipeline and then compare the reconstruction result with the original input with some kind of loss function </p>
  
  <p>To define this idea a bit more formally </p>
  
  <ul>
  <li>The final autoencoder mapping is <span class="math-container">$f : X \rightarrow Y \rightarrow X$</span> with 
  
  <ul>
  <li>the <span class="math-container">$x$</span> input </li>
  <li>the <span class="math-container">$y$</span> encoded input or latent representation of the input </li>
  <li>the <span class="math-container">$\hat x$</span> reconstructed input </li>
  </ul></li>
  <li>Eventually you will get an architecture similar to </li>
  </ul>
  
  <p><img src="https://cdn-images-1.medium.com/max/1600/1*44eDEuZBEsmG_TCAKRI3Kw@2x.png" alt="AE1"></p>
  
  <ul>
  <li>You can train this architecture in an unsupervised way, using a loss function like <span class="math-container">$f : X \times X \rightarrow \mathbb{R}$</span> so that <span class="math-container">$f(x, \hat x)$</span> is the loss associated to the <span class="math-container">$\hat x$</span> reconstruction compared with the <span class="math-container">$x$</span> input which is also the ideal result </li>
  </ul>
  
  <h1>Code</h1>
  
  <p>Now let's add a simple example in Keras related to the MNIST Dataset </p>
  
  
  
  <pre><code>
  from keras.layers import Input, Dense 
  from keras.models import Model 
  
  # Defines spaces sizes 
  
  ## MNIST 28x28 Input 
  space_in_size = 28*28
  
  ## Latent Space 
  space_compressed_size = 32 
  
  # Defines the Input Tensor 
  in_img = Input(shape=(space_in_size,))
  
  encoder = Dense(space_compressed_size, activation='relu')(in_img)
  
  decoder = Dense(space_in_size, activation='sigmoid')(encoder)
  
  autoencoder = Model(in_img, decoder)
  
  autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
  
  </code></pre>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T09:11:04.757Z
Answers.Text: Is binary_crossentropy the correct loss function to use there? Wouldn't mean_squared_error be the better choice?
Answers.UserDisplayName: DrMcCleod
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T09:34:08.747Z
Answers.Text: Binary cross entropy should be a good choice in this specific case of MNIST digits reconstruction, as it is modelled as a per-pixel binary classification: we just want to know what pixels to turn on
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T12:17:35.633Z
Answers.Text: @nicolabernini why wouldn't you be interested in the grayscale values of the decoded image? MNIST pixels are not just on and off.
Answers.UserDisplayName: DrMcCleod
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T13:33:45.787Z
Answers.Text: @DuttaA because the less relevant kind of details is exactly the one related to the original pixel values: what you really want is to turn on the pixels giving you the right digit appearance (instead of defining an appearance which is properly spread across the full grayscale domain)
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T14:13:09.070Z
Answers.Text: Sorry, I do not understand what you mean: the Sigmoid Activation Th is a learned parameter, not an hyperparam 
  Just to recap, before the actual processing the pixel values get normalized (so they are in the [0..1] range) so you have to classify between 0 (turned off) and 1 (turned on)
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T16:17:51.150Z
Answers.Text: Try having a look at the per-pixel values PDF in MNIST: you clearly observe 2 strong peaks, one for {0} and the other for {255}
  That's because the MNIST image is intended to be binary and the you can consider all the other values as noise 
  Normalizing data, the NN predicts a {0,1} class and compares it with the input then evaluates with binary cross entropy
Answers.UserDisplayName: Nicola Bernini
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-04-14T08:28:48.880Z
Answers.LastActivityDate: 2019-04-14T10:52:09.873Z
Answers.LastEditDate: 2019-04-14T10:52:09.873Z
Answers.UserDisplayName: Nicola Bernini
Body: <p>How are the layers in a encoder connected across the network for normal encoders and auto-encoders? In general, what is the difference between encoders and auto-encoders?</p>
  
Tags: <machine-learning><deep-learning><comparison><autoencoders>
Title: What is the difference between encoders and auto-encoders?
UserDisplayName: m2rik
ThreadText: # What is the difference between encoders and auto-encoders?
  
  (Asked by: username_0 on 2017-10-10)
  
  How are the layers in a encoder connected across the network for normal encoders and auto-encoders? In general, what is the difference between encoders and auto-encoders?
  
  **Tags:** machine-learning, deep-learning, comparison, autoencoders
  
  ---
  
  ## 3 answers
  
  ### Answer by username_4 on 2019-04-14. Score: 8 | Accepted answer
  
  Theory
  Encoder
  
  In general, an Encoder is a mapping $f : X \rightarrow Y $ with $X$ Input Space and $Y$ Code Space 
  In case of Neural Networks, it is a Generative Model hence a function which is able to compute a Representation out of some input (like GAN)
  
  The point is: how would you train such an encoder network ? 
  
  The general answer is: it depends on what you want your code to be and ultimately depends on what kind of problem the NN has to solve, so let's pick one  
  
  Signal Compression
  The goal is to learn a compressed representation for your input that allows to reconstruct the original input minimizing the loss of information 
  In this case hence you want the dimensionality of $Y$ to be lower than the dimensionality $X$ which in the NN case means the code space will be represented by less neurons than the input space 
  Autoencoder
  Focusing on the Signal Compression problem, what we want to build is a system which is able to 
  
  take a given signal with size N bytes 
  compress it into another signal with size M bytes 
  reconstruct the original signal, starting from the compressed representation, as good as possible 
  
  To be able to achiebve this goal, we need basically 2 components 
  
  an Encoder which compresses its input, performing the $f : X \rightarrow Y$ mapping 
  a Decoder which decompresses its input, performing the $f: Y \rightarrow X$ mapping 
  
  We can approach this problem with the Neural Network Framework, defining an Encoder NN and a Decoder NN and training them 
  It is important to observe this kind of problem can be effectively approached with the convenient learning strategy of unsupervised learning : there is no need to spend any human work (expensive) to build a supervision signal as the original input can be used for this purpose 
  This means we have to build a NN which operates essentially between 2 spaces 
  
  the $X$ Input Space 
  the $Y$ Latent or Compressed Space 
  
  The general idea behind the training is to make a certain input go along the encoder + decoder pipeline and then compare the reconstruction result with the original input with some kind of loss function 
  To define this idea a bit more formally 
  
  The final autoencoder mapping is $f : X \rightarrow Y \rightarrow X$ with 
  
  
  the $x$ input 
  the $y$ encoded input or latent representation of the input 
  the $\hat x$ reconstructed input 
  
  Eventually you will get an architecture similar to 
  
  
  
  You can train this architecture in an unsupervised way, using a loss function like $f : X \times X \rightarrow \mathbb{R}$ so that $f(x, \hat x)$ is the loss associated to the $\hat x$ reconstruction compared with the $x$ input which is also the ideal result 
  
  Code
  Now let's add a simple example in Keras related to the MNIST Dataset 
  
  from keras.layers import Input, Dense 
  from keras.models import Model 
  
  # Defines spaces sizes 
  
  ## MNIST 28x28 Input 
  space_in_size = 28*28
  
  ## Latent Space 
  space_compressed_size = 32 
  
  # Defines the Input Tensor 
  in_img = Input(shape=(space_in_size,))
  
  encoder = Dense(space_compressed_size, activation='relu')(in_img)
  
  decoder = Dense(space_in_size, activation='sigmoid')(encoder)
  
  autoencoder = Model(in_img, decoder)
  
  autoencoder.compile(optimizer='adam', loss='binary_crossentropy')
  
  > **Comments:**
  > **username_3**: Is binary_crossentropy the correct loss function to use there? Wouldn't mean_squared_error be the better choice?  
  > **username_4**: Binary cross entropy should be a good choice in this specific case of MNIST digits reconstruction, as it is modelled as a per-pixel binary classification: we just want to know what pixels to turn on  
  > **username_3**: @nicolabernini why wouldn't you be interested in the grayscale values of the decoded image? MNIST pixels are not just on and off.  
  > **username_4**: @DuttaA because the less relevant kind of details is exactly the one related to the original pixel values: what you really want is to turn on the pixels giving you the right digit appearance (instead of defining an appearance which is properly spread across the full grayscale domain)  
  > **username_4**: Sorry, I do not understand what you mean: the Sigmoid Activation Th is a learned parameter, not an hyperparam  Just to recap, before the actual processing the pixel values get normalized (so they are in the [0..1] range) so you have to classify between 0 (turned off) and 1 (turned on)  
  > **username_4**: Try having a look at the per-pixel values PDF in MNIST: you clearly observe 2 strong peaks, one for {0} and the other for {255} That's because the MNIST image is intended to be binary and the you can consider all the other values as noise  Normalizing data, the NN predicts a {0,1} class and compares it with the input then evaluates with binary cross entropy  
  
  ---
  
  ### Answer by username_1 on 2017-10-12. Score: 1
  
  To answer this rather succinctly, an encoder is a function mapping some input to some different space. An example of this is what the brain does. We have to process the sensory input that the environment gives us in order for it to be storable.
  
  An autoencoder's job, on the other hand, is to learn a representation(encoding). An autoencoder will have the same number of output nodes as there are inputs for the purposes of reconstructing the inputs instead of trying to predict the Y target. Autoencoders are usually used in reducing output dimensions in high dimensional data sets.
  
  Hope I answered your question!
  
  > **Comments:**
  > **username_2**: Why would you need to reconstruct the inputs if you already have them? I think you should answer to this question too.  
  
  ---
  
  ### Answer by username_3 on 2019-04-14. Score: 1
  
  As an addition to NicolaBernini's answer. Here is a full listing which should work with a Python 3 installation that includes Tensorflow:
  """MNIST autoencoder"""
  
  from tensorflow.python.keras.layers import  Input, Dense, Flatten, Reshape
  from tensorflow.python.keras.models import Model 
  keras.datasets import mnist
  import matplotlib.pyplot as plt
  from matplotlib.pyplot import figure
  
  """## Load the MNIST dataset"""
  
  (x_train, y_train), (x_test, y_test) = mnist.load_data()
  
  """## Define the autoencoder model"""
  
  ## MNIST 28x28 Input 
  image_shape = (28,28)
  
  ## Latent Space 
  space_compressed_size = 25 
  
  in_img = Input(shape=image_shape)
  img = Flatten()(in_img)
  encoder = Dense(space_compressed_size, activation='elu')(img)
  decoder = Dense(28*28, activation='elu')(encoder)
  reshaped = Reshape(image_shape)(decoder)
  autoencoder = Model(in_img, reshaped)
  autoencoder.compile(optimizer='adam', loss='mean_squared_error')
  
  """## Train the autoencoder"""
  
  history = autoencoder.fit(x_train, x_train, epochs=10, shuffle=True, validation_data=(x_test, x_test))
  
  """## Plot the training curves"""
  
  plt.plot(history.history['loss'])
  plt.plot(history.history['val_loss'])
  plt.legend(['loss', 'val_loss'])
  plt.show()
  
  """## Generate some output images given some input images. This will allow us to see the quality of the reconstruction for the current value of ```space_compressed_size```"""
  
  rebuilt_images = autoencoder.predict([x_test[0:10]])
  
  """## Plot the reconstructed images and compare them to the originals"""
  
  
  figure(num=None, figsize=(8, 32), dpi=80, facecolor='w', edgecolor='k')
  plot_ref = 0
  
  for i in range(len(rebuilt_images)):
  
    plot_ref += 1
    plt.subplot(len(rebuilt_images), 3, plot_ref)
    
    if i==0:
      plt.title("Reconstruction")
    
    plt.imshow(rebuilt_images[i].reshape((28,28)), cmap="gray")
    
    plot_ref += 1
    plt.subplot(len(rebuilt_images), 3, plot_ref)
    
    if i==0:
      plt.title("Original")
    
    plt.imshow(x_test[i].reshape((28,28)), cmap="gray")
    
    plot_ref += 1
    plt.subplot(len(rebuilt_images), 3, plot_ref)
    
    if i==0:
      plt.title("Error")
  
    plt.imshow(abs(rebuilt_images[i] - x_test[i]).reshape((28,28)), cmap="gray")
  
  plt.show(block=True)
  
  I have changed the loss function of the training optimiser to "mean_squared_error" to capture the grayscale output of the images.
  Change the value of
  space_compressed_size
  to see how that effects the quality of the image reconstructions.
  
  
  ---
  
  