Answers.Body: <p>You are using Dense layers, try 1d convolution instead.
  Have you tried a different activation function such as softmax and instead of Binary cross entropy try MSE loss? Are all your inputs between 0 and 1? Also, I think your noise amplitude is too much in 2nd and 3rd case as compared to the actual signal. Can you try training on different types of spectra separately and check the result, if the DAE is learning anything. Also, try trining for more epochs. </p>
  
  <p>I am not sure and would have commented this but I can't due to low rep, but I am interested in the solution.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-23T20:06:04.847Z
Answers.Text: Thank you for your suggestions. I have tried them and found that the main issue was that I performed scaling by diving to the max value of ALL the spectra. After performing the scaling individually for each spectrum, changing to MSE loss and using LeakyReLU as the activation function, with 100 training epochs, the solution works better. I will try to implement with Conv1D, but I have a hard time understanding how to implement conv nets. I would be very grateful if you could show/help me how to rewrite the model as a convolutional one.
Answers.UserDisplayName: Cristian M
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-24T14:26:35.753Z
Answers.Text: @CristianM [the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D?version=stable) will help you with the implementation of convnets. Instead of dense, you just use conv1d layers and pass the size of convolutioton filter with it.
Answers.UserDisplayName: SajanGohil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-12-22T11:00:25.207Z
Answers.LastActivityDate: 2019-12-22T11:00:25.207Z
Answers.UserDisplayName: SajanGohil
Answers.Body: <p>If the auto-encoder is converging to the same encoding for different instances, there may be a problem in the loss function. Check the size and shape of the output of the loss function, as it may be getting confused and evaluating the wrong tensors (i.e. you may need to transpose something somewhere).</p>
  <p>Basically, assuming you are using an auto-encoder to encode <span class="math-container">$M$</span> features of <span class="math-container">$N$</span> training instances, your loss function should return <span class="math-container">$N$</span> values i.e.,
  <em><strong>the size of your loss tensor should be the amount of instances in your training set</strong></em>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-04-08T11:18:40.453Z
Answers.LastActivityDate: 2021-04-11T18:28:55.120Z
Answers.LastEditDate: 2021-04-11T18:28:55.120Z
Answers.UserDisplayName: Juan Esteban Florez
Body: <p>I am trying to implement a denoising autoencoder (DAE) to remove noise from 1024-point FFT spectra. I am using two types of spectra: (1) that contain a distinctive high amplitude spectral peak and (2) that contain only noise peaks.</p>
  
  <p>If I understood correctly, I can train the DAE using the corruputed spectra (spectra+noise) and afterwards I can use it the remove noise from new datasets. The problem is that when testing the DAE, it returns the type (1) spectrum mentioned above, regardless of the input. The same case when I apply predict on the training data. This is the code I am using (Python/Tensorflow):</p>
  
  <pre><code>def BuildModel(nInput):
      input_dim = Input(shape = (nInput, ))
  
      # Encoder Layers
      encoded1 = Dense(896, activation = 'relu')(input_dim)
      encoded2 = Dense(768, activation = 'relu')(encoded1)
      encoded3 = Dense(640, activation = 'relu')(encoded2)
      encoded4 = Dense(512, activation = 'relu')(encoded3)
      encoded5 = Dense(384, activation = 'relu')(encoded4)
      encoded6 = Dense(256, activation = 'relu')(encoded5)
      encoded7 = Dense(encoding_dim, activation = 'relu')(encoded6)
  
      # Decoder Layers
      decoded1 = Dense(256, activation = 'relu')(encoded7)
      decoded2 = Dense(384, activation = 'relu')(decoded1)
      decoded3 = Dense(512, activation = 'relu')(decoded2)
      decoded4 = Dense(640, activation = 'relu')(decoded3)
      decoded5 = Dense(768, activation = 'relu')(decoded4)
      decoded6 = Dense(896, activation = 'relu')(decoded5)
      decoded7 = Dense(nInput, activation = 'sigmoid')(decoded6)
  
      # Combine Encoder and Deocoder layers
      autoencoder = Model(inputs = input_dim, outputs = decoded7)
  
      autoencoder.summary()
      # Compile the Model
      autoencoder.compile(optimizer=OPTIMIZER, loss='binary_crossentropy')
      #autoencoder.compile(loss='mean_squared_error', optimizer = RMSprop())
  
      return autoencoder
  
  X_train, X_test, y_train, y_test = train_test_split(spectra.iloc[:,0:spectra.shape[1]-1], spectra['Class'], test_size=testDatasetSize, stratify=spectra.Class, random_state=seedValue)
  
  X_train, y_train = shuffle(X_train, y_train, random_state=seedValue)
  X_test, y_test = shuffle(X_test, y_test, random_state=seedValue)
  
  X_unseen = X_train.to_numpy()[0:1000,:] # Data not used for training, only for testing
  y_unseen = y_train.to_numpy()[0:1000]
  X_train = X_train.iloc[1000:]
  y_train = y_train.iloc[1000:]
  
  # Scaling
  maxVal = max(X_train)
  X_train = (X_train/maxVal).to_numpy()
  X_test = (X_test/maxVal).to_numpy()
  X_unseen = (X_unseen/maxVal)#.to_numpy()
  
  # Corrupted data
  noise_factor = 0.01
  X_train_noisy = X_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_train.shape)
  X_test_noisy = X_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_test.shape)
  X_unseen_noisy = X_unseen + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_unseen.shape)
  
  ae = BuildModel(X_train.shape[1])
  PrintConsoleLine('Creating model finished')
  print('')
  
  history = ae.fit(X_train_noisy, X_train, epochs=NB_EPOCH, batch_size=BATCH_SIZE, validation_data=[X_test_noisy, X_test])
  save_model(ae, modelFile, overwrite=True)
  
  # Test
  X = X_unseen
  X_noisy = X_unseen_noisy
  X_denoised = ae.predict(X_noisy) # X_train gives the same result (spectra type (1)) !?!
  N = len(X_denoised[0,:])
  index = 6
  PlotDataSimple(3, np.linspace(0,N-1,N), X[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
  PlotDataSimple(4, np.linspace(0,N-1,N), X_noisy[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
  PlotDataSimple(5, np.linspace(0,N-1,N), X_denoised[index, :], 'Frequency domain', 'Index', 'Amplitude', None)
  
  Dataset shape: (17000, 65, 65, 1) (files, samples X axis, samples Y axis, class)
  Train on 12600 samples, validate on 3400 samples
  Epoch 1/3
  12600/12600 [==============================] - 26s 2ms/sample - loss: 0.6813 - val_loss: 0.4913
  Epoch 2/3
  12600/12600 [==============================] - 14s 1ms/sample - loss: 0.1621 - val_loss: 0.0578
  Epoch 3/3
  12600/12600 [==============================] - 16s 1ms/sample - loss: 0.0230 - val_loss: 0.0169
  </code></pre>
  
  <p>The results I am getting (Column 1 - Initial signal, Column 2 - Corrupted signal, Column 3 - Denoised signal):</p>
  
  <p><a href="https://i.sstatic.net/aXKsr.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/aXKsr.png" alt=""></a></p>
  
  <p>So why does the DAE output the same spectra regardless of the inputs? Am I misunderstanding the DAE principle or is there a problem in my implementation?</p>
  
Tags: <machine-learning><tensorflow><python><autoencoders>
Title: Why does the denoising autoencoder always returns the same output?
UserDisplayName: Cristian M
ThreadText: # Why does the denoising autoencoder always returns the same output?
  
  (Asked by: username_0 on 2019-12-21)
  
  I am trying to implement a denoising autoencoder (DAE) to remove noise from 1024-point FFT spectra. I am using two types of spectra: (1) that contain a distinctive high amplitude spectral peak and (2) that contain only noise peaks.
  
  If I understood correctly, I can train the DAE using the corruputed spectra (spectra+noise) and afterwards I can use it the remove noise from new datasets. The problem is that when testing the DAE, it returns the type (1) spectrum mentioned above, regardless of the input. The same case when I apply predict on the training data. This is the code I am using (Python/Tensorflow):
  
  ```
  def BuildModel(nInput):
      input_dim = Input(shape = (nInput, ))
  
      # Encoder Layers
      encoded1 = Dense(896, activation = 'relu')(input_dim)
      encoded2 = Dense(768, activation = 'relu')(encoded1)
      encoded3 = Dense(640, activation = 'relu')(encoded2)
      encoded4 = Dense(512, activation = 'relu')(encoded3)
      encoded5 = Dense(384, activation = 'relu')(encoded4)
      encoded6 = Dense(256, activation = 'relu')(encoded5)
      encoded7 = Dense(encoding_dim, activation = 'relu')(encoded6)
  
      # Decoder Layers
      decoded1 = Dense(256, activation = 'relu')(encoded7)
      decoded2 = Dense(384, activation = 'relu')(decoded1)
      decoded3 = Dense(512, activation = 'relu')(decoded2)
      decoded4 = Dense(640, activation = 'relu')(decoded3)
      decoded5 = Dense(768, activation = 'relu')(decoded4)
      decoded6 = Dense(896, activation = 'relu')(decoded5)
      decoded7 = Dense(nInput, activation = 'sigmoid')(decoded6)
  
      # Combine Encoder and Deocoder layers
      autoencoder = Model(inputs = input_dim, outputs = decoded7)
  
      autoencoder.summary()
      # Compile the Model
      autoencoder.compile(optimizer=OPTIMIZER, loss='binary_crossentropy')
      #autoencoder.compile(loss='mean_squared_error', optimizer = RMSprop())
  
      return autoencoder
  
  X_train, X_test, y_train, y_test = train_test_split(spectra.iloc[:,0:spectra.shape[1]-1], spectra['Class'], test_size=testDatasetSize, stratify=spectra.Class, random_state=seedValue)
  
  X_train, y_train = shuffle(X_train, y_train, random_state=seedValue)
  X_test, y_test = shuffle(X_test, y_test, random_state=seedValue)
  
  X_unseen = X_train.to_numpy()[0:1000,:] # Data not used for training, only for testing
  y_unseen = y_train.to_numpy()[0:1000]
  X_train = X_train.iloc[1000:]
  y_train = y_train.iloc[1000:]
  
  # Scaling
  maxVal = max(X_train)
  X_train = (X_train/maxVal).to_numpy()
  X_test = (X_test/maxVal).to_numpy()
  X_unseen = (X_unseen/maxVal)#.to_numpy()
  
  # Corrupted data
  noise_factor = 0.01
  X_train_noisy = X_train + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_train.shape)
  X_test_noisy = X_test + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_test.shape)
  X_unseen_noisy = X_unseen + noise_factor * np.random.normal(loc=0.0, scale=1.0, size=X_unseen.shape)
  
  ae = BuildModel(X_train.shape[1])
  PrintConsoleLine('Creating model finished')
  print('')
  
  history = ae.fit(X_train_noisy, X_train, epochs=NB_EPOCH, batch_size=BATCH_SIZE, validation_data=[X_test_noisy, X_test])
  save_model(ae, modelFile, overwrite=True)
  
  # Test
  X = X_unseen
  X_noisy = X_unseen_noisy
  X_denoised = ae.predict(X_noisy) # X_train gives the same result (spectra type (1)) !?!
  N = len(X_denoised[0,:])
  index = 6
  PlotDataSimple(3, np.linspace(0,N-1,N), X[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
  PlotDataSimple(4, np.linspace(0,N-1,N), X_noisy[index,:], 'Frequency domain', 'Index', 'Amplitude', None)
  PlotDataSimple(5, np.linspace(0,N-1,N), X_denoised[index, :], 'Frequency domain', 'Index', 'Amplitude', None)
  
  Dataset shape: (17000, 65, 65, 1) (files, samples X axis, samples Y axis, class)
  Train on 12600 samples, validate on 3400 samples
  Epoch 1/3
  12600/12600 [==============================] - 26s 2ms/sample - loss: 0.6813 - val_loss: 0.4913
  Epoch 2/3
  12600/12600 [==============================] - 14s 1ms/sample - loss: 0.1621 - val_loss: 0.0578
  Epoch 3/3
  12600/12600 [==============================] - 16s 1ms/sample - loss: 0.0230 - val_loss: 0.0169
  
  ```
  
  The results I am getting (Column 1 - Initial signal, Column 2 - Corrupted signal, Column 3 - Denoised signal):
  
  [![](https://i.sstatic.net/aXKsr.png)](https://i.sstatic.net/aXKsr.png)
  
  So why does the DAE output the same spectra regardless of the inputs? Am I misunderstanding the DAE principle or is there a problem in my implementation?
  
  **Tags:** machine-learning, tensorflow, python, autoencoders
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2019-12-22. Score: 0 | Accepted answer
  
  You are using Dense layers, try 1d convolution instead.
  Have you tried a different activation function such as softmax and instead of Binary cross entropy try MSE loss? Are all your inputs between 0 and 1? Also, I think your noise amplitude is too much in 2nd and 3rd case as compared to the actual signal. Can you try training on different types of spectra separately and check the result, if the DAE is learning anything. Also, try trining for more epochs.
  
  I am not sure and would have commented this but I can't due to low rep, but I am interested in the solution.
  
  > **Comments:**
  > **username_0**: Thank you for your suggestions. I have tried them and found that the main issue was that I performed scaling by diving to the max value of ALL the spectra. After performing the scaling individually for each spectrum, changing to MSE loss and using LeakyReLU as the activation function, with 100 training epochs, the solution works better. I will try to implement with Conv1D, but I have a hard time understanding how to implement conv nets. I would be very grateful if you could show/help me how to rewrite the model as a convolutional one.  
  > **username_1**: @username_0 [the documentation](https://www.tensorflow.org/api_docs/python/tf/keras/layers/Conv1D?version=stable) will help you with the implementation of convnets. Instead of dense, you just use conv1d layers and pass the size of convolutioton filter with it.  
  
  ---
  
  ### Answer by username_2 on 2021-04-08. Score: 0
  
  If the auto-encoder is converging to the same encoding for different instances, there may be a problem in the loss function. Check the size and shape of the output of the loss function, as it may be getting confused and evaluating the wrong tensors (i.e. you may need to transpose something somewhere).
  
  Basically, assuming you are using an auto-encoder to encode $M$ features of $N$ training instances, your loss function should return $N$ values i.e.,
  ***the size of your loss tensor should be the amount of instances in your training set***.
  
  
  ---
  
  