Answers.Body: <p>My closest guess is because you are using the activation function ReLU, which pushes data to be greater than zero. However, because of your data's nature, the autoencoder is highly dependent on negative calculations to reconstruct your data, but the best it can achieve is zero.</p>
  <blockquote>
  <p>In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument:
  <span class="math-container">$$f(x) = x^+ = max(0, x)$$</span>
  <a href="https://en.wikipedia.org/wiki/Rectifier_(neural_networks)" rel="nofollow noreferrer">https://en.wikipedia.org/wiki/Rectifier_(neural_networks)</a></p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T15:30:25.050Z
Answers.Text: I'm not sure I fully understand your answer. 
  
  I do understand that relu pushes any negative numbers to zero, however my data contains no negative number so what's the logic behind the bottleneck layer predicting negative numbers?
  
  Anyway, I will try to use LeakyReLu as activation wich should in theory fix the problem right? Do you suggest I try this change only on the bottleneck layer or on all layers?
Answers.UserDisplayName: beerzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T16:08:39.547Z
Answers.Text: I've tried the following: 1) Convert my dataset to ]0,~620000]; 2) Apply log2 transformation converting it to [-23.25, 19.24]. My hope is that changing to activations that accept negative numbers will fix it, but I still have the same issue.
Answers.UserDisplayName: beerzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T18:05:54.027Z
Answers.Text: I mean, the auto encoder probably wanted to calculate negative outputs in hidden layers but the relu won't let it to be. Have you tried to use linear activation?
Answers.UserDisplayName: kiarash_kiani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T18:14:03.813Z
Answers.Text: Try to change all activation function in all layers
Answers.UserDisplayName: kiarash_kiani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T18:20:51.647Z
Answers.Text: Does decreasing bottleneck size affects reconstruction error?
Answers.UserDisplayName: kiarash_kiani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T18:36:10.540Z
Answers.Text: Indeed you were right, I tried to change all the activations to linear and the nodes predicting all zeroes are predicting negative numbers. I don't understand, however, why it predicts negative weights if my dataset contains none, but maybe I shouldn't worry about that as long as I can minimize my error.
  Regarding decreasing the bottleneck size it worsens the reconstruction error in general. I will accept your answer, but will let it open a bit more if someone wants to jump in and explain why it is happening.
Answers.UserDisplayName: beerzy
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T03:23:53.040Z
Answers.Text: I'm glad to hear that. When you are using an autoencoder, you are doing dimension reduction—you can see it as a linear or non-Linear PCA. This means you are transforming data from one vector space (input layer) to another vector Space (bottleneck layer). Your new representation is calculated by gradient descent. So it is acceptable that you get negative features on the bottleneck (after transformation, in the new vector space) while you don't have any negative number in the input layer which is your original vector space.
Answers.UserDisplayName: kiarash_kiani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T03:30:50.420Z
Answers.Text: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/116839/discussion-between-kiarash-kiani-and-beerzy).
Answers.UserDisplayName: kiarash_kiani
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-11-30T07:25:07.790Z
Answers.LastActivityDate: 2020-11-30T07:25:07.790Z
Answers.UserDisplayName: kiarash_kiani
Body: <p>I'm using tf.Keras to build a deep-fully connected autoencoder. My input dataset is a dataframe with shape (19947,), and the purpose of the autoencoder is to predict normalized gene expression values. They are continuous values that range from [0,~620000].</p>
  <p>I tried different architectures and I'm using relu activation for all layers. To optimize I'm using adam with mae loss.</p>
  <p>The problem I have is the network trains successfully (although the train loss is still terrible) but when I'm predicting I notice that although the predictions do make sense for some nodes, there are always a certain number of nodes that only output 0. I've tried changing the number of nodes of my bottleneck layer (output) and it always happens even when I decrease the output number.</p>
  <p>Any ideas on what I'm doing wrong?</p>
  <p><em>tf.Keras code:</em></p>
  <pre><code>input_layer = keras.Input(shape=(19947,))
  simple_encoder = keras.models.Sequential([
      input_layer,
      keras.layers.Dense(512, activation='relu'),
      keras.layers.Dense(128, activation='relu'),
      keras.layers.Dense(16, activation='relu')
  ])
  simple_decoder = keras.models.Sequential([
      keras.layers.Dense(128, activation='relu'),
      keras.layers.Dense(512, activation='relu'),
      keras.layers.Dense(19947, activation='relu')
  ])
  simple_ae = keras.models.Sequential([simple_encoder, simple_decoder])
  simple_ae.compile(optimizer='adam', loss='mae')
  simple_ae.fit(X_train, X_train,
                epochs=1000,
                validation_data=(X_valid, X_valid),
                callbacks=[early_stopping])
  </code></pre>
  <p><em>Output of encoder.predict with 16 nodes on the bottleneck layer. 7 nodes predict only 0's and 8 nodes predict &quot;correctly&quot;</em>
  <a href="https://i.sstatic.net/rBpEX.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/rBpEX.png" alt="Output from encoder.predict with 16 nodes on bottleneck layer" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-29T22:48:17.297Z
Comments.Text: Have you done any normalization of the data?  The weights and biases will have a hard time influencing a number with a magnitude like 620000.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-11-29T23:42:06.927Z
Comments.Text: The data is already normalized using a specific algorithm to normalize genomic expression data, I could normalize on top of that using min-max or standard scaler but I'm not sure that would be accurate
Comments.UserDisplayName: beerzy
Tags: <python><tensorflow><keras><autoencoders><bottlenecks>
Title: Autoencoder: predictions missing for nodes in the bottleneck layer
UserDisplayName: beerzy
ThreadText: # Autoencoder: predictions missing for nodes in the bottleneck layer
  
  (Asked by: username_0 on 2020-11-29)
  
  I'm using tf.Keras to build a deep-fully connected autoencoder. My input dataset is a dataframe with shape (19947,), and the purpose of the autoencoder is to predict normalized gene expression values. They are continuous values that range from [0,~620000].
  
  I tried different architectures and I'm using relu activation for all layers. To optimize I'm using adam with mae loss.
  
  The problem I have is the network trains successfully (although the train loss is still terrible) but when I'm predicting I notice that although the predictions do make sense for some nodes, there are always a certain number of nodes that only output 0. I've tried changing the number of nodes of my bottleneck layer (output) and it always happens even when I decrease the output number.
  
  Any ideas on what I'm doing wrong?
  
  *tf.Keras code:*
  
  ```
  input_layer = keras.Input(shape=(19947,))
  simple_encoder = keras.models.Sequential([
      input_layer,
      keras.layers.Dense(512, activation='relu'),
      keras.layers.Dense(128, activation='relu'),
      keras.layers.Dense(16, activation='relu')
  ])
  simple_decoder = keras.models.Sequential([
      keras.layers.Dense(128, activation='relu'),
      keras.layers.Dense(512, activation='relu'),
      keras.layers.Dense(19947, activation='relu')
  ])
  simple_ae = keras.models.Sequential([simple_encoder, simple_decoder])
  simple_ae.compile(optimizer='adam', loss='mae')
  simple_ae.fit(X_train, X_train,
                epochs=1000,
                validation_data=(X_valid, X_valid),
                callbacks=[early_stopping])
  
  ```
  
  *Output of encoder.predict with 16 nodes on the bottleneck layer. 7 nodes predict only 0's and 8 nodes predict "correctly"*
  [![Output from encoder.predict with 16 nodes on bottleneck layer](https://i.sstatic.net/rBpEX.png)](https://i.sstatic.net/rBpEX.png)
  
  **Tags:** python, tensorflow, keras, autoencoders, bottlenecks
  
  > **Comments:**
  > **username_1**: Have you done any normalization of the data?  The weights and biases will have a hard time influencing a number with a magnitude like 620000.  
  > **username_0**: The data is already normalized using a specific algorithm to normalize genomic expression data, I could normalize on top of that using min-max or standard scaler but I'm not sure that would be accurate  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-11-30. Score: 0 | Accepted answer
  
  My closest guess is because you are using the activation function ReLU, which pushes data to be greater than zero. However, because of your data's nature, the autoencoder is highly dependent on negative calculations to reconstruct your data, but the best it can achieve is zero.
  
  > In the context of artificial neural networks, the rectifier is an activation function defined as the positive part of its argument:
  > $$f(x) = x^+ = max(0, x)$$
  > <https://en.wikipedia.org/wiki/Rectifier_(neural_networks)>
  
  > **Comments:**
  > **username_0**: I'm not sure I fully understand your answer.   I do understand that relu pushes any negative numbers to zero, however my data contains no negative number so what's the logic behind the bottleneck layer predicting negative numbers?  Anyway, I will try to use LeakyReLu as activation wich should in theory fix the problem right? Do you suggest I try this change only on the bottleneck layer or on all layers?  
  > **username_0**: I've tried the following: 1) Convert my dataset to ]0,~620000]; 2) Apply log2 transformation converting it to [-23.25, 19.24]. My hope is that changing to activations that accept negative numbers will fix it, but I still have the same issue.  
  > **username_2**: I mean, the auto encoder probably wanted to calculate negative outputs in hidden layers but the relu won't let it to be. Have you tried to use linear activation?  
  > **username_2**: Try to change all activation function in all layers  
  > **username_2**: Does decreasing bottleneck size affects reconstruction error?  
  > **username_0**: Indeed you were right, I tried to change all the activations to linear and the nodes predicting all zeroes are predicting negative numbers. I don't understand, however, why it predicts negative weights if my dataset contains none, but maybe I shouldn't worry about that as long as I can minimize my error. Regarding decreasing the bottleneck size it worsens the reconstruction error in general. I will accept your answer, but will let it open a bit more if someone wants to jump in and explain why it is happening.  
  > **username_2**: I'm glad to hear that. When you are using an autoencoder, you are doing dimension reduction—you can see it as a linear or non-Linear PCA. This means you are transforming data from one vector space (input layer) to another vector Space (bottleneck layer). Your new representation is calculated by gradient descent. So it is acceptable that you get negative features on the bottleneck (after transformation, in the new vector space) while you don't have any negative number in the input layer which is your original vector space.  
  > **username_2**: Let us [continue this discussion in chat](https://chat.stackexchange.com/rooms/116839/discussion-between-username_2-and-username_0).  
  
  ---
  
  