Answers.Body: <p>You should try experimenting with a lower learning rate as a starting point. You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-08-27T08:52:06.277Z
Answers.LastActivityDate: 2019-08-27T08:52:06.277Z
Answers.UserDisplayName: razvanc92
Body: <p>I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:
  <a href="https://i.sstatic.net/GWP60.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/GWP60.png" alt="enter image description here"></a></p>
  
  <p>Data I use are typical standardised tabular numbers. The architecture is as follows:</p>
  
  <pre><code>model.add(Dropout(0.2))
  model.add(Dense(units=512, activation='relu'))
  model.add(Dropout(0.3))
  model.add(Dense(units=256, activation='relu'))
  model.add(Dropout(0.3))
  model.add(Dense(units=128, activation='relu'))
  model.add(Dense(units=128, activation='relu'))
  model.add(Dense(units=1))
  
  adam = optimizers.Adam(lr=0.1)
  
  model.compile(loss='mean_squared_error', optimizer=adam)
  
  reduce_lr = ReduceLROnPlateau(
      monitor='val_loss',
      factor=0.9,
      patience=10,
      min_lr=0.0001,
      verbose=1)
  
  tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))
  
  history = model.fit(
      x_train,
      y_train,
      epochs=500,
      verbose=10,
      batch_size=128,
      callbacks=[reduce_lr, tensorboard],
      validation_split=0.1)
  </code></pre>
  
  <p>It seems to me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value, but I am not sure.</p>
  
  <p>I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T07:31:12.010Z
Comments.Text: Have you tried experimenting with a lower learning rate as a starting point? You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.
Comments.UserDisplayName: razvanc92
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T08:18:45.283Z
Comments.Text: I have now and it worked out! Lowering initial learning rate to 0.005 eliminated straight line as an output. You can post it as an answer :) Thanks!
Comments.UserDisplayName: GKozinski
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-08-27T08:26:03.077Z
Comments.Text: and if you have some idea why this happens so, with greater learning rate, please do share as well.
Comments.UserDisplayName: GKozinski
Tags: <neural-networks><training><tensorflow><keras>
Title: Why do I get a straight line as an output from a neural network?
UserDisplayName: GKozinski
ThreadText: # Why do I get a straight line as an output from a neural network?
  
  (Asked by: username_0 on 2019-08-26)
  
  I am using feedforward neural network for regression and what I get as a result of prediction is a constant value visible on the graph below:
  [![enter image description here](https://i.sstatic.net/GWP60.png)](https://i.sstatic.net/GWP60.png)
  
  Data I use are typical standardised tabular numbers. The architecture is as follows:
  
  ```
  model.add(Dropout(0.2))
  model.add(Dense(units=512, activation='relu'))
  model.add(Dropout(0.3))
  model.add(Dense(units=256, activation='relu'))
  model.add(Dropout(0.3))
  model.add(Dense(units=128, activation='relu'))
  model.add(Dense(units=128, activation='relu'))
  model.add(Dense(units=1))
  
  adam = optimizers.Adam(lr=0.1)
  
  model.compile(loss='mean_squared_error', optimizer=adam)
  
  reduce_lr = ReduceLROnPlateau(
      monitor='val_loss',
      factor=0.9,
      patience=10,
      min_lr=0.0001,
      verbose=1)
  
  tensorboard = TensorBoard(log_dir="logs\{}".format(NAME))
  
  history = model.fit(
      x_train,
      y_train,
      epochs=500,
      verbose=10,
      batch_size=128,
      callbacks=[reduce_lr, tensorboard],
      validation_split=0.1)
  
  ```
  
  It seems to me that all weights are zeroed and only constant bias is present here, since for different data samples from a test set I get the same value, but I am not sure.
  
  I understand that the algorithm has found smallest MSE for such a constant value, but is there a way of avoiding such situation, since straight line is not really good solution for my project?
  
  **Tags:** neural-networks, training, tensorflow, keras
  
  > **Comments:**
  > **username_1**: Have you tried experimenting with a lower learning rate as a starting point? You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.  
  > **username_0**: I have now and it worked out! Lowering initial learning rate to 0.005 eliminated straight line as an output. You can post it as an answer :) Thanks!  
  > **username_0**: and if you have some idea why this happens so, with greater learning rate, please do share as well.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-08-27. Score: 1 | Accepted answer
  
  You should try experimenting with a lower learning rate as a starting point. You're starting with 0.1 which is quite high for most of the cases, and only reduce by 0.9 which is not much.
  
  
  ---
  
  