Answers.Body: <p>I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing.</p>
  
  <p>The NN architecture is too complex for the given task (too deep, too many neurons) and that means that any problems you have with other design choices will tend to get amplified. It could be that your NN is close to diverging on the given data and architecture, and that sigmoid is more resilient to it. </p>
  
  <p>I'd suggest the following changes:</p>
  
  <ul>
  <li><p>Dropping the learning rate, try 0.01 or even 0.001</p></li>
  <li><p>Normalising the two input features. NNs like to work with data that is mean 0, standard deviation 1, although there is some flexibility here, your values ranging from 0 to 6 are probably starting to cause minor problems</p></li>
  <li><p>Look at standard initialisation routines for weights, available within TensorFlow framework, such as Glorot uniform. Your random -1 to +1 is probably too high a range for the given network, and NNs - especially "deep" NNs with 3+ hidden layers - are very sensitive to how initial weights are set.</p></li>
  <li><p>Simplify the network architecture a little. Five hidden layers and 200 neurons seems a bit much for your goal of over-fitting to this small data set. Try something like 3 hidden layers and 50 neurons.</p></li>
  <li><p>Your output layer and loss function are designed for a regression task, but you mention that the goal is to identify classes. You need a softmax layer and multiclass log loss for predicting exclusive classes.</p></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T11:21:48.150Z
Answers.Text: If you have any questions about details of any of the suggestions, then please ask a new, more specific question on AI Stack Exchange or Data Science Stack Exchange. If the question is more about theory/maths then probably better here. If it's about your data or implementations using TensorFlow then Data Science would be better.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T11:29:50.650Z
Answers.Text: tks, i will try the suggested changes
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-13T10:26:11.003Z
Answers.Text: i reduced number of layers, and relu now works! leaky relu too :) tks
Answers.UserDisplayName: Dan D.
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-12T11:20:20.880Z
Answers.LastActivityDate: 2019-09-12T11:36:06.910Z
Answers.LastEditDate: 2019-09-12T11:36:06.910Z
Answers.UserDisplayName: Neil Slater
Body: <p>I have these training data to separate, the classes are rather randomly scattered:</p>
  <p><img src="https://i.sstatic.net/P2RUT.png" alt="" /></p>
  <p>My first attempt was using <code>tf.nn.relu</code> activation function, but output was stuck with whatever number of training steps. So I guessed it could be because of <strong>dead ReLU</strong> units, thus I changed the activation function in hidden layers to <code>tf.nn.leaky_relu</code>, but it's still no good.</p>
  <p>It works when all hidden layers come with <code>tf.sigmoid</code>, yes, but why doesn't ReLU work here? Is it because of dead ReLU units, or exploding gradients, or anything else?</p>
  <p><strong>Source code</strong> (TensorFlow):</p>
  
  <pre><code>#core
  import time;
  
  #libs
  import tensorflow        as tf;
  import matplotlib.pyplot as pyplot;
  
  #mockup to emphasize value name
  def units(Num):
    return Num;
  #end def
  
  #PROGRAMME ENTRY POINT==========================================================
  #data
  #https://i.imgur.com/uVOxZR7.png
  X = [[1,1],[1,2],[1,3],[2,1],[2,2],[2,3],[3,1],[3,2],[3,3],[4,1],[4,2],[4,3],[5,1],[6,1]];
  Y = [[0],  [1],  [0],  [1],  [0],  [1],  [0],  [2],  [1],  [1],  [1],  [0],  [0],  [1]  ];
  Max_X      = 6;
  Max_Y      = 2;
  Batch_Size = 14;
  
  #normalise
  for I in range(len(X)):
    X[I][0] /= Max_X;
    X[I][1] /= Max_X;
    Y[I][0] /= Max_Y;
  #end for
  
  #model
  Input     = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,2]);
  Expected  = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,1]);
  
  #RELU DOESN'T WORK, DEAD RELU? SIGMOID WORKS BUT SLOW.
  #CHANGE TO tf.sigmoid OR tf.tanh AND IT WORKS:
  activation_fn = tf.nn.leaky_relu;
  
  #1
  Weight1   = tf.Variable(tf.random_uniform(shape=[2,units(60)], minval=-1, maxval=1));
  Bias1     = tf.Variable(tf.random_uniform(shape=[  units(60)], minval=-1, maxval=1));
  Hidden1   = activation_fn(tf.matmul(Input,Weight1) + Bias1);
  
  #2
  Weight2   = tf.Variable(tf.random_uniform(shape=[60,units(50)], minval=-1, maxval=1));
  Bias2     = tf.Variable(tf.random_uniform(shape=[   units(50)], minval=-1, maxval=1));
  Hidden2   = activation_fn(tf.matmul(Hidden1,Weight2) + Bias2);
  
  #3
  Weight3   = tf.Variable(tf.random_uniform(shape=[50,units(40)], minval=-1, maxval=1));
  Bias3     = tf.Variable(tf.random_uniform(shape=[   units(40)], minval=-1, maxval=1));
  Hidden3   = activation_fn(tf.matmul(Hidden2,Weight3) + Bias3);
  
  #4
  Weight4   = tf.Variable(tf.random_uniform(shape=[40,units(30)], minval=-1, maxval=1));
  Bias4     = tf.Variable(tf.random_uniform(shape=[   units(30)], minval=-1, maxval=1));
  Hidden4   = activation_fn(tf.matmul(Hidden3,Weight4) + Bias4);
  
  #5
  Weight5   = tf.Variable(tf.random_uniform(shape=[30,units(20)], minval=-1, maxval=1));
  Bias5     = tf.Variable(tf.random_uniform(shape=[   units(20)], minval=-1, maxval=1));
  Hidden5   = activation_fn(tf.matmul(Hidden4,Weight5) + Bias5);
  
  #out
  Weight6   = tf.Variable(tf.random_uniform(shape=[20,units(1)], minval=-1, maxval=1));
  Bias6     = tf.Variable(tf.random_uniform(shape=[   units(1)], minval=-1, maxval=1));
  Output    = tf.sigmoid(tf.matmul(Hidden5,Weight6) + Bias6);
  
  Loss      = tf.reduce_sum(tf.square(Expected-Output));
  Optimiser = tf.train.GradientDescentOptimizer(1e-1);
  Training  = Optimiser.minimize(Loss);
  
  #training
  Sess = tf.Session();
  Init = tf.global_variables_initializer();
  Sess.run(Init);
  
  Feed   = {Input:X, Expected:Y};
  Losses = [];
  Start  = time.time();
  
  for I in range(10000):
    if (I%1000==0):
      Lossvalue = Sess.run(Loss, feed_dict=Feed);
      Losses   += [Lossvalue];
      
      if (I==0):
        print(&quot;Loss:&quot;,Lossvalue,&quot;(first)&quot;);
      else:
        print(&quot;Loss:&quot;,Lossvalue);
    #end if
    
    Sess.run(Training, feed_dict=Feed);
  #end for
  
  Lastloss = Sess.run(Loss, feed_dict=Feed);
  Losses  += [Lastloss];
  print(&quot;Loss:&quot;,Lastloss,&quot;(last)&quot;);
  
  Finish = time.time();
  print(&quot;Time:&quot;,Finish-Start,&quot;seconds&quot;);
  
  #eval
  print(&quot;\nEval:&quot;);
  Evalresults = Sess.run(Output,feed_dict=Feed).tolist();
  for I in range(len(Evalresults)):
    Evalresults[I] = [round(Evalresults[I][0]*Max_Y)];
  #end for
  print(Evalresults);
  Sess.close();
  
  #result: diagram
  print(&quot;\nLoss curve:&quot;);
  pyplot.plot(Losses,&quot;-bo&quot;);
  #eof
  </code></pre>
  
Tags: <tensorflow><gradient-descent><activation-functions><relu><sigmoid>
Title: Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh
UserDisplayName: Dan D.
ThreadText: # Network doesn't converge with ReLU or Leaky ReLU, but works well with sigmoid/tanh
  
  (Asked by: username_0 on 2019-09-12)
  
  I have these training data to separate, the classes are rather randomly scattered:
  
  ![](https://i.sstatic.net/P2RUT.png)
  
  My first attempt was using `tf.nn.relu` activation function, but output was stuck with whatever number of training steps. So I guessed it could be because of **dead ReLU** units, thus I changed the activation function in hidden layers to `tf.nn.leaky_relu`, but it's still no good.
  
  It works when all hidden layers come with `tf.sigmoid`, yes, but why doesn't ReLU work here? Is it because of dead ReLU units, or exploding gradients, or anything else?
  
  **Source code** (TensorFlow):
  
  ```
  #core
  import time;
  
  #libs
  import tensorflow        as tf;
  import matplotlib.pyplot as pyplot;
  
  #mockup to emphasize value name
  def units(Num):
    return Num;
  #end def
  
  #PROGRAMME ENTRY POINT==========================================================
  #data
  #https://i.imgur.com/uVOxZR7.png
  X = [[1,1],[1,2],[1,3],[2,1],[2,2],[2,3],[3,1],[3,2],[3,3],[4,1],[4,2],[4,3],[5,1],[6,1]];
  Y = [[0],  [1],  [0],  [1],  [0],  [1],  [0],  [2],  [1],  [1],  [1],  [0],  [0],  [1]  ];
  Max_X      = 6;
  Max_Y      = 2;
  Batch_Size = 14;
  
  #normalise
  for I in range(len(X)):
    X[I][0] /= Max_X;
    X[I][1] /= Max_X;
    Y[I][0] /= Max_Y;
  #end for
  
  #model
  Input     = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,2]);
  Expected  = tf.placeholder(dtype=tf.float32, shape=[Batch_Size,1]);
  
  #RELU DOESN'T WORK, DEAD RELU? SIGMOID WORKS BUT SLOW.
  #CHANGE TO tf.sigmoid OR tf.tanh AND IT WORKS:
  activation_fn = tf.nn.leaky_relu;
  
  #1
  Weight1   = tf.Variable(tf.random_uniform(shape=[2,units(60)], minval=-1, maxval=1));
  Bias1     = tf.Variable(tf.random_uniform(shape=[  units(60)], minval=-1, maxval=1));
  Hidden1   = activation_fn(tf.matmul(Input,Weight1) + Bias1);
  
  #2
  Weight2   = tf.Variable(tf.random_uniform(shape=[60,units(50)], minval=-1, maxval=1));
  Bias2     = tf.Variable(tf.random_uniform(shape=[   units(50)], minval=-1, maxval=1));
  Hidden2   = activation_fn(tf.matmul(Hidden1,Weight2) + Bias2);
  
  #3
  Weight3   = tf.Variable(tf.random_uniform(shape=[50,units(40)], minval=-1, maxval=1));
  Bias3     = tf.Variable(tf.random_uniform(shape=[   units(40)], minval=-1, maxval=1));
  Hidden3   = activation_fn(tf.matmul(Hidden2,Weight3) + Bias3);
  
  #4
  Weight4   = tf.Variable(tf.random_uniform(shape=[40,units(30)], minval=-1, maxval=1));
  Bias4     = tf.Variable(tf.random_uniform(shape=[   units(30)], minval=-1, maxval=1));
  Hidden4   = activation_fn(tf.matmul(Hidden3,Weight4) + Bias4);
  
  #5
  Weight5   = tf.Variable(tf.random_uniform(shape=[30,units(20)], minval=-1, maxval=1));
  Bias5     = tf.Variable(tf.random_uniform(shape=[   units(20)], minval=-1, maxval=1));
  Hidden5   = activation_fn(tf.matmul(Hidden4,Weight5) + Bias5);
  
  #out
  Weight6   = tf.Variable(tf.random_uniform(shape=[20,units(1)], minval=-1, maxval=1));
  Bias6     = tf.Variable(tf.random_uniform(shape=[   units(1)], minval=-1, maxval=1));
  Output    = tf.sigmoid(tf.matmul(Hidden5,Weight6) + Bias6);
  
  Loss      = tf.reduce_sum(tf.square(Expected-Output));
  Optimiser = tf.train.GradientDescentOptimizer(1e-1);
  Training  = Optimiser.minimize(Loss);
  
  #training
  Sess = tf.Session();
  Init = tf.global_variables_initializer();
  Sess.run(Init);
  
  Feed   = {Input:X, Expected:Y};
  Losses = [];
  Start  = time.time();
  
  for I in range(10000):
    if (I%1000==0):
      Lossvalue = Sess.run(Loss, feed_dict=Feed);
      Losses   += [Lossvalue];
      
      if (I==0):
        print("Loss:",Lossvalue,"(first)");
      else:
        print("Loss:",Lossvalue);
    #end if
    
    Sess.run(Training, feed_dict=Feed);
  #end for
  
  Lastloss = Sess.run(Loss, feed_dict=Feed);
  Losses  += [Lastloss];
  print("Loss:",Lastloss,"(last)");
  
  Finish = time.time();
  print("Time:",Finish-Start,"seconds");
  
  #eval
  print("\nEval:");
  Evalresults = Sess.run(Output,feed_dict=Feed).tolist();
  for I in range(len(Evalresults)):
    Evalresults[I] = [round(Evalresults[I][0]*Max_Y)];
  #end for
  print(Evalresults);
  Sess.close();
  
  #result: diagram
  print("\nLoss curve:");
  pyplot.plot(Losses,"-bo");
  #eof
  
  ```
  
  **Tags:** tensorflow, gradient-descent, activation-functions, relu, sigmoid
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2019-09-12. Score: 3 | Accepted answer
  
  I don't think it is dead ReLU units as a main cause, although they may be happening as part of the NN failing.
  
  The NN architecture is too complex for the given task (too deep, too many neurons) and that means that any problems you have with other design choices will tend to get amplified. It could be that your NN is close to diverging on the given data and architecture, and that sigmoid is more resilient to it.
  
  I'd suggest the following changes:
  
  * Dropping the learning rate, try 0.01 or even 0.001
  * Normalising the two input features. NNs like to work with data that is mean 0, standard deviation 1, although there is some flexibility here, your values ranging from 0 to 6 are probably starting to cause minor problems
  * Look at standard initialisation routines for weights, available within TensorFlow framework, such as Glorot uniform. Your random -1 to +1 is probably too high a range for the given network, and NNs - especially "deep" NNs with 3+ hidden layers - are very sensitive to how initial weights are set.
  * Simplify the network architecture a little. Five hidden layers and 200 neurons seems a bit much for your goal of over-fitting to this small data set. Try something like 3 hidden layers and 50 neurons.
  * Your output layer and loss function are designed for a regression task, but you mention that the goal is to identify classes. You need a softmax layer and multiclass log loss for predicting exclusive classes.
  
  > **Comments:**
  > **username_1**: If you have any questions about details of any of the suggestions, then please ask a new, more specific question on AI Stack Exchange or Data Science Stack Exchange. If the question is more about theory/maths then probably better here. If it's about your data or implementations using TensorFlow then Data Science would be better.  
  > **username_0**: tks, i will try the suggested changes  
  > **username_0**: i reduced number of layers, and relu now works! leaky relu too :) tks  
  
  ---
  
  