Answers.Body: <p>Could you post the pseudocode of your backpropagation algorithm?
  I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it works and then continue adding things. If it's your first time with neural networks, I recommend you check this <a href="http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html" rel="nofollow noreferrer">link</a> out and you could also try practising the algorithms on a programming language like Octave/Matlab (it can be very efficient speed wise). Also check this question out <a href="https://ai.stackexchange.com/questions/19936/is-my-backpropagation-code-correct">(link)</a>. At the bottom there is a code example for the XOR problem. Please post the pseudocode of your code instead of just dumping it there.  Finally, don't just copy paste algorithms into your code, you need to understand them.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T14:59:22.973Z
Answers.Text: Hi, thanks for the reply. All the code inside my python project was written by me, i had to study the maths and everything behind it, so it took a lot of time, but everything worked well (with sigmoid). Apparently something is missing. I did start out with simple things and just then i passed to the mnist classification task. My problem now is that i dont get the expected results using other activations other than sigmoid, but with that, is working pretty neat so i was wondering if it was something specific to my code that was causing the error. I will edit my question with the pseudo code
Answers.UserDisplayName: Alessandro Valentino
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T15:15:35.803Z
Answers.Text: I have only tried using the sigmoid and the step function so I can't really help you with the other ones.  But you can check this link: https://en.wikipedia.org/wiki/Activation_function .
Answers.UserDisplayName: david david
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T06:44:31.117Z
Answers.Text: So, i tried using Yexpected-Youtput (as the cost derivative to calculate gradients) instead of Youtput-Yexpected. The results become as bad as relu with all activation functions
Answers.UserDisplayName: Alessandro Valentino
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T08:37:36.187Z
Answers.Text: As far as I know you don't need to find the derivative of the cost function. Only the activation function one. Check this link out if you are still unsure about backpropagation. I think that is your issue. http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html. You probably didn't convert the maths to the algorithms well enough and left something out
Answers.UserDisplayName: david david
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T08:57:25.633Z
Answers.Text: From what i understand, the cost function is used to find the cost of the network, but to optimize the whole thing in backpropagation you need the slope of the cost function, that tells you what direction and how much you should modify your weights. The mean squared cost function that i mentioned is the mean of the sum of  (Youtput - Yexpected) ^ 2 applied to each output neuron, which derivative is 2*(Youtput - Yexpected) or just (Youtput - Yexpected) since 2 in this context becomes irrelevant because it is a constant. Unless this is false, i don't think the problem is in the cost function.
Answers.UserDisplayName: Alessandro Valentino
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-08T09:07:21.490Z
Answers.Text: I'm sorry but if this is a problem with your code I can't really help you out. I don't know python. But anyway maybe you could plot a graph with your results after, I don't know, 500 epochs ...
Answers.UserDisplayName: david david
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-04-07T11:56:22.693Z
Answers.LastActivityDate: 2020-04-07T11:56:22.693Z
Answers.UserDisplayName: david david
Body: <p>I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to implement SGD in a neural network that i trained to recognize the classic MNIST digits dataset.
  As activation function i always used sigmoid for both hidden and output neurons, and everything seems to work more or less ok, but now i wanted to step it up a bit and try to let SGD operate with different activations, so i added 2 other  functions to my code: relu and tanh.
  The behaviours that i expected based on articles, documentation and "tutorials" found online were:<br>
  <strong>tanh</strong>: Should be slightly better than sigmoid<br>
  <strong>relu</strong>: should be much better than sigmoid and tanh<br>
  (By better i mean faster or at least higher accuracy the the end, or a mix of both)</p>
  
  <p>Using tanh it looks like it's much slower converging to a minimum compared to sigmoid</p>
  
  <p>Using relu...well, the results were very, VERY horrible
  Here's the outputs with the different activations (Learning rate: 0.1, Epochs: 5, MiniBatch size: 10, Momentum: 0.9)</p>
  
  <p><strong>Sigmoid training</strong></p>
  
  <pre>
  [Sigmoid for hidden layers, sigmoid for output layer]
  Epoch: 1/5 (14.3271 s): Loss: 0.0685, Accuracy: 0.6231, Learning rate: 0.10000
  Epoch: 2/5 (14.0060 s): Loss: 0.0503, Accuracy: 0.6281, Learning rate: 0.10000
  Epoch: 3/5 (14.0081 s): Loss: 0.0482, Accuracy: 0.6382, Learning rate: 0.10000
  Epoch: 4/5 (13.8516 s): Loss: 0.0471, Accuracy: 0.7085, Learning rate: 0.10000
  Epoch: 5/5 (13.9411 s): Loss: 0.0374, Accuracy: 0.7990, Learning rate: 0.10000
  </pre>
  
  <p><strong>Tanh training</strong></p>
  
  <pre>
  [Tanh for hidden layers, sigmoid for output layer]
  Epoch: 1/5 (13.7553 s): Loss: 0.3708, Accuracy: 0.4171, Learning rate: 0.10000
  Epoch: 2/5 (13.7666 s): Loss: 0.2580, Accuracy: 0.4623, Learning rate: 0.10000
  Epoch: 3/5 (13.5550 s): Loss: 0.2289, Accuracy: 0.4824, Learning rate: 0.10000
  Epoch: 4/5 (13.7311 s): Loss: 0.2211, Accuracy: 0.5729, Learning rate: 0.10000
  Epoch: 5/5 (13.6996 s): Loss: 0.2142, Accuracy: 0.5779, Learning rate: 0.10000
  </pre>
  
  <p><strong>Relu training</strong></p>
  
  <pre>
  [Relu for hidden layers, sigmoid for output layer]
  Epoch: 1/5 (14.2100 s): Loss: 0.7725, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 2/5 (14.6218 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 3/5 (14.2116 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 4/5 (14.1657 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 5/5 (14.1427 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  </pre>
  
  <p><strong>Another run with relu</strong></p>
  
  <pre>
  Epoch: 1/5 (14.7391 s): Loss: 15.4055, Accuracy: 0.1658, Learning rate: 0.10000
  Epoch: 2/5 (14.8203 s): Loss: 59.2707, Accuracy: 0.1709, Learning rate: 0.10000
  Epoch: 3/5 (15.3785 s): Loss: 166.1310, Accuracy: 0.1407, Learning rate: 0.10000
  Epoch: 4/5 (14.9285 s): Loss: 109.9386, Accuracy: 0.1859, Learning rate: 0.10000
  Epoch: 5/5 (15.1280 s): Loss: 158.9268, Accuracy: 0.1859, Learning rate: 0.10000
  </pre>
  
  <p>For these examples the epochs are just 5 but incrementing the epochs the results dont change, tanh and relu for me perform worse than sigmoid.</p>
  
  <p>Here is my python code reference for SGD:<br></p>
  
  <p><a href="https://pastebin.com/Nhp2vCg6" rel="nofollow noreferrer">SGD with momentum</a><br></p>
  
  <p>This method was created to accept different activation functions to dynamically use them when creating the neural network object</p>
  
  <p>The activation functions and their derivatives:</p>
  
  <p><a href="https://pastebin.com/YjhAKfVd" rel="nofollow noreferrer">Activation functions and derivatives</a></p>
  
  <p>The loss function i used is the mean squared error:</p>
  
  <pre>
  def mean_squared(output, expected_result):
      return numpy.sum((output - expected_result) ** 2) / expected_result.shape[0]
  
  
  def mean_squared_derivative(output, expected_result):
      return output - expected_result
  
  </pre>
  
  <p>Is there some concept i am missing? Am i using the activation functions the wrong way? I really cannot find the answer to this even after searching for a long time.
  I feel like the problem is somewhere in the backpropagation but i can't find it.
  Any kind of help would be greatly appriciated</p>
  
  <p>PS: I hope i posted this in the right place, i am pretty new to asking questions here, so if there is any problem i will move the question somewhere else</p>
  
  <h2>Edit:</h2>
  
  <p>I tried to implement this with tensorflow, using relu for hidden layers and sigmoid for output. The results i get with this implementation are the same as the ones i mentioned in my question, so unless i am doing something wrong in both situations i am left to think i cannot use relu with sigmoid, which makes sense cause relu can have very high values while sigmoid pushes them down between 0 and 1, therefore most of the times giving values very close to 1.<br>
  Code reference:<br>
  <a href="https://pastebin.com/CiaNat0j" rel="nofollow noreferrer">TensorFlow implementation</a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-07T22:18:32.913Z
Comments.Text: Did you try a lower learning rate for relu ?
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-04-07T22:21:59.047Z
Comments.Text: I tried any kind of learning rate, from 0.000001 to 10, also with schedules, but nothing improves
Comments.UserDisplayName: Alessandro Valentino
Tags: <neural-networks><python><training><backpropagation><stochastic-gradient-descent>
Title: Stochastic gradient descent does not behave as expected, even with different activation functions
UserDisplayName: Alessandro Valentino
ThreadText: # Stochastic gradient descent does not behave as expected, even with different activation functions
  
  (Asked by: username_0 on 2020-04-06)
  
  I have been working on my own AI for a while now, trying to implemented SGD with momentum from scratch in python. After looking around and studying all the maths behind it, i finally managed to implement SGD in a neural network that i trained to recognize the classic MNIST digits dataset.
  As activation function i always used sigmoid for both hidden and output neurons, and everything seems to work more or less ok, but now i wanted to step it up a bit and try to let SGD operate with different activations, so i added 2 other  functions to my code: relu and tanh.
  The behaviours that i expected based on articles, documentation and "tutorials" found online were:
  tanh: Should be slightly better than sigmoid
  relu: should be much better than sigmoid and tanh
  (By better i mean faster or at least higher accuracy the the end, or a mix of both)
  Using tanh it looks like it's much slower converging to a minimum compared to sigmoid
  Using relu...well, the results were very, VERY horrible
  Here's the outputs with the different activations (Learning rate: 0.1, Epochs: 5, MiniBatch size: 10, Momentum: 0.9)
  Sigmoid training
  
  [Sigmoid for hidden layers, sigmoid for output layer]
  Epoch: 1/5 (14.3271 s): Loss: 0.0685, Accuracy: 0.6231, Learning rate: 0.10000
  Epoch: 2/5 (14.0060 s): Loss: 0.0503, Accuracy: 0.6281, Learning rate: 0.10000
  Epoch: 3/5 (14.0081 s): Loss: 0.0482, Accuracy: 0.6382, Learning rate: 0.10000
  Epoch: 4/5 (13.8516 s): Loss: 0.0471, Accuracy: 0.7085, Learning rate: 0.10000
  Epoch: 5/5 (13.9411 s): Loss: 0.0374, Accuracy: 0.7990, Learning rate: 0.10000
  
  Tanh training
  
  [Tanh for hidden layers, sigmoid for output layer]
  Epoch: 1/5 (13.7553 s): Loss: 0.3708, Accuracy: 0.4171, Learning rate: 0.10000
  Epoch: 2/5 (13.7666 s): Loss: 0.2580, Accuracy: 0.4623, Learning rate: 0.10000
  Epoch: 3/5 (13.5550 s): Loss: 0.2289, Accuracy: 0.4824, Learning rate: 0.10000
  Epoch: 4/5 (13.7311 s): Loss: 0.2211, Accuracy: 0.5729, Learning rate: 0.10000
  Epoch: 5/5 (13.6996 s): Loss: 0.2142, Accuracy: 0.5779, Learning rate: 0.10000
  
  Relu training
  
  [Relu for hidden layers, sigmoid for output layer]
  Epoch: 1/5 (14.2100 s): Loss: 0.7725, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 2/5 (14.6218 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 3/5 (14.2116 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 4/5 (14.1657 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  Epoch: 5/5 (14.1427 s): Loss: 0.1000, Accuracy: 0.0854, Learning rate: 0.10000
  
  Another run with relu
  
  Epoch: 1/5 (14.7391 s): Loss: 15.4055, Accuracy: 0.1658, Learning rate: 0.10000
  Epoch: 2/5 (14.8203 s): Loss: 59.2707, Accuracy: 0.1709, Learning rate: 0.10000
  Epoch: 3/5 (15.3785 s): Loss: 166.1310, Accuracy: 0.1407, Learning rate: 0.10000
  Epoch: 4/5 (14.9285 s): Loss: 109.9386, Accuracy: 0.1859, Learning rate: 0.10000
  Epoch: 5/5 (15.1280 s): Loss: 158.9268, Accuracy: 0.1859, Learning rate: 0.10000
  
  For these examples the epochs are just 5 but incrementing the epochs the results dont change, tanh and relu for me perform worse than sigmoid.
  Here is my python code reference for SGD:
  SGD with momentum
  This method was created to accept different activation functions to dynamically use them when creating the neural network object
  The activation functions and their derivatives:
  Activation functions and derivatives
  The loss function i used is the mean squared error:
  
  def mean_squared(output, expected_result):
      return numpy.sum((output - expected_result) ** 2) / expected_result.shape[0]
  
  
  def mean_squared_derivative(output, expected_result):
      return output - expected_result
  
  
  Is there some concept i am missing? Am i using the activation functions the wrong way? I really cannot find the answer to this even after searching for a long time.
  I feel like the problem is somewhere in the backpropagation but i can't find it.
  Any kind of help would be greatly appriciated
  PS: I hope i posted this in the right place, i am pretty new to asking questions here, so if there is any problem i will move the question somewhere else
  Edit:
  I tried to implement this with tensorflow, using relu for hidden layers and sigmoid for output. The results i get with this implementation are the same as the ones i mentioned in my question, so unless i am doing something wrong in both situations i am left to think i cannot use relu with sigmoid, which makes sense cause relu can have very high values while sigmoid pushes them down between 0 and 1, therefore most of the times giving values very close to 1.
  Code reference:
  TensorFlow implementation
  
  **Tags:** neural-networks, python, training, backpropagation, stochastic-gradient-descent
  > **Comments:**
  > **username_1**: Did you try a lower learning rate for relu ?  
  > **username_0**: I tried any kind of learning rate, from 0.000001 to 10, also with schedules, but nothing improves  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-04-07. Score: 1
  
  Could you post the pseudocode of your backpropagation algorithm?
  I recommend you start off as simple as possible (this includes your cost f(x), I would simply use Yexpected-Youtput) and see if it works and then continue adding things. If it's your first time with neural networks, I recommend you check this [link](http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html) out and you could also try practising the algorithms on a programming language like Octave/Matlab (it can be very efficient speed wise). Also check this question out [(link)](https://ai.stackexchange.com/questions/19936/is-my-backpropagation-code-correct). At the bottom there is a code example for the XOR problem. Please post the pseudocode of your code instead of just dumping it there. Finally, don't just copy paste algorithms into your code, you need to understand them.
  
  > **Comments:**
  > **username_0**: Hi, thanks for the reply. All the code inside my python project was written by me, i had to study the maths and everything behind it, so it took a lot of time, but everything worked well (with sigmoid). Apparently something is missing. I did start out with simple things and just then i passed to the mnist classification task. My problem now is that i dont get the expected results using other activations other than sigmoid, but with that, is working pretty neat so i was wondering if it was something specific to my code that was causing the error. I will edit my question with the pseudo code  
  > **username_2**: I have only tried using the sigmoid and the step function so I can't really help you with the other ones.  But you can check this link: https://en.wikipedia.org/wiki/Activation_function .  
  > **username_0**: So, i tried using Yexpected-Youtput (as the cost derivative to calculate gradients) instead of Youtput-Yexpected. The results become as bad as relu with all activation functions  
  > **username_2**: As far as I know you don't need to find the derivative of the cost function. Only the activation function one. Check this link out if you are still unsure about backpropagation. I think that is your issue. http://home.agh.edu.pl/~vlsi/AI/backp_t_en/backprop.html. You probably didn't convert the maths to the algorithms well enough and left something out  
  > **username_0**: From what i understand, the cost function is used to find the cost of the network, but to optimize the whole thing in backpropagation you need the slope of the cost function, that tells you what direction and how much you should modify your weights. The mean squared cost function that i mentioned is the mean of the sum of  (Youtput - Yexpected) ^ 2 applied to each output neuron, which derivative is 2*(Youtput - Yexpected) or just (Youtput - Yexpected) since 2 in this context becomes irrelevant because it is a constant. Unless this is false, i don't think the problem is in the cost function.  
  > **username_2**: I'm sorry but if this is a problem with your code I can't really help you out. I don't know python. But anyway maybe you could plot a graph with your results after, I don't know, 500 epochs ...  
  
  ---
  
  