Answers.Body: <p>You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value.</p>
  <p>See if the results has changed after adjusting the learning rate.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T08:13:05.803Z
Answers.Text: I tried that as well, What do you suggest my learning rate should be?
Answers.UserDisplayName: Siddarth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T08:18:24.013Z
Answers.Text: You can try to set the learning rate to 0.01 or 0.1 to see if the results of outcome is better or not
Answers.UserDisplayName: Oscar916
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T08:49:59.533Z
Answers.Text: I tried 0.01, 0.1 and even 1 what I have noticed is the rate at which the cost function is decreasing is good but the problem is it still getting plateaued at 0.64 after 2500 epochs
Answers.UserDisplayName: Siddarth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-16T08:04:33.927Z
Answers.LastActivityDate: 2020-07-16T08:04:33.927Z
Answers.UserDisplayName: Oscar916
Answers.Body: <p>I would try at least a dozen of random architectures, there's a interesting research paper suggesting randomization (can't find It right now, but believe me I read it) is the best bet to find optimal hyperparameters. And indeed in my opensource project it worked like a charm. The first attempt likes half of the neural networks performed at 10% accuracy on MNIST task( same as randomly guessing a digit), I removed from random selection parameters that seemed to not work (in example I removed layers with 9 and 16 neurons and Sigmoid activation function) and tried again the following time i obtained many networks above 75% accuracy. I even obtained a Small Fully Connected network that performs 87,1% accuracy with 5 layers and less than 200 neurons. And that with less that 60.000 samples (not even used the whole training set).. which Is remarkable results, but mostly due to.. Luck.</p>
  <p>The thing is no one knows which architectures are going to work, you have to try. If after something like 20 random attempts you still fail then the problem is likely to lie elsewhere. Maybe in the data, maybe in the encoding, maybe in the implementation.</p>
  <p>You mentioned nothing about your architecture by the way. It Is common to use Glorot init on tanh/Sigmoid layers and He initialization on ReLU layers.</p>
  <p>EDIT:</p>
  <p>I noticed now the link. Try to randomize Number of layers and Number of neurons.</p>
  <p>Then randomize 1 to 3</p>
  <ul>
  <li>If 1 all layers are same type</li>
  <li>If 2 alternate tanh and relu layers</li>
  <li>If 3 randomize type of layer</li>
  </ul>
  <p>Make at least 10 random networks
  Tell me back the results. Don't get frustrated nobody knows what works in advance. Also note that on big images, It Is usually used some convolution layer to reduce train time.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-30T19:46:22.700Z
Answers.LastActivityDate: 2024-03-30T20:06:29.797Z
Answers.LastEditDate: 2024-03-30T20:06:29.797Z
Answers.UserDisplayName: CoffeDeveloper
Answers.Body: <p>Since after a number of iterations the cost function is not reducing, this may be able to be diagnosed as a vanishing gradient problem. A solution to this is the use of a Residual neural network.</p>
  <p>Another solution is that you carefully initialise your weights as throughout your neural network your gradient may exponentially explode or exponentially vanish.</p>
  <p>Watch this video on how to initialise weights truly randomly:
  <a href="https://www.youtube.com/watch?v=s2coXdufOzE" rel="nofollow noreferrer">https://www.youtube.com/watch?v=s2coXdufOzE</a></p>
  <p><strong>Edit</strong>:</p>
  <p>Another possible cause for your issue is that your algorithm is having an high bias problem. This is due to your algorithm not performing well on the training set. In your case one of the best solutions would be to make your network deeper and so it shall be able to conduct more complex functions and so perform better on your training set.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-18T15:16:23.877Z
Answers.Text: Thanks for the tip after doing that I have initialized my weights like this **parameters['W'+str(l)]=np.random.randn(layers_dims_vector[l],layers_dims_vector[l-1])*np.sqrt(2/layers_dims_vector[l-1])** but still my cost does not seem to get reduced, Can you point me if there is something else I can do? Or do you see any error in the algorithm?
Answers.UserDisplayName: Siddarth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-19T04:59:46.300Z
Answers.Text: @Siddarth I have made an edit which should help in case that this is not a vanishing gradient issue, as seen from the random initialisation barely having an effect on the performance of your algorithm.
Answers.UserDisplayName: jr235
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-21T11:43:57.053Z
Answers.Text: Hi I tried with this configuration [12288,7000,4000,2000,10000,500,200,150,100,50,25,12,1] first is no of inputs and last is the output layer remaining are hidden layers. The network has become slow but still use, Can you check my code and let me know what I am missing?
Answers.UserDisplayName: Siddarth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-22T07:30:30.753Z
Answers.Text: In order to reduce the time your algorithm spends training you should use another optimisation algorithm (e.g Adam optimisation algorithm, Gradient Descent with momentum, Root Mean Square Propagation etc...). In order to make my diagnostic more accurate split your data set into a training (70%) and development set (30% depending on how large your data set is, the larger the smaller the dev set size), and plot the costs over time on both sets.
Answers.UserDisplayName: jr235
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-23T15:26:52.227Z
Answers.Text: Hi I have observed inspite of increasing the height and depth of network, The cost is getting plateaued. Is the cost function getting stuck in local minima? If thats the case do you see any issue with the algorithm?
Answers.UserDisplayName: Siddarth
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-18T07:45:02.980Z
Answers.LastActivityDate: 2020-07-19T05:06:28.237Z
Answers.LastEditDate: 2020-07-19T05:06:28.237Z
Answers.UserDisplayName: jr235
Body: <p>I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.</p>
  <p>The loss function which I am using is</p>
  <p><span class="math-container">$L(\hat{y},y) = -ylog\hat{y}-(1-y)log(1-\hat{y})$</span></p>
  <p>Can you please point out where I am going wrong the link to the notebook is
  <a href="https://www.kaggle.com/sidcodegladiator/catnoncat-nn" rel="nofollow noreferrer">https://www.kaggle.com/sidcodegladiator/catnoncat-nn</a>?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-11T17:45:51.903Z
Comments.Text: It might be the vanishing gradient problem.
Comments.UserDisplayName: efedoganay
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-12-16T06:49:24.400Z
Comments.Text: This isn't a cnn, it's a basic MLP and that it performs poorly isn't surprising.
Comments.UserDisplayName: FourierFlux
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-04-11T06:31:16.200Z
Comments.Text: what optimizer are you using? I'd suggest you to try Adam
Comments.UserDisplayName: SpiderRico
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-30T19:34:58.603Z
Comments.Text: Which architectures are you using? Batch size? Learning rate?
Comments.UserDisplayName: CoffeDeveloper
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-31T00:16:03.177Z
Comments.Text: What value are you using as a learning rate, what range are your weights/biases/kernels, and what is the range of your $x$ data?
Comments.UserDisplayName: David Hoelzer
Tags: <neural-networks><deep-learning><feedforward-neural-networks>
Title: Why isn't the loss of my neural network reduced after 2500 iterations?
UserDisplayName: Siddarth
ThreadText: # Why isn't the loss of my neural network reduced after 2500 iterations?
  
  (Asked by: username_0 on 2020-07-11)
  
  I have developed a basic feedforward neural network from scratch to classify whether image is of cat or not cat. It works fine, but after 2500 iterations, my cost function is not reducing properly.
  
  The loss function which I am using is
  
  $L(\hat{y},y) = -ylog\hat{y}-(1-y)log(1-\hat{y})$
  
  Can you please point out where I am going wrong the link to the notebook is
  <https://www.kaggle.com/sidcodegladiator/catnoncat-nn>?
  
  **Tags:** neural-networks, deep-learning, feedforward-neural-networks
  > **Comments:**
  > **username_1**: It might be the vanishing gradient problem.  
  > **username_2**: This isn't a cnn, it's a basic MLP and that it performs poorly isn't surprising.  
  > **username_3**: what optimizer are you using? I'd suggest you to try Adam  
  > **username_4**: Which architectures are you using? Batch size? Learning rate?  
  > **username_5**: What value are you using as a learning rate, what range are your weights/biases/kernels, and what is the range of your $x$ data?  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_6 on 2020-07-16. Score: 0
  
  You may try to adjust the learning rate first. As the learning rate has a great effect on changing the weights and the bias value.
  
  See if the results has changed after adjusting the learning rate.
  
  > **Comments:**
  > **username_0**: I tried that as well, What do you suggest my learning rate should be?  
  > **username_6**: You can try to set the learning rate to 0.01 or 0.1 to see if the results of outcome is better or not  
  > **username_0**: I tried 0.01, 0.1 and even 1 what I have noticed is the rate at which the cost function is decreasing is good but the problem is it still getting plateaued at 0.64 after 2500 epochs  
  
  ---
  
  ### Answer by username_4 on 2024-03-30. Score: 0
  
  I would try at least a dozen of random architectures, there's a interesting research paper suggesting randomization (can't find It right now, but believe me I read it) is the best bet to find optimal hyperparameters. And indeed in my opensource project it worked like a charm. The first attempt likes half of the neural networks performed at 10% accuracy on MNIST task( same as randomly guessing a digit), I removed from random selection parameters that seemed to not work (in example I removed layers with 9 and 16 neurons and Sigmoid activation function) and tried again the following time i obtained many networks above 75% accuracy. I even obtained a Small Fully Connected network that performs 87,1% accuracy with 5 layers and less than 200 neurons. And that with less that 60.000 samples (not even used the whole training set).. which Is remarkable results, but mostly due to.. Luck.
  
  The thing is no one knows which architectures are going to work, you have to try. If after something like 20 random attempts you still fail then the problem is likely to lie elsewhere. Maybe in the data, maybe in the encoding, maybe in the implementation.
  
  You mentioned nothing about your architecture by the way. It Is common to use Glorot init on tanh/Sigmoid layers and He initialization on ReLU layers.
  
  EDIT:
  
  I noticed now the link. Try to randomize Number of layers and Number of neurons.
  
  Then randomize 1 to 3
  
  * If 1 all layers are same type
  * If 2 alternate tanh and relu layers
  * If 3 randomize type of layer
  
  Make at least 10 random networks
  Tell me back the results. Don't get frustrated nobody knows what works in advance. Also note that on big images, It Is usually used some convolution layer to reduce train time.
  
  
  ---
  
  ### Answer by username_7 on 2020-07-18. Score: 0
  
  Since after a number of iterations the cost function is not reducing, this may be able to be diagnosed as a vanishing gradient problem. A solution to this is the use of a Residual neural network.
  
  Another solution is that you carefully initialise your weights as throughout your neural network your gradient may exponentially explode or exponentially vanish.
  
  Watch this video on how to initialise weights truly randomly:
  <https://www.youtube.com/watch?v=s2coXdufOzE>
  
  **Edit**:
  
  Another possible cause for your issue is that your algorithm is having an high bias problem. This is due to your algorithm not performing well on the training set. In your case one of the best solutions would be to make your network deeper and so it shall be able to conduct more complex functions and so perform better on your training set.
  
  > **Comments:**
  > **username_0**: Thanks for the tip after doing that I have initialized my weights like this **parameters['W'+str(l)]=np.random.randn(layers_dims_vector[l],layers_dims_vector[l-1])*np.sqrt(2/layers_dims_vector[l-1])** but still my cost does not seem to get reduced, Can you point me if there is something else I can do? Or do you see any error in the algorithm?  
  > **username_7**: @Siddarth I have made an edit which should help in case that this is not a vanishing gradient issue, as seen from the random initialisation barely having an effect on the performance of your algorithm.  
  > **username_0**: Hi I tried with this configuration [12288,7000,4000,2000,10000,500,200,150,100,50,25,12,1] first is no of inputs and last is the output layer remaining are hidden layers. The network has become slow but still use, Can you check my code and let me know what I am missing?  
  > **username_7**: In order to reduce the time your algorithm spends training you should use another optimisation algorithm (e.g Adam optimisation algorithm, Gradient Descent with momentum, Root Mean Square Propagation etc...). In order to make my diagnostic more accurate split your data set into a training (70%) and development set (30% depending on how large your data set is, the larger the smaller the dev set size), and plot the costs over time on both sets.  
  > **username_0**: Hi I have observed inspite of increasing the height and depth of network, The cost is getting plateaued. Is the cost function getting stuck in local minima? If thats the case do you see any issue with the algorithm?  
  
  ---
  
  