Answers.Body: <p>In general, there's nothing wrong with training loss to increase from time-to-time during training.</p>
  <p>This is because GD with minibatch is a stochastic process and doesn't guarantee that the loss will decrease at each step.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T01:15:37.403Z
Answers.Text: But I am not training with minibatches.
Answers.UserDisplayName: ZZ1
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T12:44:03.683Z
Answers.Text: @ZainAli So, how exactly are you training? Maybe edit your post to clarify this.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T00:24:18.207Z
Answers.LastActivityDate: 2020-07-07T00:24:18.207Z
Answers.UserDisplayName: SpiderRico
Answers.Body: <p>After reading your comments, I assume you are doing full-batch training instead of mini-batch training. Therefore, stochasticity is not a factor in this part. Despite the training batch size, I couldn’t find any information about your model’s activation function, which could be relevant.</p>
  <p>Let me explain a bit: Let’s say you are using a ReLU activation function. In that case, some of the information can be directly multiplied by zero if the product of the weight and input is negative. So, after the weight update, this situation might change.</p>
  <p>Moreover, this is not the only possible explanation. For example, even though we know gradient descent will give us a vector that is ‘guaranteed’ to decrease the loss, the learning rate can affect this situation.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-07-17T09:22:56.310Z
Answers.LastActivityDate: 2024-07-17T09:22:56.310Z
Answers.UserDisplayName: Barışcan Kurtkaya
Answers.Body: <p>Since neural networks rely on stochasticity (i.e. randomness) to initialize their parameters and gradient descent selects random batches of training data at each iteration, is perfectly normal if the value of loss function fluctuates instead of decreasing monotonically.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-25T07:24:30.037Z
Answers.Text: I understand that non-monotonicity is related to minibatches, but not to parameter initialization.
Answers.UserDisplayName: Jaume Oliver Lafont
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-07-07T13:22:21.393Z
Answers.LastActivityDate: 2020-07-07T13:22:21.393Z
Answers.UserDisplayName: Anirban Saha
Body: <p>I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts to increase for some time, after that, the cost starts to decrease again, as can be seen in the figure.</p>
  <p><a href="https://i.sstatic.net/ak7oj.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ak7oj.jpg" alt="enter image description here" /></a></p>
  <p>I am using the gradient descend method for optimization. Decreasing the step size/learning rate does not change the behavior. I have checked the code and math, again and again, I don't think there is an error (I could be wrong).</p>
  <p>Why is the cost function not decreasing monotonically? Could there be a reason other than an error in my code/math? If there is an error, do you think that it is just a coincidence that each time the system finally converges to a very small value of error?</p>
  <p>I am a beginner in the field of machine learning, hence, many questions I have asked may seem foolish to you. I am saving the values after every 100 iterations so the figure is actually for 15000 iterations.</p>
  <p><strong>About training:</strong>
  I am using one-hot encoding. As the training data has only two samples (&quot;gods&quot; and &quot;dogs&quot;), where each alphabet is represented as d=[1,0,0,0],o=[0,1,0,0],g=[0,0,1,0],s=[0,0,0,1]. The recurrent neural network (RNN) goes back to a maximum of 3 time units, (e.g for dogs, the first input is 'd', then 'o', followed by 'g' and s). So, for the second input, the RNN goes back to 1 input, for the third input the RNN observes both previous inputs and so on. After calculating the gradients for the word &quot;dogs&quot;, the values of the gradients are saved and the process is repeated for the word &quot;gods&quot;. The gradients calculated for the second input &quot;gods&quot; are summed with the gradients calculated for &quot;dogs&quot; at the end of each epoch/iteration, and then the sum is used to updated all the weights. In each epoch, the inputs remain the same i.e &quot;gods&quot; and &quot;dogs&quot;. In mini-batch training, in some epoch the RNN may encounter new inputs, hence, the loss may increase. However, I do not think that what I am doing qualifies as mini-batch training as there are only two inputs, both inputs are used in each epoch, and the sum of calculated gradients is used to update the weights.</p>
  
Tags: <neural-networks><machine-learning><recurrent-neural-networks><gradient-descent>
Title: Why the cost/loss starts to increase for some iterations during the training phase?
UserDisplayName: ZZ1
ThreadText: # Why the cost/loss starts to increase for some iterations during the training phase?
  
  (Asked by: username_0 on 2020-07-06)
  
  I am trying to build a recurrent neural network from scratch. It's a very simple model. I am trying to train it to predict two words (dogs and gods). While training, the value of cost function starts to increase for some time, after that, the cost starts to decrease again, as can be seen in the figure.
  
  [![enter image description here](https://i.sstatic.net/ak7oj.jpg)](https://i.sstatic.net/ak7oj.jpg)
  
  I am using the gradient descend method for optimization. Decreasing the step size/learning rate does not change the behavior. I have checked the code and math, again and again, I don't think there is an error (I could be wrong).
  
  Why is the cost function not decreasing monotonically? Could there be a reason other than an error in my code/math? If there is an error, do you think that it is just a coincidence that each time the system finally converges to a very small value of error?
  
  I am a beginner in the field of machine learning, hence, many questions I have asked may seem foolish to you. I am saving the values after every 100 iterations so the figure is actually for 15000 iterations.
  
  **About training:**
  I am using one-hot encoding. As the training data has only two samples ("gods" and "dogs"), where each alphabet is represented as d=[1,0,0,0],o=[0,1,0,0],g=[0,0,1,0],s=[0,0,0,1]. The recurrent neural network (RNN) goes back to a maximum of 3 time units, (e.g for dogs, the first input is 'd', then 'o', followed by 'g' and s). So, for the second input, the RNN goes back to 1 input, for the third input the RNN observes both previous inputs and so on. After calculating the gradients for the word "dogs", the values of the gradients are saved and the process is repeated for the word "gods". The gradients calculated for the second input "gods" are summed with the gradients calculated for "dogs" at the end of each epoch/iteration, and then the sum is used to updated all the weights. In each epoch, the inputs remain the same i.e "gods" and "dogs". In mini-batch training, in some epoch the RNN may encounter new inputs, hence, the loss may increase. However, I do not think that what I am doing qualifies as mini-batch training as there are only two inputs, both inputs are used in each epoch, and the sum of calculated gradients is used to update the weights.
  
  **Tags:** neural-networks, machine-learning, recurrent-neural-networks, gradient-descent
  
  
  ---
  
  ## 3 answers
  
  ### Answer by username_1 on 2020-07-07. Score: 0
  
  In general, there's nothing wrong with training loss to increase from time-to-time during training.
  
  This is because GD with minibatch is a stochastic process and doesn't guarantee that the loss will decrease at each step.
  
  > **Comments:**
  > **username_0**: But I am not training with minibatches.  
  > **username_2**: @ZainAli So, how exactly are you training? Maybe edit your post to clarify this.  
  
  ---
  
  ### Answer by username_3 on 2024-07-17. Score: 0
  
  After reading your comments, I assume you are doing full-batch training instead of mini-batch training. Therefore, stochasticity is not a factor in this part. Despite the training batch size, I couldn’t find any information about your model’s activation function, which could be relevant.
  
  Let me explain a bit: Let’s say you are using a ReLU activation function. In that case, some of the information can be directly multiplied by zero if the product of the weight and input is negative. So, after the weight update, this situation might change.
  
  Moreover, this is not the only possible explanation. For example, even though we know gradient descent will give us a vector that is ‘guaranteed’ to decrease the loss, the learning rate can affect this situation.
  
  
  ---
  
  ### Answer by username_4 on 2020-07-07. Score: 0
  
  Since neural networks rely on stochasticity (i.e. randomness) to initialize their parameters and gradient descent selects random batches of training data at each iteration, is perfectly normal if the value of loss function fluctuates instead of decreasing monotonically.
  
  > **Comments:**
  > **username_5**: I understand that non-monotonicity is related to minibatches, but not to parameter initialization.  
  
  ---
  
  