Answers.Body: <p>This really depends on how you define &quot;iteration&quot;. Here it's not so simple, given that the number of training steps per epoch varies based on the number of correct predictions, which would obviously change as you continue to train the model.</p>
  <p>Generally I have found iterations refers to the number of times you run a batch through a network. This is the more reasonable definition, as when you are doing real-world machine learning, using batches of training examples allows you to utilize as much memory as possible, thus making the training much faster.</p>
  <p>Using this definition with your example you would always have the same number of iterations as the batch size would be consistent, it's just inside the massive matrix that is your batch, you would have update values of 0 where the network correctly predicted.</p>
  <p>It ultimately comes down to <em>what's the best way to convey information to a reader</em>? I think describing the number of iterations as variable based on network outputs is confusing and non-descriptive. In this case, if you don't want to use any batches, it is best to say you have 120 iterations per epoch. Saying you have <em>up to</em> 120 iterations is confusing. Instead, just specify that in some iterations the network may not be updated.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T04:55:43.597Z
Answers.LastActivityDate: 2021-08-17T04:55:43.597Z
Answers.UserDisplayName: Recessive
Answers.Body: <p>A quick google the keyword &quot;definition of iteration in machine learning&quot; gives us a lot of results. I would like to stick with <a href="https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks">this StackOverFlow question</a>.</p>
  <p>As your example, if we have 100 samples, let me assume batch size is 20, so the number of iteration is 5. If there is one iteration that both 20 samples are predicted correctly, this epoch should be counted which means one epoch you still have 5 iterations since the number of iterations is important in some situations such as control the learning rate while training (decay or cycle).</p>
  <p>If you feel uncomfortable with the non-gradient / non-updating on that iteration, you can understand as your model's weight is updated with the gradient is 0.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-12T03:01:08.943Z
Answers.LastActivityDate: 2021-08-12T03:01:08.943Z
Answers.UserDisplayName: CuCaRot
Body: <p>Per <a href="https://developers.google.com/machine-learning/glossary#iteration" rel="nofollow noreferrer">google's glossary</a>, an iteration refers to</p>
  <blockquote>
  <p>A single update of a model's weights during training ...</p>
  </blockquote>
  <p>The following code comes from <a href="https://github.com/albert10jp/deep-learning/blob/main/LR_on_not_linearly_separable_data.ipynb" rel="nofollow noreferrer">a github repo</a></p>
  <pre><code>def fit(self, x, y, verbose=False, seed=None):
      indices = np.arange(len(x))
      for i in range(self.n_epoch):
          n_iter = 0
          np.random.seed(seed)
          np.random.shuffle(indices)
          for idx in indices:
              if(self.predict(x[idx])!=y[idx]):
                  self.update_weights(x[idx], y[idx], verbose)
              else:
                  n_iter += 1
          if(n_iter==len(x)):
              print('model gets 100% train accuracy after {} epoch(s)'.format(i))
              break
  </code></pre>
  <p>Note that this model doesn't update weights for each single example, because when the model make a correct prediction for some example, it skips the example without updating weights.</p>
  <p>In this kind of scenario where model makes a correct prediction for <span class="math-container">$i$</span>th input <span class="math-container">$x_i$</span> and jump into next example <span class="math-container">$x_{i+1}$</span> without updating weights for <span class="math-container">$x_i$</span>, does it count as an iteration?</p>
  <p>Assume there are 120 training examples, in one epoch, the model makes 20 correct prediction and updates weight for the other 100. Should I count this epoch 100 iterations or 120 iterations?</p>
  <p><strong>Note</strong>: This question is NOT about coding. The code cited above works well. This question is about terminology. The code is just to illustrate the scenario in question.</p>
  
Tags: <terminology>
Title: Assume 120 examples, a model makes 20 correct predictions and updates weight for the other 100. Should I count this epoch 100 iterations or 120?
UserDisplayName: JJJohn
ThreadText: # Assume 120 examples, a model makes 20 correct predictions and updates weight for the other 100. Should I count this epoch 100 iterations or 120?
  
  (Asked by: username_0 on 2021-08-07)
  
  Per google's glossary, an iteration refers to
  
  A single update of a model's weights during training ...
  
  The following code comes from a github repo
  def fit(self, x, y, verbose=False, seed=None):
      indices = np.arange(len(x))
      for i in range(self.n_epoch):
          n_iter = 0
          np.random.seed(seed)
          np.random.shuffle(indices)
          for idx in indices:
              if(self.predict(x[idx])!=y[idx]):
                  self.update_weights(x[idx], y[idx], verbose)
              else:
                  n_iter += 1
          if(n_iter==len(x)):
              print('model gets 100% train accuracy after {} epoch(s)'.format(i))
              break
  
  Note that this model doesn't update weights for each single example, because when the model make a correct prediction for some example, it skips the example without updating weights.
  In this kind of scenario where model makes a correct prediction for $i$th input $x_i$ and jump into next example $x_{i+1}$ without updating weights for $x_i$, does it count as an iteration?
  Assume there are 120 training examples, in one epoch, the model makes 20 correct prediction and updates weight for the other 100. Should I count this epoch 100 iterations or 120 iterations?
  Note: This question is NOT about coding. The code cited above works well. This question is about terminology. The code is just to illustrate the scenario in question.
  
  **Tags:** terminology
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2021-08-17. Score: 1 | Accepted answer
  
  This really depends on how you define "iteration". Here it's not so simple, given that the number of training steps per epoch varies based on the number of correct predictions, which would obviously change as you continue to train the model.
  
  Generally I have found iterations refers to the number of times you run a batch through a network. This is the more reasonable definition, as when you are doing real-world machine learning, using batches of training examples allows you to utilize as much memory as possible, thus making the training much faster.
  
  Using this definition with your example you would always have the same number of iterations as the batch size would be consistent, it's just inside the massive matrix that is your batch, you would have update values of 0 where the network correctly predicted.
  
  It ultimately comes down to *what's the best way to convey information to a reader*? I think describing the number of iterations as variable based on network outputs is confusing and non-descriptive. In this case, if you don't want to use any batches, it is best to say you have 120 iterations per epoch. Saying you have *up to* 120 iterations is confusing. Instead, just specify that in some iterations the network may not be updated.
  
  
  ---
  
  ### Answer by username_2 on 2021-08-12. Score: 0
  
  A quick google the keyword "definition of iteration in machine learning" gives us a lot of results. I would like to stick with [this StackOverFlow question](https://stackoverflow.com/questions/4752626/epoch-vs-iteration-when-training-neural-networks).
  
  As your example, if we have 100 samples, let me assume batch size is 20, so the number of iteration is 5. If there is one iteration that both 20 samples are predicted correctly, this epoch should be counted which means one epoch you still have 5 iterations since the number of iterations is important in some situations such as control the learning rate while training (decay or cycle).
  
  If you feel uncomfortable with the non-gradient / non-updating on that iteration, you can understand as your model's weight is updated with the gradient is 0.
  
  
  ---
  
  