Answers.Body: <p>In general, the methods still work even with an infinite amount of data, as long as there are common/reoccuring patterns that a neural network can learn to identify. For example: If you would have infinitely many images of dogs and cats, there are features that discriminate the two animals that are mostly consistent like the shape of the nose. Having infinitely many samples in these cases is generally desirable, because it can benefit the ability of the model to generalize.</p>
  <p>In contrast, there are cases where this is not true that depend on the data: If your data contains (concept-) drift, meaning that your data distribution changes over time, the model you train might not be able to learn a consistently performing function and therefore chases a moving objective. For images, this can be the case if the labels would depend on lighting conditions that constantly change (concept drift), or if the objects in the images that you want to classify continuously change shape (drift).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T13:34:00.620Z
Answers.Text: No they don't do that, they are purely local functions, i.e. they use the current batch and don't keep any sort of 'record' of the whole dataset. With the exception of the momentum term, but that one doesn't require your dataset to repeat at some point.
Answers.UserDisplayName: Chillston
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T13:06:10.587Z
Answers.Text: I assume my data has no 'drift', the first training dataset will serve the purpose as well as any later one. I am just wonder if ADAM and other methods somehow assume the same dataset for consecutive batches or something to be able to tune their internal parameters.
Answers.UserDisplayName: dronus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-20T14:27:28.980Z
Answers.LastActivityDate: 2023-01-20T14:27:28.980Z
Answers.UserDisplayName: Chillston
Answers.Body: <p>Subdividing datasets into batches is only done for computational reasons (RAM, computation time etc). Optimizers do not care about repeated data. They get a batch of data to optimize on and will calculate the gradient for that batch and update the model accordingly. Nothing in this process requires the batches to reappear at all.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-14T13:53:58.447Z
Answers.Text: I agree in general but dividing data into batches also introduces noise (since the batch does represent the probability distribution of the whole dataset only up to a certain extent, the difference can be interpreted as noise) and might therefore help to overcome local minima
Answers.UserDisplayName: pythonic833
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-14T14:53:16.887Z
Answers.Text: I never pretended that having smaller or bigger batches does not matter in an optimization process. They definitely do. However, whether batches reappear exactly as is should not matter for optimization, which was what the author asked.
Answers.UserDisplayName: Robin van Hoorn
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-14T15:58:38.130Z
Answers.Text: I just wanted to add this since the formulation in your post says "is only done for computational reasons" which IMHO neglects the theoretical aspect of introducing noise.
Answers.UserDisplayName: pythonic833
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-20T14:22:26.913Z
Answers.LastActivityDate: 2023-01-20T14:22:26.913Z
Answers.UserDisplayName: Robin van Hoorn
Body: <p>When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?</p>
  <p>I have an DNN training from an infinite stream of samples, that most likely won't repeat. So there is no real notion of &quot;epoch&quot;.</p>
  <p>Now I wonder if the math behind ADAM or other popular optimizers expect the repetition of data over the epochs?</p>
  <p>If so, should I collect a limited amount of those samples and use them for training data and validation data, or would it be better to use all data available (even if the training data never repeats then)?</p>
  
Tags: <machine-learning><training><adam><optimizers>
Title: When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
UserDisplayName: dronus
ThreadText: # When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
  
  (Asked by: username_0 on 2023-01-20)
  
  When training a DNN on infinite samples, do ADAM or other popular optimization algorithms still work as intended?
  
  I have an DNN training from an infinite stream of samples, that most likely won't repeat. So there is no real notion of "epoch".
  
  Now I wonder if the math behind ADAM or other popular optimizers expect the repetition of data over the epochs?
  
  If so, should I collect a limited amount of those samples and use them for training data and validation data, or would it be better to use all data available (even if the training data never repeats then)?
  
  **Tags:** machine-learning, training, adam, optimizers
  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_1 on 2023-01-20. Score: 1
  
  In general, the methods still work even with an infinite amount of data, as long as there are common/reoccuring patterns that a neural network can learn to identify. For example: If you would have infinitely many images of dogs and cats, there are features that discriminate the two animals that are mostly consistent like the shape of the nose. Having infinitely many samples in these cases is generally desirable, because it can benefit the ability of the model to generalize.
  
  In contrast, there are cases where this is not true that depend on the data: If your data contains (concept-) drift, meaning that your data distribution changes over time, the model you train might not be able to learn a consistently performing function and therefore chases a moving objective. For images, this can be the case if the labels would depend on lighting conditions that constantly change (concept drift), or if the objects in the images that you want to classify continuously change shape (drift).
  
  > **Comments:**
  > **username_0**: I assume my data has no 'drift', the first training dataset will serve the purpose as well as any later one. I am just wonder if ADAM and other methods somehow assume the same dataset for consecutive batches or something to be able to tune their internal parameters.  
  > **username_1**: No they don't do that, they are purely local functions, i.e. they use the current batch and don't keep any sort of 'record' of the whole dataset. With the exception of the momentum term, but that one doesn't require your dataset to repeat at some point.  
  
  ---
  
  ### Answer by username_2 on 2023-01-20. Score: 0
  
  Subdividing datasets into batches is only done for computational reasons (RAM, computation time etc). Optimizers do not care about repeated data. They get a batch of data to optimize on and will calculate the gradient for that batch and update the model accordingly. Nothing in this process requires the batches to reappear at all.
  
  > **Comments:**
  > **username_3**: I agree in general but dividing data into batches also introduces noise (since the batch does represent the probability distribution of the whole dataset only up to a certain extent, the difference can be interpreted as noise) and might therefore help to overcome local minima  
  > **username_2**: I never pretended that having smaller or bigger batches does not matter in an optimization process. They definitely do. However, whether batches reappear exactly as is should not matter for optimization, which was what the author asked.  
  > **username_3**: I just wanted to add this since the formulation in your post says "is only done for computational reasons" which IMHO neglects the theoretical aspect of introducing noise.  
  
  ---
  
  