Answers.Body: <p>The trivial answer is that yes, it is possible. Consider standard Gaussian data and a generator sampling points <span class="math-container">$1/n$</span> with <span class="math-container">$n \rightarrow \infty$</span>. Since you never see points beyond <span class="math-container">$(0,1]$</span>, you are unlikely to learn the parameters of the distribution.</p>
  <p>More generally, and for a supervised problem with data <span class="math-container">$X,Y$</span>, if your data generator covers a subset of the domain with probability <span class="math-container">$1-\delta$</span> (with respect to the distribution of <span class="math-container">$X,Y$</span>) for a very small <span class="math-container">$\delta$</span> and your model <span class="math-container">$\hat{f}$</span> fits with error <span class="math-container">$\epsilon$</span> here, then your generalization error is <span class="math-container">$\mathbb{E}_{X Y}[\text{loss}(\hat{f}(X),Y)] \leq \epsilon + \delta \sup_{X Y} |\text{loss}(\hat{f}(X),Y)|$</span>. So if you knew that the loss were bounded and you could make <span class="math-container">$\delta$</span> arbitrarily small, you wouldn't overfit, because there wouldn't be any region of the sample space significant enough where you wouldn't be able to generalize to.</p>
  <p>In other words, it all boils down to your generator being able to cover a set with high enough probability and your loss being &quot;well behaved&quot;.</p>
  <p>Note that because you will only have a finite coverage of the domain, e.g. an <span class="math-container">$\epsilon$</span>-cover, you need some assumption on the growth of the loss for points which are <span class="math-container">$\epsilon$</span> apart, e.g. a Lipschitz condition (and also for the learned <span class="math-container">$\hat{f}$</span>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-06-03T09:19:46.927Z
Answers.LastActivityDate: 2021-06-03T09:19:46.927Z
Answers.UserDisplayName: Miguel
Body: <p>This is a theoretical question. <em>Is it possible to overfit a model on infinite amounts of data?</em></p>
  <p>Let me clarify <em>there are no duplicates.</em></p>
  <p>Say, we have a generator function that produces data, with the correct classification/regression value, and we can generate infinite amounts of valid data. How long does it take for the model to overfit?</p>
  <p>This question arose because I'm training an RNN model for fake news classification, and MSE loss is almost always 0.000, only 25% of the training data.</p>
  <p>Will it be possible to overfit with one epoch of training on the infinite data generator?</p>
  <p>(I'm thinking what will happen is the model will either get perfect, or sync into the generator's non-perfect randomness, and learn nothing)</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-02T16:32:00.590Z
Comments.Text: If your data covers the entire sample space (easy to conceptualize with discrete-valued features), you don't need to abstract any model at all - you can just spit out the mapping from input to output, which much exist in the training data. I'm not sure the concept of overfitting is meaningful if it's impossible to have "unseen data". There's also no concept of "noise", as every unique combination of feature inputs maps to one and only one output - if "noise" can be perfectly predicted from the data, it's signal.
Comments.UserDisplayName: Nuclear Hoagie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-02T17:31:04.660Z
Comments.Text: This is to generalize the concept of which the dataset is large enough that the features learned at the beginning of the epoch practically don't exist by the end. This is currently the case with the model I am training. If we have perfect randomness included in the model, then the noise becomes impossible to predict. There is some 'unseen' data, although it is very arbitrary in state-space
Comments.UserDisplayName: ToAskOrNotToAsk
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-06-02T18:00:15.467Z
Comments.Text: I feel that there's a disconnect here between "very large data" and "infinite data". With an infinite sample size, you'll cover the entire sample space almost surely. When selecting a new test sample from the population, the probability that it's not already in an infinite training sample of that population is zero (no unseen data). I don't follow your point about noise - since there are no duplicates in the data, every input has a fixed, deterministic mapping to an output (there is no randomness, since the output is perfectly predictable from the input features).
Comments.UserDisplayName: Nuclear Hoagie
Tags: <machine-learning><training><recurrent-neural-networks><overfitting><computational-learning-theory>
Title: Is it possible to overfit a model on infinite amounts of data?
UserDisplayName: ToAskOrNotToAsk
ThreadText: # Is it possible to overfit a model on infinite amounts of data?
  
  (Asked by: username_0 on 2021-06-02)
  
  This is a theoretical question. *Is it possible to overfit a model on infinite amounts of data?*
  
  Let me clarify *there are no duplicates.*
  
  Say, we have a generator function that produces data, with the correct classification/regression value, and we can generate infinite amounts of valid data. How long does it take for the model to overfit?
  
  This question arose because I'm training an RNN model for fake news classification, and MSE loss is almost always 0.000, only 25% of the training data.
  
  Will it be possible to overfit with one epoch of training on the infinite data generator?
  
  (I'm thinking what will happen is the model will either get perfect, or sync into the generator's non-perfect randomness, and learn nothing)
  
  **Tags:** machine-learning, training, recurrent-neural-networks, overfitting, computational-learning-theory
  
  > **Comments:**
  > **username_1**: If your data covers the entire sample space (easy to conceptualize with discrete-valued features), you don't need to abstract any model at all - you can just spit out the mapping from input to output, which much exist in the training data. I'm not sure the concept of overfitting is meaningful if it's impossible to have "unseen data". There's also no concept of "noise", as every unique combination of feature inputs maps to one and only one output - if "noise" can be perfectly predicted from the data, it's signal.  
  > **username_0**: This is to generalize the concept of which the dataset is large enough that the features learned at the beginning of the epoch practically don't exist by the end. This is currently the case with the model I am training. If we have perfect randomness included in the model, then the noise becomes impossible to predict. There is some 'unseen' data, although it is very arbitrary in state-space  
  > **username_1**: I feel that there's a disconnect here between "very large data" and "infinite data". With an infinite sample size, you'll cover the entire sample space almost surely. When selecting a new test sample from the population, the probability that it's not already in an infinite training sample of that population is zero (no unseen data). I don't follow your point about noise - since there are no duplicates in the data, every input has a fixed, deterministic mapping to an output (there is no randomness, since the output is perfectly predictable from the input features).  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-06-03. Score: 0 | Accepted answer
  
  The trivial answer is that yes, it is possible. Consider standard Gaussian data and a generator sampling points $1/n$ with $n \rightarrow \infty$. Since you never see points beyond $(0,1]$, you are unlikely to learn the parameters of the distribution.
  
  More generally, and for a supervised problem with data $X,Y$, if your data generator covers a subset of the domain with probability $1-\delta$ (with respect to the distribution of $X,Y$) for a very small $\delta$ and your model $\hat{f}$ fits with error $\epsilon$ here, then your generalization error is $\mathbb{E}\_{X Y}[\text{loss}(\hat{f}(X),Y)] \leq \epsilon + \delta \sup\_{X Y} |\text{loss}(\hat{f}(X),Y)|$. So if you knew that the loss were bounded and you could make $\delta$ arbitrarily small, you wouldn't overfit, because there wouldn't be any region of the sample space significant enough where you wouldn't be able to generalize to.
  
  In other words, it all boils down to your generator being able to cover a set with high enough probability and your loss being "well behaved".
  
  Note that because you will only have a finite coverage of the domain, e.g. an $\epsilon$-cover, you need some assumption on the growth of the loss for points which are $\epsilon$ apart, e.g. a Lipschitz condition (and also for the learned $\hat{f}$).
  
  
  ---
  
  