Answers.Body: <p>As you pointed out, duplicates can be a possible reason for such behavior.
  There are a few more possibilities:</p>
  <ol>
  <li><p>Class Imbalance - data is skewed towards a particular class(if you are solving a classification problem).</p>
  </li>
  <li><p>Model isn't learning well for 100% of data for given parameters. Try changing the learning rate, or adding some constraints such as:</p>
  </li>
  </ol>
  <pre><code>if t == int(args.num_iter * 0.5):
       lr = some_small_value*lr
  if t == int(args.num_iter * 0.75):
       lr = smaller_value*lr
  </code></pre>
  <ol start="3">
  <li>It may so happen that the model you have selected is unable to handle weights when input data is large. Try changing the model and see if some other model works better.</li>
  </ol>
  <p>I hope this helps.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-24T18:29:40.767Z
Answers.Text: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).
Answers.UserDisplayName: Community
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-24T20:27:04.507Z
Answers.Text: Hi, thank you for your answer. I don't have a classification problem, but it's useful to point that out. Your second point simply looks like a learning rate scheduler, which I could indeed implement. I don't know what you mean with 3., though. It seems like rather general advice, so can you specify what you mean by "unable to handle weights when input data is large"?
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-26T18:06:55.203Z
Answers.Text: It's similar to the reason we need LSTM/GRU over RNN. Also, LSTM has some variations like peephole LSTM to further increase the capacity of the model to process data and store weights. It may so happen that your model is as good as an RNN, i.e. it can handle data when its size is less, but as you increase the data, you need a more advanced model such as a LSTM.
Answers.UserDisplayName: Nityanand Mathur
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-28T14:02:36.037Z
Answers.Text: I am using GRUs, but I will look into peephole LSTM to understand the difference you are talking about.
Answers.UserDisplayName: postnubilaphoebus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-24T17:48:50.807Z
Answers.LastActivityDate: 2023-03-24T17:48:50.807Z
Answers.UserDisplayName: Nityanand Mathur
Body: <p>I trained a neural network on an NLP problem and compared the loss and BLEU score on the validation data with the same training parameters in two scenarios: a) when I trained on 25% of the data, b) when I trained on 100% of the data. I observed a peculiar pattern: the validation loss and BLEU score is much lower in b) as compared to a). What could this mean? My guess is that there are duplicates in the training data, which lead to overfitting, but I still have to investigate that.</p>
  
Tags: <datasets><overfitting><validation-loss>
Title: What are possible reasons for the validation loss increasing with more data?
UserDisplayName: postnubilaphoebus
ThreadText: # What are possible reasons for the validation loss increasing with more data?
  
  (Asked by: username_0 on 2023-03-23)
  
  I trained a neural network on an NLP problem and compared the loss and BLEU score on the validation data with the same training parameters in two scenarios: a) when I trained on 25% of the data, b) when I trained on 100% of the data. I observed a peculiar pattern: the validation loss and BLEU score is much lower in b) as compared to a). What could this mean? My guess is that there are duplicates in the training data, which lead to overfitting, but I still have to investigate that.
  
  **Tags:** datasets, overfitting, validation-loss
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-03-24. Score: 1 | Accepted answer
  
  As you pointed out, duplicates can be a possible reason for such behavior.
  There are a few more possibilities:
  
  1. Class Imbalance - data is skewed towards a particular class(if you are solving a classification problem).
  2. Model isn't learning well for 100% of data for given parameters. Try changing the learning rate, or adding some constraints such as:
  
  ```
  if t == int(args.num_iter * 0.5):
       lr = some_small_value*lr
  if t == int(args.num_iter * 0.75):
       lr = smaller_value*lr
  
  ```
  
  3. It may so happen that the model you have selected is unable to handle weights when input data is large. Try changing the model and see if some other model works better.
  
  I hope this helps.
  
  > **Comments:**
  > **username_2**: Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer).  
  > **username_0**: Hi, thank you for your answer. I don't have a classification problem, but it's useful to point that out. Your second point simply looks like a learning rate scheduler, which I could indeed implement. I don't know what you mean with 3., though. It seems like rather general advice, so can you specify what you mean by "unable to handle weights when input data is large"?  
  > **username_1**: It's similar to the reason we need LSTM/GRU over RNN. Also, LSTM has some variations like peephole LSTM to further increase the capacity of the model to process data and store weights. It may so happen that your model is as good as an RNN, i.e. it can handle data when its size is less, but as you increase the data, you need a more advanced model such as a LSTM.  
  > **username_0**: I am using GRUs, but I will look into peephole LSTM to understand the difference you are talking about.  
  
  ---
  
  