Answers.Body: <p>Based on the colab notebook that you used, the periodic oscillation of the training loss does not seem to originate from the learning rate scheduler which is set to &quot;constant&quot;.
  However, I found that setting <code>group_by_length=True</code> <a href="https://huggingface.co/transformers/v4.9.0/main_classes/trainer.html" rel="nofollow noreferrer">(link)</a> in the trainer can result in periodic oscillation of the training loss.</p>
  <p>Below are 2 training loss examples for the same model/same dataset finetuned with:
  <img src="https://i.sstatic.net/BiD2j.png" alt="group_by_length=True" /><a href="https://i.sstatic.net/BiD2j.png" rel="nofollow noreferrer">1</a>
  <code>group_by_length=True</code></p>
  <p><img src="https://i.sstatic.net/80aXr.png" alt="group_by_length=False" /><a href="https://i.sstatic.net/80aXr.png" rel="nofollow noreferrer">2</a>
  <code>group_by_length=False</code></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-15T03:03:36.670Z
Answers.LastActivityDate: 2023-11-15T10:52:17.817Z
Answers.LastEditDate: 2023-11-15T10:52:17.817Z
Answers.UserDisplayName: JMarc
Body: <p><a href="https://i.sstatic.net/RckTe.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RckTe.png" alt="loss curve" /></a></p>
  <p>Why is my loss curve consistently oscillating? Every 50 steps it jumps back up. I'm assumming there's a bug in my data, since I'm using <a href="https://colab.research.google.com/drive/1BiQiw31DT7-cDp1-0ySXvvhzqomTdI-o?usp=sharing" rel="nofollow noreferrer">this colab notebook</a> that shows a proper train/loss at the bottom. The only thing I changed was the dataset used. I'm using the <a href="https://huggingface.co/ybelkada/falcon-7b-sharded-bf16" rel="nofollow noreferrer">falcon-7b-sharded-bf16</a> huggingface model.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-08-17T01:11:55.610Z
Comments.Text: probably stepsize scheduling?
Comments.UserDisplayName: Alberto
Tags: <deep-learning><training><data-preprocessing><large-language-models><training-datasets>
Title: What causes my loss curve to consistently oscillate when training an LLM?
UserDisplayName: bmatzelle
ThreadText: # What causes my loss curve to consistently oscillate when training an LLM?
  
  (Asked by: username_0 on 2023-08-16)
  
  [![loss curve](https://i.sstatic.net/RckTe.png)](https://i.sstatic.net/RckTe.png)
  
  Why is my loss curve consistently oscillating? Every 50 steps it jumps back up. I'm assumming there's a bug in my data, since I'm using [this colab notebook](https://colab.research.google.com/drive/1BiQiw31DT7-cDp1-0ySXvvhzqomTdI-o?usp=sharing) that shows a proper train/loss at the bottom. The only thing I changed was the dataset used. I'm using the [falcon-7b-sharded-bf16](https://huggingface.co/ybelkada/falcon-7b-sharded-bf16) huggingface model.
  
  **Tags:** deep-learning, training, data-preprocessing, large-language-models, training-datasets
  > **Comments:**
  > **username_1**: probably stepsize scheduling?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-11-15. Score: 1
  
  Based on the colab notebook that you used, the periodic oscillation of the training loss does not seem to originate from the learning rate scheduler which is set to "constant".
  However, I found that setting `group_by_length=True` [(link)](https://huggingface.co/transformers/v4.9.0/main_classes/trainer.html) in the trainer can result in periodic oscillation of the training loss.
  
  Below are 2 training loss examples for the same model/same dataset finetuned with:
  ![group_by_length=True](https://i.sstatic.net/BiD2j.png)[1](https://i.sstatic.net/BiD2j.png)
  `group_by_length=True`
  
  ![group_by_length=False](https://i.sstatic.net/80aXr.png)[2](https://i.sstatic.net/80aXr.png)
  `group_by_length=False`
  
  
  ---
  
  