Answers.Body: <p>So one thing that might occur:</p>
  <blockquote>
  <p>Note that we typically only train PMs for a single epoch, so the learning curves
  themselves (Figure 7 left) indicate how performance scales with dataset size (we used a fixed learning rate).</p>
  </blockquote>
  <p>From:
  <a href="https://arxiv.org/pdf/2204.05862.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2204.05862.pdf</a></p>
  <p>Additionally:</p>
  <blockquote>
  <p>Supervised fine-tuning (SFT). We fine-tune GPT-3 on our labeler demonstrations using supervised
  learning. We trained for 16 epochs, using a cosine learning rate decay, and residual dropout of 0.2.
  We do our final SFT model selection based on the RM score on the validation set. Similarly to Wu
  et al. (2021), we find that our SFT models overfit on validation loss after 1 epoch; however, we find
  that training for more epochs helps both the RM score and human preference ratings, despite this
  overfitting.
  <a href="https://arxiv.org/pdf/2203.02155.pdf" rel="nofollow noreferrer">https://arxiv.org/pdf/2203.02155.pdf</a></p>
  </blockquote>
  <p>Here PM refers to the preference model used to evaluate generated outputs</p>
  <p>In particular, the final line</p>
  <blockquote>
  <p>we find that our SFT models overfit on validation loss after 1 epoch; however, we find
  that training for more epochs helps both the RM score and human preference ratings, despite this
  overfitting.</p>
  </blockquote>
  <p>indicates some sort of overtraining/overparameterized phenomenon</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T21:34:27.787Z
Answers.Text: Would you add some explanation on the reference contents? It is abstract, so hard to connect to Large Language Model training.
Answers.UserDisplayName: Cloud Cho
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T18:30:21.727Z
Answers.LastActivityDate: 2023-09-22T18:42:27.093Z
Answers.LastEditDate: 2023-09-22T18:42:27.093Z
Answers.UserDisplayName: information_interchange
Answers.Body: <p>Yes, when we work with LLM, first what we want to do is to calculate how much computational power we may need.</p>
  <ol>
  <li>Find energy needed that is 100M between 1,000 to 10,000, then just multiply it by 10 all numbers to get an idea for 1 billion 10 etc.</li>
  <li>Find how much memory we need that is one 16-bit float per parameter. In other word, 1 billion × 2 bytes = how much GPU memory we need.</li>
  <li>(After we did all of trivial calculations) Explore dataset and identify bias as we will have to explain it.</li>
  <li>Identify what we are looking for. If we want to make our LLM less biased we use GELU.</li>
  <li>Find approximate learning rate for LLM in a unique way: We can use central limit theorem to figure out minimum set size for the dataset in order to insure that our results have 95% confidence interval and 5% error rate.<br />
  In other words, whatever results we produce, we have 95% confidence that we reproduce identical results. For dataset we select random data, but in non trivial way. We need to divide our parameters in percentage as the model has different categories. Then we choose some material from those topics (randomly), and it is probably non trivial task.</li>
  <li>Run grid search on the dataset, get parameters and rerun on bigger set. (Shoot out to all researchers, who forgot about.)</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-22T20:37:45.960Z
Answers.LastActivityDate: 2023-09-23T13:06:54.560Z
Answers.LastEditDate: 2023-09-23T13:06:54.560Z
Answers.UserDisplayName: Viktor Veselov
Body: <p>Have never trained a (very) large language model, so I am wondering if the process is the same as training a (regular) language model, i.e. you prepare the data, set up the architecture, hyperparameters, loss function to minimize perplexity and predicting the next word, and then do gradient descent over the giant dataset. Or if there are any special gotchas or tricks you must do when training it.
  I know there's at least one involving the training dynamics:</p>
  <ol>
  <li>training dynamics: most LLMs stop seeing performance improvement even before a single epoch is finished.</li>
  </ol>
  <p>I am wondering if there are any others</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-19T16:25:35.777Z
Comments.Text: How big dataset for training do you consider?
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-21T01:22:53.613Z
Comments.Text: Let's say as big as the dataset used to train ChatGPT :)
Comments.UserDisplayName: information_interchange
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-21T16:09:29.890Z
Comments.Text: I heard that OpenAI used Wikipedia. I think probably used the ZIM (around 100GB with image compressed Wikipedia) (https://www.howtogeek.com/260023/how-to-download-wikipedia-for-offline-at-your-fingertips-reading/). You may search how OpenAI did in their training.
Comments.UserDisplayName: Cloud Cho
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-21T16:16:04.980Z
Comments.Text: Here are one webpage by OpenAI: https://openai.com/research/techniques-for-training-large-neural-networks
Comments.UserDisplayName: Cloud Cho
Tags: <neural-networks><training><chatgpt><architecture><large-language-models>
Title: Tips and tricks when training a very large language model?
UserDisplayName: information_interchange
ThreadText: # Tips and tricks when training a very large language model?
  
  (Asked by: username_0 on 2023-09-19)
  
  Have never trained a (very) large language model, so I am wondering if the process is the same as training a (regular) language model, i.e. you prepare the data, set up the architecture, hyperparameters, loss function to minimize perplexity and predicting the next word, and then do gradient descent over the giant dataset. Or if there are any special gotchas or tricks you must do when training it.
  I know there's at least one involving the training dynamics:
  
  1. training dynamics: most LLMs stop seeing performance improvement even before a single epoch is finished.
  
  I am wondering if there are any others
  
  **Tags:** neural-networks, training, chatgpt, architecture, large-language-models
  > **Comments:**
  > **username_1**: How big dataset for training do you consider?  
  > **username_0**: Let's say as big as the dataset used to train ChatGPT :)  
  > **username_1**: I heard that OpenAI used Wikipedia. I think probably used the ZIM (around 100GB with image compressed Wikipedia) (https://www.howtogeek.com/260023/how-to-download-wikipedia-for-offline-at-your-fingertips-reading/). You may search how OpenAI did in their training.  
  > **username_1**: Here are one webpage by OpenAI: https://openai.com/research/techniques-for-training-large-neural-networks  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2023-09-22. Score: 1
  
  So one thing that might occur:
  
  > Note that we typically only train PMs for a single epoch, so the learning curves
  > themselves (Figure 7 left) indicate how performance scales with dataset size (we used a fixed learning rate).
  
  From:
  <https://arxiv.org/pdf/2204.05862.pdf>
  
  Additionally:
  
  > Supervised fine-tuning (SFT). We fine-tune GPT-3 on our labeler demonstrations using supervised
  > learning. We trained for 16 epochs, using a cosine learning rate decay, and residual dropout of 0.2.
  > We do our final SFT model selection based on the RM score on the validation set. Similarly to Wu
  > et al. (2021), we find that our SFT models overfit on validation loss after 1 epoch; however, we find
  > that training for more epochs helps both the RM score and human preference ratings, despite this
  > overfitting.
  > <https://arxiv.org/pdf/2203.02155.pdf>
  
  Here PM refers to the preference model used to evaluate generated outputs
  
  In particular, the final line
  
  > we find that our SFT models overfit on validation loss after 1 epoch; however, we find
  > that training for more epochs helps both the RM score and human preference ratings, despite this
  > overfitting.
  
  indicates some sort of overtraining/overparameterized phenomenon
  
  > **Comments:**
  > **username_1**: Would you add some explanation on the reference contents? It is abstract, so hard to connect to Large Language Model training.  
  
  ---
  
  ### Answer by username_2 on 2023-09-22. Score: 0
  
  Yes, when we work with LLM, first what we want to do is to calculate how much computational power we may need.
  
  1. Find energy needed that is 100M between 1,000 to 10,000, then just multiply it by 10 all numbers to get an idea for 1 billion 10 etc.
  2. Find how much memory we need that is one 16-bit float per parameter. In other word, 1 billion × 2 bytes = how much GPU memory we need.
  3. (After we did all of trivial calculations) Explore dataset and identify bias as we will have to explain it.
  4. Identify what we are looking for. If we want to make our LLM less biased we use GELU.
  5. Find approximate learning rate for LLM in a unique way: We can use central limit theorem to figure out minimum set size for the dataset in order to insure that our results have 95% confidence interval and 5% error rate.  
     In other words, whatever results we produce, we have 95% confidence that we reproduce identical results. For dataset we select random data, but in non trivial way. We need to divide our parameters in percentage as the model has different categories. Then we choose some material from those topics (randomly), and it is probably non trivial task.
  6. Run grid search on the dataset, get parameters and rerun on bigger set. (Shoot out to all researchers, who forgot about.)
  
  
  ---
  
  