Answers.Body: <p>After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my application the sweet spot is around 30-35k tokens.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-12-01T11:54:07.470Z
Answers.LastActivityDate: 2020-12-01T11:54:07.470Z
Answers.UserDisplayName: Andreas Toresäter
Body: <p>I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material.</p>
  <p>How large should the corpus be to optimally retrain the GPT-2 model? And what is the bare minimum size? Should it simply be as large as possible or can it flip over and make the model worse in some way?</p>
  <p>I am also not certain how many steps you should let the retraining run. I have been using 6000 steps when testing, and it seems not much happens after that, loss only moved from 0.2 to 0.18 last 1000 steps.</p>
  
Tags: <deep-learning><natural-language-processing><tensorflow><training><gpt>
Title: How large should the corpus be to optimally retrain the GPT-2 model?
UserDisplayName: Andreas Toresäter
ThreadText: # How large should the corpus be to optimally retrain the GPT-2 model?
  
  (Asked by: username_0 on 2020-07-29)
  
  I just started working with the GPT-2 models and want to retrain one on a pretty narrow topic, so I have problems finding training material.
  
  How large should the corpus be to optimally retrain the GPT-2 model? And what is the bare minimum size? Should it simply be as large as possible or can it flip over and make the model worse in some way?
  
  I am also not certain how many steps you should let the retraining run. I have been using 6000 steps when testing, and it seems not much happens after that, loss only moved from 0.2 to 0.18 last 1000 steps.
  
  **Tags:** deep-learning, natural-language-processing, tensorflow, training, gpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_0 on 2020-12-01. Score: 1
  
  After doing many runs with GPT-2 355M I come to the conclusion that anything below 20k tokens yield worse and worse results the less tokens you have regardless of how many steps you train for. For my application the sweet spot is around 30-35k tokens.
  
  
  ---
  
  