Answers.Body: <p>I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.</p>
  
  <p>From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.</p>
  
  <p>So using its pre-trained understanding, it analyzes the <strong>structure of sample texts</strong> and replicates this structure.<br>
  For scientific articles this structure could be:</p>
  
  <ul>
  <li>Abstract</li>
  <li>Context of research topic</li>
  <li>Introduction of researchers</li>
  <li>Explanation of methods/experiments/discoveries</li>
  <li>Results and interpretation</li>
  <li>Future research and application</li>
  </ul>
  
  <p>For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.</p>
  
  <p><strong>TL;DR:</strong> The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).</p>
  
  <p><strong>P.S.:</strong> Take this with a grain of salt. It's 90% conjecture from incomplete information.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-03T13:03:36.823Z
Answers.Text: your not far off, but thats only a piece of it, it also extends to task specificity in an overarching vague sense. In other words, it depends what you want to get out of it.
Answers.UserDisplayName: mshlis
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2019-09-02T15:30:53.780Z
Answers.LastActivityDate: 2019-09-02T15:30:53.780Z
Answers.UserDisplayName: NoAnswer
Body: <p>I'm following <a href="https://medium.com/@ngwaifoong92/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f" rel="nofollow noreferrer">this</a> tutorial, and I wonder why is there a train-step - why is it necessary? I thought the whole idea of GPT-2 is that you do not need to train it on specific text domain, as it's already pre-trained on a large amount of data.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-10-04T05:38:40.610Z
Comments.Text: The web link required registration. Do you have public link? Also please put more explanation of "retrain". Do you mean transfer learning or fine tuning (ref: https://www.tensorflow.org/tutorials/images/transfer_learning)?
Comments.UserDisplayName: Cloud Cho
Tags: <natural-language-processing>
Title: Why do you need to retrain GPT-2?
UserDisplayName: Maverick Meerkat
ThreadText: # Why do you need to retrain GPT-2?
  
  (Asked by: username_0 on 2019-09-02)
  
  I'm following [this](https://medium.com/@ngwaifoong92/beginners-guide-to-retrain-gpt-2-117m-to-generate-custom-text-content-8bb5363d8b7f) tutorial, and I wonder why is there a train-step - why is it necessary? I thought the whole idea of GPT-2 is that you do not need to train it on specific text domain, as it's already pre-trained on a large amount of data.
  
  **Tags:** natural-language-processing
  
  > **Comments:**
  > **username_1**: The web link required registration. Do you have public link? Also please put more explanation of "retrain". Do you mean transfer learning or fine tuning (ref: https://www.tensorflow.org/tutorials/images/transfer_learning)?  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2019-09-02. Score: 1
  
  I've come to learn about GPT-2 through Robert Miles AI safety Youtube channel and intend to look into it in more detail.
  
  From my current understanding, GPT-2 is pre-trained to "understand" "natural" language (for any definition of the words in quotes). However you would want it to not only understand general text but generate text similar to some specific "genre", e.g. scientific articles, youtube comments, twitter messages, you name it.
  
  So using its pre-trained understanding, it analyzes the **structure of sample texts** and replicates this structure.  
  For scientific articles this structure could be:
  
  * Abstract
  * Context of research topic
  * Introduction of researchers
  * Explanation of methods/experiments/discoveries
  * Results and interpretation
  * Future research and application
  
  For Youtube comments the structure is probably more chaotic but could include a vague reference to former comments, insults, nonsensical bar-grade philosophy, internet slang and smileys.
  
  **TL;DR:** The domain specific text is only used to tell GPT-2 what you're looking for. You basically hand it context to work with, instead of prompting "Say something clever" (my least favorite line at parties, when I've been introduced as clever).
  
  **P.S.:** Take this with a grain of salt. It's 90% conjecture from incomplete information.
  
  > **Comments:**
  > **username_3**: your not far off, but thats only a piece of it, it also extends to task specificity in an overarching vague sense. In other words, it depends what you want to get out of it.  
  
  ---
  
  