Answers.Body: <h3>In short: It depends.</h3>
  <h5>Where will you run it?</h5>
  <ul>
  <li><strong>On Premises</strong>: You may want want to run in your own environment.</li>
  <li><strong>IaaS</strong>: GPT models are often too big, so people might prefer to setup a different server for that, serving your API.</li>
  <li><strong>PaaS</strong>: If it's more experimental, I recommend running it on Google Colab.</li>
  <li><strong>SaaS</strong>: Or even use some external API, so you don't need to worry about this setup and just use it as a service. (easiest)</li>
  </ul>
  <p>Each approach demands a different architecture and a different code.</p>
  <p>Once you've setup the environment / API, you'll run it by providing an <em>initial prompt</em> and some <em>parameters</em>:</p>
  <h3>GPT</h3>
  <p>GPT-2 (any GPT model) is a general, open-domain text-generating model, which tries to predict the next word for any given context.</p>
  <p>So, setting up a <em>&quot;summarize mode</em>&quot; is <strong>not</strong> just flagging a parameter. It's a non-deterministic process and requires tries and errors.</p>
  <p>The GPT setup is experimental:</p>
  <ul>
  <li>You use a sandbox.</li>
  <li>Create an initial prompt.</li>
  <li>Set some parameters (Temperature, Top-P, Top-K,...)</li>
  <li>Evaluate your results</li>
  <li>Adjust the prompt</li>
  <li>Adjust the parameters</li>
  <li>Until you consistently achieve the desirable results.</li>
  </ul>
  <h3>The prompt</h3>
  <h5>1. Simple way (no shot learning)</h5>
  <p>A very simple way to do it is something like:</p>
  <pre><code>prompt = text+&quot;\nTL;DR:&quot;
  </code></pre>
  <p>It should work specially well if your text is simple and small.</p>
  <h5>2. Explicit Introduction (no shot learning)</h5>
  <p>You <em>may</em> have a better result if initially prompt some context, for example:</p>
  <pre><code>&quot;Here is a text and it's respective summary.\n&quot;+
  &quot;#Full text:&quot; + text + &quot;\n&quot;+
  &quot;#Summary:&quot;
  </code></pre>
  <h5>3. Add examples (few shot learning)</h5>
  <p>Another approach is filling your prompt with a few (high-quality) examples before your original prompt:</p>
  <pre><code>prompt = sample_1+&quot;\nTL;DR:&quot;+summary_1
  prompt += &quot;\n###\n&quot;
  prompt += sample_2+&quot;\nTL;DR:&quot;+summary_2
  prompt += &quot;\n###\n&quot;
  prompt += your_input+&quot;\nTL;DR:&quot;
  </code></pre>
  <p>And you realize the next logical thing for the generator to make is a summary of your input.</p>
  <p>The results will greatly depend on your text size and style. So you should find what prompt template best suits your needs.</p>
  <h3>Execution</h3>
  <p>Keep in mind that <strong>GPT will not learn from previous executions</strong>.
  It has no memory or learning in between executions, which means that each input will require the whole prompt again.</p>
  <p>So your main program should <strong>run a loop and prompt GPT once for every text file</strong>. Something like:</p>
  <pre><code>for file in glob(./files.txt)
    text = open(file, &quot;r&quot;)
    prompt = text+&quot;\nTLDR:\n&quot;
    result = GPT(prompt, parameters)
  </code></pre>
  <h5>Other tips to keep in mind:</h5>
  <p>If you have no examples:</p>
  <ol>
  <li>Results may be different from desired (Too big, too short, too informal, omit something you consider important)</li>
  <li>Be less stable. (Nail it sometimes and ruin other times).</li>
  </ol>
  <p>If you have some examples, it should replicate your example style. But:</p>
  <ol>
  <li>It may wrongly make references to the example (instead of the desired text).</li>
  <li>Gets confused if your text does not match the example(s) lenght / style.</li>
  <li>Prompt may get too big.</li>
  </ol>
  <p>If the original text is too big:</p>
  <ol>
  <li>It will require more computational power.</li>
  <li>It might extrapolate the limit of tokens it can digest. (So you might need to split the text into chapters).</li>
  <li>The model might get confused, like: Summarizing only the last paragraph (so you need to make it clear when the text starts and ends).</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-18T10:20:26.590Z
Answers.Text: You wrote TD;DR each time you meant TL;DR
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-17T03:27:57.863Z
Answers.LastActivityDate: 2023-06-29T03:10:31.857Z
Answers.LastEditDate: 2023-06-29T03:10:31.857Z
Answers.UserDisplayName: Andre Goulart
Body: <p>In section 3.6 of the <a href="https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf" rel="noreferrer">OpenAI GPT-2 paper</a> it mentions summarising text based relates to this, but the method is described in very high-level terms:</p>
  
  <blockquote>
    <p>To induce summarization behavior we add the text <code>TL;DR:</code> after the article and generate 100 tokens with Top-k random sampling (Fan et al., 2018) with k=2 which reduces repetition and encourages more abstractive summaries than greedy decoding. We use the first 3 generated sentences in these 100 tokens as the summary.</p>
  </blockquote>
  
  <p>Given a corpus of text, in concrete code terms (python preferred), how would I go about generating a summary of it?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2019-03-03T13:36:11.540Z
Comments.Text: I think the referenced article explains clearly how to do it: http://www.aclweb.org/anthology/P18-1082
Comments.UserDisplayName: OmG
Tags: <open-ai><text-summarization><text-generation><gpt-2>
Title: How do I use GPT-2 to summarise text?
UserDisplayName: Tom Hale
ThreadText: # How do I use GPT-2 to summarise text?
  
  (Asked by: username_0 on 2019-03-03)
  
  In section 3.6 of the [OpenAI GPT-2 paper](https://d4mucfpksywv.cloudfront.net/better-language-models/language-models.pdf) it mentions summarising text based relates to this, but the method is described in very high-level terms:
  
  > To induce summarization behavior we add the text `TL;DR:` after the article and generate 100 tokens with Top-k random sampling (Fan et al., 2018) with k=2 which reduces repetition and encourages more abstractive summaries than greedy decoding. We use the first 3 generated sentences in these 100 tokens as the summary.
  
  Given a corpus of text, in concrete code terms (python preferred), how would I go about generating a summary of it?
  
  **Tags:** open-ai, text-summarization, text-generation, gpt-2
  
  > **Comments:**
  > **username_1**: I think the referenced article explains clearly how to do it: http://www.aclweb.org/anthology/P18-1082  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2021-08-17. Score: 3
  
  ### In short: It depends.
  
  ##### Where will you run it?
  
  * **On Premises**: You may want want to run in your own environment.
  * **IaaS**: GPT models are often too big, so people might prefer to setup a different server for that, serving your API.
  * **PaaS**: If it's more experimental, I recommend running it on Google Colab.
  * **SaaS**: Or even use some external API, so you don't need to worry about this setup and just use it as a service. (easiest)
  
  Each approach demands a different architecture and a different code.
  
  Once you've setup the environment / API, you'll run it by providing an *initial prompt* and some *parameters*:
  
  ### GPT
  
  GPT-2 (any GPT model) is a general, open-domain text-generating model, which tries to predict the next word for any given context.
  
  So, setting up a *"summarize mode*" is **not** just flagging a parameter. It's a non-deterministic process and requires tries and errors.
  
  The GPT setup is experimental:
  
  * You use a sandbox.
  * Create an initial prompt.
  * Set some parameters (Temperature, Top-P, Top-K,...)
  * Evaluate your results
  * Adjust the prompt
  * Adjust the parameters
  * Until you consistently achieve the desirable results.
  
  ### The prompt
  
  ##### 1. Simple way (no shot learning)
  
  A very simple way to do it is something like:
  
  ```
  prompt = text+"\nTL;DR:"
  
  ```
  
  It should work specially well if your text is simple and small.
  
  ##### 2. Explicit Introduction (no shot learning)
  
  You *may* have a better result if initially prompt some context, for example:
  
  ```
  "Here is a text and it's respective summary.\n"+
  "#Full text:" + text + "\n"+
  "#Summary:"
  
  ```
  
  ##### 3. Add examples (few shot learning)
  
  Another approach is filling your prompt with a few (high-quality) examples before your original prompt:
  
  ```
  prompt = sample_1+"\nTL;DR:"+summary_1
  prompt += "\n###\n"
  prompt += sample_2+"\nTL;DR:"+summary_2
  prompt += "\n###\n"
  prompt += your_input+"\nTL;DR:"
  
  ```
  
  And you realize the next logical thing for the generator to make is a summary of your input.
  
  The results will greatly depend on your text size and style. So you should find what prompt template best suits your needs.
  
  ### Execution
  
  Keep in mind that **GPT will not learn from previous executions**.
  It has no memory or learning in between executions, which means that each input will require the whole prompt again.
  
  So your main program should **run a loop and prompt GPT once for every text file**. Something like:
  
  ```
  for file in glob(./files.txt)
    text = open(file, "r")
    prompt = text+"\nTLDR:\n"
    result = GPT(prompt, parameters)
  
  ```
  
  ##### Other tips to keep in mind:
  
  If you have no examples:
  
  1. Results may be different from desired (Too big, too short, too informal, omit something you consider important)
  2. Be less stable. (Nail it sometimes and ruin other times).
  
  If you have some examples, it should replicate your example style. But:
  
  1. It may wrongly make references to the example (instead of the desired text).
  2. Gets confused if your text does not match the example(s) lenght / style.
  3. Prompt may get too big.
  
  If the original text is too big:
  
  1. It will require more computational power.
  2. It might extrapolate the limit of tokens it can digest. (So you might need to split the text into chapters).
  3. The model might get confused, like: Summarizing only the last paragraph (so you need to make it clear when the text starts and ends).
  
  > **Comments:**
  > **username_3**: You wrote TD;DR each time you meant TL;DR  
  
  ---
  
  