Answers.Body: <p>Yes, it is possible to finetune GPT2 to extract relevant data from a given text. For your example, you could add a prompt that looks for a <strong>manufacturer</strong>, <strong>max speed</strong>, and <strong>horsepower</strong>, and then provide an output example that includes that information.</p>
  <p>It is possible to add a prefix to every training sample that contains an actual example. <em>This could help the model learn from data that has fewer labels</em>.</p>
  <blockquote>
  <p><code>INPUT</code>: though is mostly considered an outdated version of the <code>cls 200</code>, the new Mercedes has the capabilities of reaching up to <code>100kmh</code> in turns and <code>300kmh</code> overall?
  The <code>OUTPUT</code> would be the capabilities of the new Mercedes, including its max speed and ability to handle turns.</p>
  </blockquote>
  <p>Yes, this is considered a common way of engineering the prompt.</p>
  <blockquote>
  <p>Providing the <code>same</code> example for every training sample also help the
  model learn from data that has fewer labels.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-10-17T09:46:46.630Z
Answers.LastActivityDate: 2022-10-17T09:46:46.630Z
Answers.UserDisplayName: Faizy
Body: <p>I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text &quot;<em>the car was manufactured in X, can reach Y km/h, and has Z horse powers</em>&quot;, my desired output would be <em>manufacturer: <code>X</code>, max speed:<code>Y</code>, horsepowers:<code>Z</code></em>.</p>
  <p>I don't have a lot of labeled data, so I thought it would be reasonable to take every training sample and add to it a prefix than contains an actual example. That is - Instead of providing the model with the text</p>
  <blockquote>
  <p>INPUT: a well-known brand, the new <code>X</code>, can reach <code>Y</code> km/h, and has <code>Z</code> horsepower</p>
  <p>OUTPUT:</p>
  </blockquote>
  <p>and expect the model to understand how to fill in the details, I would provide a longer prompt that contains an actual example like</p>
  <blockquote>
  <p>INPUT: the new <code>BMW</code> can reach up to <code>200</code> kmh. Even though the previous model disappointed the users, the brand-new one rocks a <code>thousand</code> engine horsepower</p>
  <p>OUTPUT: Manufacturer: <code>BMW</code>, Max Speed: <code>200</code>, Horse Powers: <code>thousand</code></p>
  <p>INPUT: though is mostly considered an outdated version of the cls200, the new <code>Mercedes</code> has the capabilities of reaching up to 100kmh in turns and <code>300</code>kmh overall</p>
  <p>OUTPUT:</p>
  </blockquote>
  <p>Is this considered a common way of engineering the prompt? Do notice that the first provided example is the same for every training sample.</p>
  
Tags: <transformer><gpt><fine-tuning><prompt>
Title: can I add to a language model a prompt with output example?
UserDisplayName: Hadar Sharvit
ThreadText: # can I add to a language model a prompt with output example?
  
  (Asked by: username_0 on 2022-10-07)
  
  I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text "*the car was manufactured in X, can reach Y km/h, and has Z horse powers*", my desired output would be *manufacturer: `X`, max speed:`Y`, horsepowers:`Z`*.
  
  I don't have a lot of labeled data, so I thought it would be reasonable to take every training sample and add to it a prefix than contains an actual example. That is - Instead of providing the model with the text
  
  > INPUT: a well-known brand, the new `X`, can reach `Y` km/h, and has `Z` horsepower
  >
  > OUTPUT:
  
  and expect the model to understand how to fill in the details, I would provide a longer prompt that contains an actual example like
  
  > INPUT: the new `BMW` can reach up to `200` kmh. Even though the previous model disappointed the users, the brand-new one rocks a `thousand` engine horsepower
  >
  > OUTPUT: Manufacturer: `BMW`, Max Speed: `200`, Horse Powers: `thousand`
  >
  > INPUT: though is mostly considered an outdated version of the cls200, the new `Mercedes` has the capabilities of reaching up to 100kmh in turns and `300`kmh overall
  >
  > OUTPUT:
  
  Is this considered a common way of engineering the prompt? Do notice that the first provided example is the same for every training sample.
  
  **Tags:** transformer, gpt, fine-tuning, prompt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2022-10-17. Score: 1
  
  Yes, it is possible to finetune GPT2 to extract relevant data from a given text. For your example, you could add a prompt that looks for a **manufacturer**, **max speed**, and **horsepower**, and then provide an output example that includes that information.
  
  It is possible to add a prefix to every training sample that contains an actual example. *This could help the model learn from data that has fewer labels*.
  
  > `INPUT`: though is mostly considered an outdated version of the `cls 200`, the new Mercedes has the capabilities of reaching up to `100kmh` in turns and `300kmh` overall?
  > The `OUTPUT` would be the capabilities of the new Mercedes, including its max speed and ability to handle turns.
  
  Yes, this is considered a common way of engineering the prompt.
  
  > Providing the `same` example for every training sample also help the
  > model learn from data that has fewer labels.
  
  
  ---
  
  