<p>Yes, it is possible to finetune GPT2 to extract relevant data from a given text. For your example, you could add a prompt that looks for a <strong>manufacturer</strong>, <strong>max speed</strong>, and <strong>horsepower</strong>, and then provide an output example that includes that information.</p>
<p>It is possible to add a prefix to every training sample that contains an actual example. <em>This could help the model learn from data that has fewer labels</em>.</p>
<blockquote>
<p><code>INPUT</code>: though is mostly considered an outdated version of the <code>cls 200</code>, the new Mercedes has the capabilities of reaching up to <code>100kmh</code> in turns and <code>300kmh</code> overall?
The <code>OUTPUT</code> would be the capabilities of the new Mercedes, including its max speed and ability to handle turns.</p>
</blockquote>
<p>Yes, this is considered a common way of engineering the prompt.</p>
<blockquote>
<p>Providing the <code>same</code> example for every training sample also help the
model learn from data that has fewer labels.</p>
</blockquote>

CC BY-SA 4.0
2022-10-17T09:46:46.630Z
2022-10-17T09:46:46.630Z
Faizy
<p>I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text &quot;<em>the car was manufactured in X, can reach Y km/h, and has Z horse powers</em>&quot;, my desired output would be <em>manufacturer: <code>X</code>, max speed:<code>Y</code>, horsepowers:<code>Z</code></em>.</p>
<p>I don't have a lot of labeled data, so I thought it would be reasonable to take every training sample and add to it a prefix than contains an actual example. That is - Instead of providing the model with the text</p>
<blockquote>
<p>INPUT: a well-known brand, the new <code>X</code>, can reach <code>Y</code> km/h, and has <code>Z</code> horsepower</p>
<p>OUTPUT:</p>
</blockquote>
<p>and expect the model to understand how to fill in the details, I would provide a longer prompt that contains an actual example like</p>
<blockquote>
<p>INPUT: the new <code>BMW</code> can reach up to <code>200</code> kmh. Even though the previous model disappointed the users, the brand-new one rocks a <code>thousand</code> engine horsepower</p>
<p>OUTPUT: Manufacturer: <code>BMW</code>, Max Speed: <code>200</code>, Horse Powers: <code>thousand</code></p>
<p>INPUT: though is mostly considered an outdated version of the cls200, the new <code>Mercedes</code> has the capabilities of reaching up to 100kmh in turns and <code>300</code>kmh overall</p>
<p>OUTPUT:</p>
</blockquote>
<p>Is this considered a common way of engineering the prompt? Do notice that the first provided example is the same for every training sample.</p>

<transformer><gpt><fine-tuning><prompt>
can I add to a language model a prompt with output example?
Hadar Sharvit
# can I add to a language model a prompt with output example?

(Asked by: username_0 on 2022-10-07)

I want to finetune GPT2 to extract relevant data from a given text. So for (a trivial) example, given the text "*the car was manufactured in X, can reach Y km/h, and has Z horse powers*", my desired output would be *manufacturer: `X`, max speed:`Y`, horsepowers:`Z`*.

I don't have a lot of labeled data, so I thought it would be reasonable to take every training sample and add to it a prefix than contains an actual example. That is - Instead of providing the model with the text

> INPUT: a well-known brand, the new `X`, can reach `Y` km/h, and has `Z` horsepower
>
> OUTPUT:

and expect the model to understand how to fill in the details, I would provide a longer prompt that contains an actual example like

> INPUT: the new `BMW` can reach up to `200` kmh. Even though the previous model disappointed the users, the brand-new one rocks a `thousand` engine horsepower
>
> OUTPUT: Manufacturer: `BMW`, Max Speed: `200`, Horse Powers: `thousand`
>
> INPUT: though is mostly considered an outdated version of the cls200, the new `Mercedes` has the capabilities of reaching up to 100kmh in turns and `300`kmh overall
>
> OUTPUT:

Is this considered a common way of engineering the prompt? Do notice that the first provided example is the same for every training sample.

**Tags:** transformer, gpt, fine-tuning, prompt


---

## 1 answers

### Answer by username_1 on 2022-10-17. Score: 1

Yes, it is possible to finetune GPT2 to extract relevant data from a given text. For your example, you could add a prompt that looks for a **manufacturer**, **max speed**, and **horsepower**, and then provide an output example that includes that information.

It is possible to add a prefix to every training sample that contains an actual example. *This could help the model learn from data that has fewer labels*.

> `INPUT`: though is mostly considered an outdated version of the `cls 200`, the new Mercedes has the capabilities of reaching up to `100kmh` in turns and `300kmh` overall?
> The `OUTPUT` would be the capabilities of the new Mercedes, including its max speed and ability to handle turns.

Yes, this is considered a common way of engineering the prompt.

> Providing the `same` example for every training sample also help the
> model learn from data that has fewer labels.


---

