Answers.Body: <p>as far as I can tell training gpt neo with only prompt (if you use the completion as the last part of the prompt) will let you achieve the same results.</p>
  <p>You probably notice that, on HuggingFace, there is no way to make a supervised Learning with a text generation model using X as the prompt and y as the completion and the reason is that is not needed. Just add the completion to the prompt and you’ll be fine.</p>
  <p>Everything I said is well written and explained inside this HuggingFace notebook: <a href="https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb" rel="nofollow noreferrer">https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-10T01:44:28.927Z
Answers.LastActivityDate: 2023-01-10T01:45:09.877Z
Answers.LastEditDate: 2023-01-10T01:45:09.877Z
Answers.UserDisplayName: Gianmarco Ciarfaglia
Body: <p>I'm new to AI and machine learning.</p>
  <p>To <a href="https://beta.openai.com/docs/guides/fine-tuning/preparing-your-dataset" rel="nofollow noreferrer">fine-tune GPT-3</a>, I understand that we need a set of training examples that each consist of a single input (&quot;prompt&quot;) and its associated output (&quot;completion&quot;).</p>
  <p>I have prepared a dataset with &quot;prompt&quot; and &quot;completion&quot;. And I expect that a fine-tuned model would return the corresponding completion after receiving a prompt in my dataset. But <a href="https://stackoverflow.com/questions/74666268/openai-stream-interrupted-client-disconnected">due to some reason</a>, I cannot fine-tune GPT-3 at the moment.</p>
  <p>So I plan to fine-tune GPT-Neo (or GPT-J or GPT-NeoX). From <a href="https://www.youtube.com/watch?v=07ppAKvOhqk" rel="nofollow noreferrer">this video</a> and <a href="https://www.youtube.com/watch?v=uE0_XKh2d6g&amp;list=LL&amp;index=4&amp;t=401s" rel="nofollow noreferrer">this video</a>, it seems that they only accept a dataset containing only &quot;prompt&quot;.</p>
  <p>Does anyone know how I could modify my dataset with &quot;prompt&quot; and &quot;completion&quot; such that it could be used to fine-tune GPT-Neo?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-05T02:24:00.330Z
Comments.Text: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.
Comments.UserDisplayName: Community
Tags: <datasets><training-datasets><gpt><fine-tuning><gpt-3>
Title: Fine-tune GPT-Neo with prompt and completion?
UserDisplayName: SoftTimur
ThreadText: # Fine-tune GPT-Neo with prompt and completion?
  
  (Asked by: username_0 on 2022-12-04)
  
  I'm new to AI and machine learning.
  
  To [fine-tune GPT-3](https://beta.openai.com/docs/guides/fine-tuning/preparing-your-dataset), I understand that we need a set of training examples that each consist of a single input ("prompt") and its associated output ("completion").
  
  I have prepared a dataset with "prompt" and "completion". And I expect that a fine-tuned model would return the corresponding completion after receiving a prompt in my dataset. But [due to some reason](https://stackoverflow.com/questions/74666268/openai-stream-interrupted-client-disconnected), I cannot fine-tune GPT-3 at the moment.
  
  So I plan to fine-tune GPT-Neo (or GPT-J or GPT-NeoX). From [this video](https://www.youtube.com/watch?v=07ppAKvOhqk) and [this video](https://www.youtube.com/watch?v=uE0_XKh2d6g&list=LL&index=4&t=401s), it seems that they only accept a dataset containing only "prompt".
  
  Does anyone know how I could modify my dataset with "prompt" and "completion" such that it could be used to fine-tune GPT-Neo?
  
  **Tags:** datasets, training-datasets, gpt, fine-tuning, gpt-3
  > **Comments:**
  > **username_1**: Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-01-10. Score: 0
  
  as far as I can tell training gpt neo with only prompt (if you use the completion as the last part of the prompt) will let you achieve the same results.
  
  You probably notice that, on HuggingFace, there is no way to make a supervised Learning with a text generation model using X as the prompt and y as the completion and the reason is that is not needed. Just add the completion to the prompt and you’ll be fine.
  
  Everything I said is well written and explained inside this HuggingFace notebook: <https://github.com/huggingface/notebooks/blob/main/examples/language_modeling.ipynb>
  
  
  ---
  
  