Answers.Body: <h3>Data</h3>
  <p>As it seems like ChatGPT already works well, you could use ChatGPT to generate training data.</p>
  <h3>Models</h3>
  <p>I would look at finetuning an existing text paraphrasing model or abstractive summarization model as your task seems extremely similar.</p>
  <p>You can find these models on HuggingFace (e.g., <a href="https://huggingface.co/humarin/chatgpt_paraphraser_on_T5_base" rel="nofollow noreferrer">this paraphrasing model</a>).</p>
  <h3>Training</h3>
  <p>In regard to training, you can take a look at how people train models on other sequence to sequence tasks. e.g., these HuggingFace tutorials on <a href="https://huggingface.co/docs/transformers/tasks/translation" rel="nofollow noreferrer">translation</a> and <a href="https://huggingface.co/docs/transformers/tasks/summarization" rel="nofollow noreferrer">summarization</a>.</p>
  <h3>Evaluation</h3>
  <p>For metrics, you should look at how text-summarization papers/paraphrasing papers evaluate their models. For example, <a href="https://arxiv.org/pdf/1911.09661.pdf" rel="nofollow noreferrer">this paper</a> looks at embedding similarity, ROUGE, and BLEU.</p>
  <p>For your setting specifically, you should also be designing specialized metrics. Look into tooks like Spacy and NLTK to extract, e.g., named entities, parts of speech, and design specialized heuristics to test for your requirements (<a href="https://stackoverflow.com/a/74594808">this SO post for active/passive voice detection might be useful inspiration</a>).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-01T07:53:27.190Z
Answers.Text: Thank you so much for this, I'm gonna try these paraphrasing models. I think the challenge is  with the rules that  should be followed when paraphrasing. Otherwise  I will try to train the model on a lot of documents following the rules. if this works, I will accept this answer. Thank you again.
Answers.UserDisplayName: Wissem Boujlida
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-30T21:25:31.317Z
Answers.LastActivityDate: 2023-09-30T21:25:31.317Z
Answers.UserDisplayName: Alexander Wan
Body: <p>I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules :</p>
  <p>1- always follow the pattern &quot;Entity Verb Entity&quot;</p>
  <p>2- only use simple sentences : never combine sentences</p>
  <p>3- Don't replace existing words</p>
  <p>4- Don't lose the overall meaning of the text or any information in it.</p>
  <p>For example:</p>
  <p>text = &quot;Project Risk Management includes the processes of conducting risk management planning, identification, analysis, response planning, response implementation, and monitoring risk on a project&quot;</p>
  <p>Standardized Text = &quot;Project Risk Management conducts risk management planning. Project Risk Management conducts risk identification. Project Risk Management conducts risk analysis. Project Risk Management plans responses. Project Risk Management implements responses. Project Risk Management monitors risk on a project.&quot;</p>
  <p>Using ChatGPT the results were very good, but I want to know if I can fine tune a model (BERT, T5, any LM) locally, what should be the data format for training such a model, evaluation metrics ?</p>
  
Tags: <natural-language-processing><large-language-models><seq2seq>
Title: How to train a seq2seq model to rephrase input text following given rules
UserDisplayName: Wissem Boujlida
ThreadText: # How to train a seq2seq model to rephrase input text following given rules
  
  (Asked by: username_0 on 2023-09-30)
  
  I want to train (fine-tune) a seq2seq model to perform the task of rephrasing input following these rules :
  
  1- always follow the pattern "Entity Verb Entity"
  
  2- only use simple sentences : never combine sentences
  
  3- Don't replace existing words
  
  4- Don't lose the overall meaning of the text or any information in it.
  
  For example:
  
  text = "Project Risk Management includes the processes of conducting risk management planning, identification, analysis, response planning, response implementation, and monitoring risk on a project"
  
  Standardized Text = "Project Risk Management conducts risk management planning. Project Risk Management conducts risk identification. Project Risk Management conducts risk analysis. Project Risk Management plans responses. Project Risk Management implements responses. Project Risk Management monitors risk on a project."
  
  Using ChatGPT the results were very good, but I want to know if I can fine tune a model (BERT, T5, any LM) locally, what should be the data format for training such a model, evaluation metrics ?
  
  **Tags:** natural-language-processing, large-language-models, seq2seq
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-30. Score: 1
  
  ### Data
  
  As it seems like ChatGPT already works well, you could use ChatGPT to generate training data.
  
  ### Models
  
  I would look at finetuning an existing text paraphrasing model or abstractive summarization model as your task seems extremely similar.
  
  You can find these models on HuggingFace (e.g., [this paraphrasing model](https://huggingface.co/humarin/chatgpt_paraphraser_on_T5_base)).
  
  ### Training
  
  In regard to training, you can take a look at how people train models on other sequence to sequence tasks. e.g., these HuggingFace tutorials on [translation](https://huggingface.co/docs/transformers/tasks/translation) and [summarization](https://huggingface.co/docs/transformers/tasks/summarization).
  
  ### Evaluation
  
  For metrics, you should look at how text-summarization papers/paraphrasing papers evaluate their models. For example, [this paper](https://arxiv.org/pdf/1911.09661.pdf) looks at embedding similarity, ROUGE, and BLEU.
  
  For your setting specifically, you should also be designing specialized metrics. Look into tooks like Spacy and NLTK to extract, e.g., named entities, parts of speech, and design specialized heuristics to test for your requirements ([this SO post for active/passive voice detection might be useful inspiration](https://stackoverflow.com/a/74594808)).
  
  > **Comments:**
  > **username_0**: Thank you so much for this, I'm gonna try these paraphrasing models. I think the challenge is  with the rules that  should be followed when paraphrasing. Otherwise  I will try to train the model on a lot of documents following the rules. if this works, I will accept this answer. Thank you again.  
  
  ---
  
  