Answers.Body: <p>From my understanding of your task, you're looking to get the overall emotion classification score for a long piece of dialogue.</p>
  <p>BERT can handle contexts up to 512 tokens in length, so the task becomes a matter of handling differences in training vs testing distributions. If your training set (for recognizing emotions) contains only single sentences, then you're correct in assuming that the model will struggle on longer pieces of text.</p>
  <p><strong>The best solution would be to find an in-domain dataset to train on.</strong> If you finetune your model on emotion classification for long-contexts, then the model will be able to handle long-contexts natively. In particular, you're looking for <em>conversational</em> datasets.</p>
  <h2>MELD</h2>
  <p>One popular dataset is <a href="https://affective-meld.github.io/" rel="nofollow noreferrer">MELD</a>. It consists of dialogue instances from the TV show <em>Friends</em>. Samples have fine-grained labels per utterance, which tend to be pretty short, <em>but</em> they are labeled in-context of the rest of the text. So, you can either train a model to predict the average label of the entire dialogue, or to predict the label of the <span class="math-container">$i$</span>th utterance in the context of the rest of the dialogue, then averaging the scores after the fact.</p>
  <p>This is a pretty popular dataset, so there's a number of existing papers that train models for this task, <a href="https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-meld" rel="nofollow noreferrer">many of which have code and/or checkpoints available.</a></p>
  <h2>EmoryNLP</h2>
  <p>A similar dataset is the <a href="https://github.com/emorynlp/emotion-detection" rel="nofollow noreferrer">EmoryNLP emotion recognition dataset</a>, part of their character mining project. You can find models trained for this task <a href="https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-4" rel="nofollow noreferrer">here</a>.</p>
  <h2>DailyDialog</h2>
  <p>Another relevant dataset is the <a href="http://yanran.li/dailydialog" rel="nofollow noreferrer">DailyDialog dataset</a>, which features more &quot;every-day&quot; conversational data. You can find models trained for this task <a href="https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-3" rel="nofollow noreferrer">here</a>.</p>
  <h2>Other Datasets</h2>
  <p>I found many of these (and other) emotion recognition datasets on this <a href="https://paperswithcode.com/task/emotion-recognition-in-conversation" rel="nofollow noreferrer">paperswithcode listing</a>.</p>
  <h2>Generating Synthetic Data</h2>
  <p>Some large language models are trained on enough tasks/data that they may be able to generalize to arbitrary tasks without additional finetuning or with only a few labeled examples. The most famous one would be ChatGPT/GPT-4. You could also try models like <a href="https://huggingface.co/docs/transformers/model_doc/flan-t5" rel="nofollow noreferrer">FLAN</a> or <a href="https://huggingface.co/allenai/tk-instruct-11b-def" rel="nofollow noreferrer">Tk-Instruct</a>.</p>
  <p>Be sure to check the performance of these models manually first, but you can then use these models to label existing dialog data, which you can use to train a smaller model (like BERT). This gives you the flexibility of using <em>any</em> dialog data that's similar to what will be seen during test-time. You can even try prompting GPT-4 to generate similar data to any existing samples that you may already have.</p>
  <h2>Training your model</h2>
  <p>I would recommend you take a look at some of these papers to find an architecture/codebase that's effective &amp; easy to replicate, and train on a <strong>combination of these datasets</strong>. Training on just one may be an issue as these datasets tend to be pretty domain-specific.</p>
  <p>I am unsure of how exactly this was trained, but <a href="https://huggingface.co/j-hartmann/emotion-english-distilroberta-base" rel="nofollow noreferrer">this emotion detection model on huggingface</a> seems to do exactly this. If it was trained on the full dialogs (rather than single sentences/short utterances), then this would work for your use-case -- you may want to test the accuracy of it yourself first.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-03T22:57:02.360Z
Answers.LastActivityDate: 2023-09-03T23:08:19.180Z
Answers.LastEditDate: 2023-09-03T23:08:19.180Z
Answers.UserDisplayName: Alexander Wan
Body: <p>From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences.
  But if the text you want to evaluate is a large script with many sentences, do I need to split the script into sentences and get a classification for each one? If I did it that way I could find the average score for each emotion for the the sentences in the script.
  The problem with doing it that way is the the context between sentences is lost. A context like sarcasm often means that sentences have to be connected in order to understand that it is sarcastic.
  I would be interested to get some advice on this.</p>
  
Tags: <natural-language-processing><bert><data-labelling><emotion-recognition>
Title: Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?
UserDisplayName: arame3333
ThreadText: # Fine Tuning a Bert Transformer. How to label for emotions and train large scripts?
  
  (Asked by: username_0 on 2023-08-31)
  
  From what I have seen you can fine tune a Bert model to detect emotions by labelling single sentences.
  But if the text you want to evaluate is a large script with many sentences, do I need to split the script into sentences and get a classification for each one? If I did it that way I could find the average score for each emotion for the the sentences in the script.
  The problem with doing it that way is the the context between sentences is lost. A context like sarcasm often means that sentences have to be connected in order to understand that it is sarcastic.
  I would be interested to get some advice on this.
  
  **Tags:** natural-language-processing, bert, data-labelling, emotion-recognition
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-09-03. Score: 2 | Accepted answer
  
  From my understanding of your task, you're looking to get the overall emotion classification score for a long piece of dialogue.
  
  BERT can handle contexts up to 512 tokens in length, so the task becomes a matter of handling differences in training vs testing distributions. If your training set (for recognizing emotions) contains only single sentences, then you're correct in assuming that the model will struggle on longer pieces of text.
  
  **The best solution would be to find an in-domain dataset to train on.** If you finetune your model on emotion classification for long-contexts, then the model will be able to handle long-contexts natively. In particular, you're looking for *conversational* datasets.
  
  ## MELD
  
  One popular dataset is [MELD](https://affective-meld.github.io/). It consists of dialogue instances from the TV show *Friends*. Samples have fine-grained labels per utterance, which tend to be pretty short, *but* they are labeled in-context of the rest of the text. So, you can either train a model to predict the average label of the entire dialogue, or to predict the label of the $i$th utterance in the context of the rest of the dialogue, then averaging the scores after the fact.
  
  This is a pretty popular dataset, so there's a number of existing papers that train models for this task, [many of which have code and/or checkpoints available.](https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-meld)
  
  ## EmoryNLP
  
  A similar dataset is the [EmoryNLP emotion recognition dataset](https://github.com/emorynlp/emotion-detection), part of their character mining project. You can find models trained for this task [here](https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-4).
  
  ## DailyDialog
  
  Another relevant dataset is the [DailyDialog dataset](http://yanran.li/dailydialog), which features more "every-day" conversational data. You can find models trained for this task [here](https://paperswithcode.com/sota/emotion-recognition-in-conversation-on-3).
  
  ## Other Datasets
  
  I found many of these (and other) emotion recognition datasets on this [paperswithcode listing](https://paperswithcode.com/task/emotion-recognition-in-conversation).
  
  ## Generating Synthetic Data
  
  Some large language models are trained on enough tasks/data that they may be able to generalize to arbitrary tasks without additional finetuning or with only a few labeled examples. The most famous one would be ChatGPT/GPT-4. You could also try models like [FLAN](https://huggingface.co/docs/transformers/model_doc/flan-t5) or [Tk-Instruct](https://huggingface.co/allenai/tk-instruct-11b-def).
  
  Be sure to check the performance of these models manually first, but you can then use these models to label existing dialog data, which you can use to train a smaller model (like BERT). This gives you the flexibility of using *any* dialog data that's similar to what will be seen during test-time. You can even try prompting GPT-4 to generate similar data to any existing samples that you may already have.
  
  ## Training your model
  
  I would recommend you take a look at some of these papers to find an architecture/codebase that's effective & easy to replicate, and train on a **combination of these datasets**. Training on just one may be an issue as these datasets tend to be pretty domain-specific.
  
  I am unsure of how exactly this was trained, but [this emotion detection model on huggingface](https://huggingface.co/j-hartmann/emotion-english-distilroberta-base) seems to do exactly this. If it was trained on the full dialogs (rather than single sentences/short utterances), then this would work for your use-case -- you may want to test the accuracy of it yourself first.
  
  
  ---
  
  