## In-Depth Analysis of Emotion Recognition through Knowledge-Based Large Language Models

| Bin Han              | Cleo Yau                     | Su Lei             | Jonathan Gratch      |
|----------------------|------------------------------|--------------------|----------------------|
| USC ICT              | California State Polytechnic | audEERING GmbH     | USC ICT              |
| Los Angeles, CA, USA | Pomona, CA, USA              | Gilching, Germany  | Los Angeles, CA, USA |
| binhan@usc.edu       | cleoyau@gmail.com            | slei@audeering.com | gratch@ict.usc.edu   |

## CONTENTS

| I   | Overview                                      | Overview                                                   |   1 |
|-----|-----------------------------------------------|------------------------------------------------------------|-----|
| II  | Emotion Probability given by Face             | Emotion Probability given by Face                          |   2 |
|     | II-A                                          | LSTM Model Details . . . . . . . . . . . . . . . . .       |   2 |
|     | II-B                                          | Emotional Probability Distribution: P ( e | f ) . . . . .  |   2 |
| III | Emotion Probability given by Face and Context | Emotion Probability given by Face and Context              |   2 |
|     | III-A                                         | GPT Prompt for Integration . . . . . . . . . . . . . .     |   2 |
|     | III-B                                         | Neural Network Integration . . . . . . . . . . . . . .     |   3 |
|     | III-C                                         | Emotional Probability Distribution: P ( e | c, f ) . . . . |   3 |
| IV  | Performance                                   | Performance                                                |   4 |
|     | IV-A                                          | Evaluation by Context (KLD, RMSE, and F1-Score)            |   4 |
|     | IV-B                                          | Confusion Matrix . . . . . . . . . . . . . . . . . . .     |   5 |
|     | IV-C                                          | How Integration improves Performance? . . . . . . .        |   6 |

## References

6

## Abstract

Emotion recognition in social situations is a complex task that requires integrating information from both facial expressions and the situational context. While traditional approaches to automatic emotion recognition have focused on decontextualized signals, recent research emphasizes the importance of context in shaping emotion perceptions. This paper contributes to the emerging field of context-based emotion recognition by leveraging psychological theories of human emotion perception to inform the design of automated methods. We propose an approach that combines emotion recognition methods with Bayesian Cue Integration (BCI) to integrate emotion inferences from decontextualized facial expressions and contextual knowledge inferred via Large-language Models. We test this approach in the context of interpreting facial expressions during a social task, the prisoner's dilemma. Our results provide clear support for BCI across a range of automatic emotion recognition methods. The best automated method achieved results comparable to human observers, suggesting the potential for this approach to advance the field of affective computing.

## I. OVERVIEW

In this work, we propose a general approach to context-dependent emotion recognition [1]. This approach consists of three main steps. The first step is to predict the emotions that people are likely to perceive from an emotional expression without context (see Section II). The second step involves predicting the emotions that people are likely to perceive from a situational description. Finally, we combine these separate sources of information using psychologically-inspired models, such as Bayesian Cue Integration [2] (see Section III). We test this approach with an emotional social task called the prisoner's dilemma (game details and corpus are explained in [1]).

## II. EMOTION PROBABILITY GIVEN BY FACE

This section focuses on emotion distribution estimated from facial cues alone. We compare three alternatives for automatically recognizing emotions from decontextualized videos.

## A. LSTM Model Details

We train an LSTM model that incorporates dynamic features using human context-free annotations as the ground truth. The model utilizes various input features, including sequences of Action Units (AUs), facial optical flows, gaze, and head pose data. Detailed descriptions of the model architecture are provided in Table I.

TABLE I: LSTM Model Architecture

| Layer Type                      | Configuration                                                                           |
|---------------------------------|-----------------------------------------------------------------------------------------|
| LSTM Dropout LSTM Dropout Dense | return sequences=True, units=50 rate=0.5 units=50 rate=0.5 units=7, activation='linear' |

## B. Emotional Probability Distribution: P e f ( | )

We treat context-free human annotations as ground truth (human annotation results) when people judge emotions without situational context. Figure 3 shows the emotional probability distribution for three facial emotion recognition methods: FACET, EAC, and LSTM. FACET and EAC overestimate joy across all conditions. FACET and EAC do not recognize surprise well, which could significantly affect their performance (see Table 3 in [1]).

Fig. 1: Emotional probability distribution for P e f ( | ) .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00780v1-figure-47f976e4-250c-4a19-a6f2-46da61cf3eea.jpg)

## III. EMOTION PROBABILITY GIVEN BY FACE AND CONTEXT

We compare alternative methods for integrating facial and contextual cues for context-aware emotion recognition. We explore three methods: BCI, GPT-4, and NN.

## A. GPT Prompt for Integration

LLMs are known for their ability to understand various situational cues and perform well, especially in terms of emotional capability [3]. We provided the LLM with situational knowledge and outcome to obtain emotional probabilities. This method utilizes the advanced capabilities of GPT-4 to directly generate a context-aware emotion probability distribution. The integration process involves GPT-4, which incorporates a representation of P e f ( | ) estimated by a context-free facial emotion recognition method. Additionally, we crafted prompts to integrate both context and face cues to the LLM. We used GPT-4. 1 . The Figure 2 shows the GPT prompt. The prompt contains four main components:

- · A general description of the prisoner's dilemma game [4].
- · The game outcome of each turn (CC,DC,CD, and DD).
- · Representation of P e f ( | ) estimated by a LSTM.
- · A request for the emotional distribution (Basic emotion).

## GPT Prompt

Imagine scenario where two people, Player A and Player B, play play multiple rounds with each other. In each round of the game, players each decide whether to Or steal from each get S5. If both choose steal" . each get $l. If one chooses "split" but the other chooses "steal" the stealer gets all $10. make their choices secretly and their choices are revealed at the end of the round. split they they They

In this round, Player A chooses to steal, while Player B chooses to steal, both $1 . get

Given the facial cue and the outcome, please consider how Player A might feel at the end of the game and generate probability distribution for their emotions. Provide your answer in the following format: 1 , Neutral: 2, Surprise: 3, Anger: prob 4, Disgust: 5. Fear: 6, Sadness: 'Joy: prob prob prob prob prob prob

Player As facial expression data, analyzed an LSTM model, 0.567 (a high level of happiness) Surprise: 0.12 (a small level of surprise) using Joy:

Fig. 2: GPT Prompt for P e c, f ( | ) : Integrating facial cue and the game outcome to predict Player A's emotion.

## B. Neural Network Integration

In this section, we discuss the Neural Network Integration (NNI) method. To improve integration, a simple neural network was trained using LSTM emotion distribution results ( P e f ( | ) ) and GPT-4 emotion distribution results ( P e c ( | ) ) as inputs, with context-based results ( P e c, f ( | ) ) as the ground truth. The NN architecture included a dense layer with 100 ReLU neurons and a softmax output layer. Using the Adam optimizer and a custom KL Divergence loss function, the model was trained over 1000 epochs. Cross-validation with 5 splits ensured robust evaluation. Table II shows the result of the comparison. To assess the performance of models, we employ three standard metrics. KLD [5] and RMSE are standard metrics to compare the distance between two probability distributions [6], [7]. Additionally, we adopt F1 (weighted) to evaluate performance if the model was forced to provide a single label. The NN's performance was somewhat worse than the BCI and GPT-4 integration results. It was only compared with the LSTM because both were fine-tuned on split-steal data, unlike the pre-trained models.

TABLE II: Comparison of alternative integration methods for P e c, f ( | ) using different combinations of facial (FACET, EAC, LSTM) and context (GPT-3 and GPT-4) and integration methods (BCI and GPT-4). The best results for each measure

| Face+Context (Integration)   |   KLD( ↓ ) |   RMSE( ↓ ) |   F1( ↑ ) |
|------------------------------|------------|-------------|-----------|
| FACET+GPT-3 (BCI)            |      1.713 |       0.215 |     0.525 |
| FACET+GPT-4 (BCI)            |      1.829 |       0.2   |     0.565 |
| EAC+GPT-3 (BCI)              |      1.34  |       0.2   |     0.519 |
| EAC+GPT-4 (BCI)              |      1.33  |       0.21  |     0.527 |
| LSTM+GPT-3 (BCI)             |      0.809 |       0.162 |     0.454 |
| LSTM+GPT-4 (BCI)             |      0.346 |       0.104 |     0.649 |
| Human+Human (BCI)            |      0.441 |       0.092 |     0.782 |
| FACET (GPT-4)                |      0.648 |       0.15  |     0.528 |
| EAC (GPT-4)                  |      0.597 |       0.155 |     0.503 |
| LSTM (GPT-4)                 |      0.354 |       0.112 |     0.53  |
| LSTM+GPT-4 (NNI)             |      0.58  |       0.151 |     0.151 |

## C. Emotional Probability Distribution: P e c, f ( | )

Fig. 3 illustrates the emotional distributions when integrating facial and contextual cue for emotion recognition. Methods integrating FACET and EAC show a propensity to overestimate joy in the CD. In addition, integrations with GPT-4 yield distributions that are more closely aligned with human context-based estimations than those with GPT-3.

Fig. 3: Emotional probability distribution for P e c, f ( | ) .

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00780v1-figure-3f5b3bd0-6dc3-4e5f-9085-c781b93438bb.jpg)

## IV. PERFORMANCE

## A. Evaluation by Context (KLD, RMSE, and F1-Score)

We assess the performance of knowledge-based emotion recognition against human context-based perception distributions. The LSTM+GPT-4 (BCI) shows robust performance across all contexts, mirroring the trend observed in human-to-human (BCI) assessments. Notably, most methods displayed weaker performance in CD and DD contexts, whereas LSTM+GPT-4 (BCI) and human-to-human (BCI) demonstrated relatively better performance in these contexts.

|                   |   Overall |    CC |    DC |    CD |    DD |
|-------------------|-----------|-------|-------|-------|-------|
| FACET+GPT-3 (BCI) |     1.713 | 1.301 | 1.158 | 2.407 | 1.986 |
| FACET+GPT-4 (BCI) |     1.829 | 1.147 | 1.946 | 1.948 | 2.275 |
| EAC+GPT-3 (BCI)   |     1.341 | 1.152 | 0.819 | 2.024 | 1.367 |
| EAC+GPT-4 (BCI)   |     1.33  | 0.886 | 1.486 | 1.495 | 1.453 |
| LSTM+GPT-3 (BCI)  |     0.809 | 0.394 | 0.935 | 0.385 | 1.521 |
| LSTM+GPT-4 (BCI)  |     0.346 | 0.222 | 0.376 | 0.389 | 0.395 |
| Human+Human (BCI) |     0.441 | 0.235 | 0.379 | 0.59  | 0.561 |
| FACET(GPT-4)      |     0.648 | 0.549 | 0.557 | 0.663 | 0.824 |
| EAC(GPT-4)        |     0.597 | 0.491 | 0.688 | 0.621 | 0.588 |
| LSTM(GPT-4)       |     0.354 | 0.179 | 0.291 | 0.538 | 0.407 |

TABLE III: Break down the KLD result by context

|                   |   Overall |     CC |     DC |     CD |     DD |
|-------------------|-----------|--------|--------|--------|--------|
| FACET+GPT-3 (BCI) |     0.215 | 0.134  | 0.189  | 0.272  | 0.267  |
| FACET+GPT-4 (BCI) |     0.2   | 0.132  | 0.175  | 0.254  | 0.238  |
| EAC+GPT-3 (BCI)   |     0.2   | 0.137  | 0.163  | 0.282  | 0.218  |
| EAC+GPT-4 (BCI)   |     0.21  | 0.138  | 0.194  | 0.26   | 0.251  |
| LSTM+GPT-3 (BCI)  |     0.162 | 0.097  | 0.241  | 0.108  | 0.202  |
| LSTM+GPT-4 (BCI)  |     0.104 | 0.068  | 0.113  | 0.116  | 0.117  |
| Human+Human (BCI) |     0.092 | 0.056  | 0.088  | 0.106  | 0.117  |
| FACET(GPT-4)      |     0.15  | 0.15   | 0.14   | 0.153  | 0.159  |
| EAC(GPT-4)        |     0.155 | 0.1484 | 0.1626 | 0.1518 | 0.1587 |
| LSTM(GPT-4)       |     0.112 | 0.076  | 0.098  | 0.145  | 0.131  |

TABLE IV: Break down the RMSE result by context

|                   |   Overall |    CC |    DC |    CD |    DD |
|-------------------|-----------|-------|-------|-------|-------|
| FACET+GPT-3 (BCI) |     0.525 | 0.882 | 0.676 | 0.333 | 0.3   |
| FACET+GPT-4 (BCI) |     0.565 | 0.861 | 0.838 | 0.375 | 0.338 |
| EAC+GPT-3 (BCI)   |     0.519 | 0.861 | 0.711 | 0.324 | 0.181 |
| EAC+GPT-4 (BCI)   |     0.527 | 0.861 | 0.773 | 0.323 | 0.328 |
| LSTM+GPT-3 (BCI)  |     0.454 | 0.882 | 0.045 | 0.39  | 0.058 |
| LSTM+GPT-4 (BCI)  |     0.649 | 0.861 | 0.804 | 0.192 | 0.51  |
| Human+Human (BCI) |     0.782 | 0.954 | 0.795 | 0.753 | 0.602 |
| FACET(GPT-4)      |     0.528 | 0.861 | 0.782 | 0.343 | 0.289 |
| EAC(GPT-4)        |     0.503 | 0.861 | 0.75  | 0.333 | 0.267 |
| LSTM(GPT-4)       |     0.53  | 0.861 | 0.773 | 0.142 | 0.333 |

TABLE V: Break down the RMSE result by context

## B. Confusion Matrix

Fig. 4 presents the confusion matrix for facial emotion recognition methods using human context-free annotations as the ground truth. All three methods tend to predict emotions as Joy. Fig. 5 shows the confusion matrix for facial and contextual emotion recognition. When integrating context, the methods tend to predict a more diverse range of emotions.

Fig. 4: Confusion matrix for facial emotion recognition. Human context-free is used as a ground truth ( J stands for Joy, N for Neutral, Su for Surprise, Sa for Sadness, D for Disgust, F for Fear, and A for Anger).

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00780v1-figure-478824d6-a1c2-4237-a816-8a592745d537.jpg)

Fig. 5: Confusion matrix for facial and contextual emotion recognition. Human context-based is used as a ground truth.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00780v1-figure-e153fd30-ac14-461b-ae87-39e5969a4ffa.jpg)

## C. How Integration improves Performance?

We conduct an analysis to assess how BCI enhances recognition performance. Fig. 6 illustrates the variation in recognition performance according to different game outcomes.

Fig. 6: Enhancement in model performance (RMSE and KLD) through facial and context integration.

![image](https://ac.marcodsn.me/arxiv-markdown-images/2408.00780v1-figure-ce53532f-1b80-4059-8d23-2f89d7d903c6.jpg)

## REFERENCES

- [1] B. Han, S. Lei, C. Yau, and J. Gratch, 'Knowledge-based emotion recognition using large language models,' in 2024 12th International Conference on Affective Computing and Intelligent Interaction (ACII) , Glasgow, UK, 2024.
- [2] D. C. Ong, J. Zaki, and N. D. Goodman, 'Affective cognition: Exploring lay theories of emotion,' Cognition , vol. 143, pp. 141-162, 2015.

- [3] A. N. Tak and J. Gratch, 'Is gpt a computational model of emotion?' in 2023 11th International Conference on Affective Computing and Intelligent Interaction (ACII) . IEEE, 2023, pp. 1-8.
- [4] A. Rapoport and A. M. Chammah, Prisoner's dilemma: A study in conflict and cooperation . University of Michigan press, 1965, vol. 165.
- [5] S. Kullback and R. A. Leibler, 'On information and sufficiency,' The annals of mathematical statistics , vol. 22, no. 1, pp. 79-86, 1951.
- [6] K.-C. Peng, T. Chen, A. Sadovnik, and A. C. Gallagher, 'A mixed bag of emotions: Model, predict, and transfer emotion distributions,' in Proceedings of the IEEE conference on computer vision and pattern recognition , 2015, pp. 860-868.
- [7] S. Zhao, G. Ding, Y. Gao, and J. Han, 'Approximating discrete probability distribution of image emotions by multi-modal features fusion,' Transfer , vol. 1000, no. 1, pp. 4669-4675, 2017.
Bin Han Cleo Yau Su Lei Jonathan Gratch
In-Depth Analysis of Emotion Recognition through Knowledge-Based Large Language Models
Emotion recognition in social situations is a complex task that requires
integrating information from both facial expressions and the situational
context. While traditional approaches to automatic emotion recognition have
focused on decontextualized signals, recent research emphasizes the importance
of context in shaping emotion perceptions. This paper contributes to the
emerging field of context-based emotion recognition by leveraging psychological
theories of human emotion perception to inform the design of automated methods.
We propose an approach that combines emotion recognition methods with Bayesian
Cue Integration (BCI) to integrate emotion inferences from decontextualized
facial expressions and contextual knowledge inferred via Large-language Models.
We test this approach in the context of interpreting facial expressions during
a social task, the prisoner's dilemma. Our results provide clear support for
BCI across a range of automatic emotion recognition methods. The best automated
method achieved results comparable to human observers, suggesting the potential
for this approach to advance the field of affective computing.