Twitter Arabic Sentiment Analysis to Detect Depression Using Machine Learning

: Depression has been a major global concern for a long time, with the disease affecting aspects of many people’s daily lives, such as their moods, eating habits, and social interactions. In Arabic culture, there is a lack of awareness regarding the importance of facing and curing mental health diseases. However, people all over the world, including Arab citizens, tend to express their feelings openly on social media, especially Twitter, as it is a platform designed to enable the expression of emotions through short texts, pictures, or videos. Users are inclined to treat their Twitter accounts as diaries because the platform affords them anonymity. Many published studies have detected the occurrence of depression among Twitter users on the basis of data on tweets posted in English, but research on Arabic tweets is lacking. The aim of the present work was to develop a model for analyzing Arabic users’ tweets and detecting depression among Arabic Twitter users. And expand the diversity of user tweets, by adding a new label (“neutral”) so the dataset include three classes (“depressed”, “non-depressed”, “neutral”). The model was created using machine learning classifiers and natural language processing techniques, such as Support Vector Machine (SVM), Random Forest (RF), Logistic Regression (LR), K-nearest Neighbors (KNN), AdaBoost, and Naïve Bayes (NB). The results showed that the RF classifier outperformed the others, registering an accuracy of 82.39%.

tool for disseminating information and a rich resource of views on many topics, including politics, business, economics, and social issues [2]. And it can be helpful to state people's emotions.
The members of Saudi Arabian society are very active on Twitter; as reported in a Business intelligence (BI) study, 41% of online users in the country use the Twitter platform [3]. Most Twitter users tend to express their feelings through texts, media, emojis, and hashtags. Instead of speaking out about their problems, they share their emotions through tweets. Sometimes, people share private information, such as their health statuses such as depression symptoms.
Depression is one of the most well-known mental health disorders and it is considered as a major issue for mental health practitioners. Therefore, there is a need for a system to detect the depression, which can help in the prevention and detection of depression. Motivated by this need, in this paper, we propose to develop a model for analyzing Arabic users' tweets and detecting depression among Arabic Twitter users.
In consideration of the above-mentioned issues, this study was aimed at experimentally identifying depressed Arabic Twitter users on the basis of tweet data. The examination was conducted using natural language processing (NLP) and machine learning techniques. Although such an experiment has been widely conducted on the English language, limited research has been done on the Arabic language. So, this research aims to expand this field since there is a lack of studies in Arabic.
To achieve the objective of the study, we documented tweets from people who completed the Center for Epidemiologic Studies Depression (CES-D) scale [4] or self-declared themselves as suffering from depression over the Twitter platform. The recorded tweets (4542 tweets), were then labeled as posted by depressed, neutral, and non-depressed before the dataset was pre-processed using NLP. A supervised machine learning model for classifying the tweets into appropriate categories was developed. The models used which are SVM, RF, LR, KNN, AdaBoost, and NB were compared using performance measures (accuracy, recall, precision, and F1 score), after which the most accurate was used to predict the occurrence of depression among users on the basis of Arabic tweets. This paper is organized as follows: Section 2 discusses the Review of Related Literature and Background. Section 3 presents the Description of the Methodology. Section 4 presents the Performance Measurement. Section 5 presents the Experimental Results and Discussion. Finally, the Conclusion and Future Work are identified in Section 6.

Review of Related Literature and Background
This section presents a background to Arabic sentiment analysis and depression and reviews related literature.

Arabic Sentiment Analysis
Conducting a sentiment analysis of texts in the Arabic language is more complex than that directed toward English texts because the former is characterized by more forms than other languages. The formal variant of Arabic is Modern Standard Arabic (MSA), but this is rarely used in spoken interactions. The most frequently used informal variant is Dialectal Arabic (DA), especially for communication purposes. A total of 30 major Arabic dialects differs from MSA, the approaches used to translate difficult MSA terms are ineffective when applied to DA translation. Recently, Arabic researchers have developed solutions for different dialects, but these remain minimally inaccurate and cover only a few dialects [5].

Depression
Depression is considered a global concern. It is a very common disease, as it affects people across the world. Over 264 million people have recently been afflicted with depression [1], which comes in many forms, each accompanied by its own symptoms [6]. The most popular form is major depressive disorder (MDD), which influences the ability of individuals to do daily tasks, such as sleep, eat, study, work, and have fun [6]. Depression does not have a target age, as it may begin at a young age. Curbing depression is essential to saving people's lives [7]. In Arabic culture, the stigma on mental illness is deeply entrenched, and there is a lack of awareness regarding this issue [8].

Related Literature
De Choudhury et al. [9] documented timeline tweets from 476 users who answered the CES-D questionnaire. They found that depressed people engage in fewer activities, express negative emotions, and share more medical and relational concerns online than individuals without this illness. Using SVM, the authors constructed a model whose best performance was an RBF kernel scoring of 70% in terms of accuracy.
Nadeem et al. [10] recorded tweets from Twitter users who set their profiles to "public" and declared over the platform that they have depression (e.g., a post that says, "Today I was diagnosed with depression."). Then, they used a bag-of-words approach, which involved placing every word in a bag and then measuring its frequency of appearance. Finally, they tested different supervised classifiers and found that the optimal accuracy (i.e., 86%) was achieved by a naïve Bayes classifier.
Multiple datasets were used by Leis et al. [11] in their research. In the first phase, they used three datasets, namely, two depressive tweet datasets and one control dataset. In the second phase, they compared their analyses and used part-of-speech (POS) tagging with an NLP tool to identify the behavioral patterns of depressed Twitter users. The results revealed that depressive users less actively post tweets than non-depressed users. When they tweet, they do so more frequently between 23:00 and 6:00. The findings also indicated that verbs and the first-person singular are most often used by depressive users.
Aldarwish et al. [6] used two datasets. The first dataset was collected from three social network sites (SNS): Facebook, Live Journal, and Twitter. Using the 2073 depressed posts and 2073 non-depressed posts, the first dataset was manually trained to categorize posts into one of the nine symptoms of depression as defined by the American Psychiatric Association Diagnostic and Statistical Manual (DSM-IV). The second dataset contained all patient SNS posts. They proposed a system that uses Rapid Miner to test SVM and NB models. When using a supervised NB model, the results showed 100% precision, 57% recall, and 63% accuracy.
Stephen et al. [12] collected their dataset from users who identified as depressed. To reach more users, they implemented specific keywords used by those who identified as depressed. Their aim was to identify how to calculate depression levels by evaluating the sentiment score produced by the algorithm. The score was checked manually to determine whether the algorithm could correctly understand depression levels within user posts. Three lexicons were used for sentiment analysis, including BING, NRC, and AFINN. Each of the three lexicons gave a final sentiment score with differing magnitudes. The average after normalizing the values into "−1" and "+1" was taken to calculate the final sentiment score. When examining Twitter cases, the authors determined that user engagement patterns on Twitter were different for all posts except those demonstrating the depression interval.
In 2019, an experiment with two phases was conducted by Almouzini et al. [13]. The first phase collected Tweets from Arabic Twitter users who answered CES-D questionnaires. The questionnaires were then labeled as either "depressed" or "non-depressed" based on CES-D and Patient Health Questionnaire-9 (PHQ-9) scales. After being labeled, the dataset was cleaned, and the features were extracted as feature vectors consisting of negation handling and a bag-of-unigrams. The second phase generated a supervised machine learning model that predicted which Arabic users were depressed. Using a Liblinear classifier in this phase, optimal accuracy was recorded at 87.5%. Tab. 1 summarizes the related literature section.  [13] has only been conducted once when using two label classes ("depressed" and "non-depressed") to evaluate the Arabic language. To expand the diversity of user Tweets included in this study, a new label ("neutral") was added that contained Tweets not considered "depressive," as they exist without subscribing to any symptoms, such as prayers, song lyrics, or Duaa (prayers). Through this study, six classifiers were be used to predict whether users were depressed: SVM, RF, LR, KNN, AdaBoost, and NB.

Description of the Methodology
This section will discuss the: Dataset collection, dataset-preprocessing, features extraction and generating the models as shown in Fig. 1.

Dataset Collection
The dataset was collected from Arabic Twitter users who answered the CES-D survey. Moreover, as the study's target audience was Arabic speakers, the CES-D scale was translated into Arabic. The CES-D scale is a short, self-reporting scale that contains 20 questions. It is designed to measure depressive symptomatology [4]. Two questions were added to the survey: the first question asked if the user could include their Twitter username. The second question asked if the user tended to express their feelings on Twitter. In addition to the CES-D survey, the dataset was collected by using a dictionary of depressive phrases. For example, which means, "I am diagnosed with depression." 3468 CMC, 2022, vol.71, no.2

Dataset Labeling
All Tweets collected and filtered from the users were labeled manually. The dataset was divided into three labels: ("depressed," "non-depressed," and "neutral.") Within the ("depressed") class, Tweets contained at least one symptom of depression. Within the ("non-depressed") class, Tweets contained positive messages, such as, meaning, "I am happy today, I love my friends." Finally, the ("neutral") class contained Tweets that are considered neither ("depressed" or "non-depressed") in nature, such as songs, movies, Duaa (prayers), a famous quote, advice, or a question.

Description of Dataset
The dataset contains three columns: Column [0] represents the Tweet's full text, and Column [1] represents the signs of depression within the Tweet. The signs of depression include the nine symptoms described in the American Psychiatric Association Diagnostic Depression Nine Symptoms [14] (shown in Tab. 2), as well as two additional signs found frequently within depressed Arabic Twitter usersmood swings and feeling lonely. The final sign was a generalized term, as some Tweets did not provide a specific symptom. For example, , which means, "I'm very depressed today." As such, this was considered a general sign titled "feeling depressed." The last column, Column [2], contained the three class labels ("depressed," "non-depressed," and "neutral"). Fig. 2 presents the percentage of each depressed class symptoms in the dataset. The study showed that 45% of the Tweets generally talked about how sad and depressed they were, 9% of the Tweets indicated that depressed Arabic users wrote about their lack of sleep, 9% of the Tweets indicated that they had gotten too much sleep, and 8% of the Tweets revealed suicidal thinking; a sensitive topic and dangerous sign of depression amongst users.

Dataset Pre-Processing
In this stage, the dataset was cleaned and prepared with the proper application of natural language processing (NLP) techniques, including normalization, stop word filtering, and stemming. This was applied as follows:

Normalization
In this step of pre-processing, the dataset was cleaned and normalized into a uniform text. This step was implemented by writing a Python script using pyarabic libraries and the regular expression "re." The application of the normalization stage was conducted through the following steps: Step 1: Removing usernames, URLs, hashtags, punctuation, and repeated letters. For example, was converted into Arabic characters were also normalized into one representation. For example, the alef letter was converted to Step 2: Removing Arabic diacritics such as the Tatweel. For example, was converted into . The Tashkel, for example, was converted into 3470 CMC, 2022, vol.71, no.2

Stop-word Removal
Stop words are terms that are removed from the text without impacting its meaning. These words appear more frequently in a document [15]. In addition to dialect stop words (e.g., ), a large, open source Sourceforge Arabic stop words dictionary was used for this step. Further, negation words (e.g., ) were kept, as they change the meaning of the text.

Tokenization
Tokenization is the process of splitting sentences into smaller, more meaningful sections (Tokens) to help text exploring [16]. For example, was tokenized into This step was done by using the pyarabic tokenize method.

Stemming
The stemming phase included removing large portions of a word, like a prefix or suffix, in order to return it to the root of the word. For example, was converted into In Arabic, a root word is a word used in its basic form so that multiple nouns and verbs can be created by adding affixes to it [17]. This step was implemented by accessing the nltk library using the ISRIStemmer proposed by Taghva et al. [18].

Features Extraction
We aimed to find the best techniques to reach our goal, which was to detect depression within Arabic tweets using machine learning. We accomplished this by finding the best model features that gave the highest performance accuracy. Different N-gram ranges and TF-IDF methods were used to extract the required features. Six supervised machine learning models were used to train the dataset, including: SVM, RF, LR, KNN, AdaBoost, and NB. As a result, we found the feature combination that gave optimal accuracy.

Term Frequency-Inverse Document Frequency (TF-IDF)
Term frequency-inverse document frequency (TF-IDF) is commonly used for text classification. Term-frequency is the number of times a word appears within a document (Eq. (1)). Inverse document frequency is a weight term scheme that gives tokens that appear more frequently in documents a lower impact, or weight, and gives tokens that occur less frequently a higher weight [19] (Eq. (2)).
where "TF(t, d)" is the number of times the word "t" appears in the document "d" and, where "n" is the total number of documents, and "DF(t)" is the number of documents that contain the word "t".

N-gram
N-grams extract characters or words from a text, and are used in stemming, spelling check, and text compression. N-grams are used as an approach to find similarities between N-grams. The N value must be an integer and is set to be a unigram n = 1 (one word or character), a bigram n = 2 (two words or characters), or a trigram n = 3 (three words or characters) [20]. For applying the N-gram technique, we used the Sklearn. Feature_extraction. text Count Vectorizer class that passes an N-gram range (min, max). This class combines the N-gram sequences into one dictionary. The N-gram range (1,1) was set to unigram only, and the same logic applied to the ranges (2,2) and (3,3). For example, the text, "I was very upset today" was divided into the following: This class ignores any text that is less than a uni-character (e.g., "a" and "I").

Generating the Models
Six supervisor machine learning models were implemented using Python, Sklearn, and Grid-SearchCV libraries to determine the classification models. The classifiers were as follows:

Naïve Bayes (NB)
NB is a probabilistic classifier that uses the Bayes theorem, where all features (attributes) are assumed to be independent of each other [21]. The following equation (Eq. (3)) shows the NB model: where "P(C|X)" is the posterior probability of class given predictor, "P(X)" is the prior probability of predictor, "P(C)" is the prior probability of class, and "P(X|C)" is the likelihood of the probability predictor class given.

Support Vector Machine (SVM)
SVM is a classifier that uses risk minimization theory to find the optimal separating hyperplane within the feature space. Simple SVM is used for linear regression and classification problems. To use the SVM classifier in a non-linear space, kernel functions must be used (e.g., RBF and Sigmoid) [22]. The following equation (Eq. (4)) demonstrates the RBF kernel: where "||X − Y || 2 " is the Euclidean distance between the two points "X" and "Y," and "σ " is the hyperparameter and variance.

Random Forest (RF)
RF is a combination of tree predictors that predicts the class label by randomly generating a forest. The forest is a collection of multiple decision trees, and each tree has the value of a random vector sampled independently. Each tree is then distributed equally among all trees. The final classification is based on majority voting [23].

Adaboost (Ensemble)
AdaBoost (Ensemble) is an approach applied to textual or numeric data types. Ensembles split the data repeatedly and continue to re-assign different weights to the training data. This ensures misclassified data from the first split will be reassigned correctly during the next data split. This process continues until the best data split is found [24].

K-Nearest Neighbors (KNN)
KNN is a classifier that uses distance, such as the Euclidean distance function, to classify new, unknown data points based on existing data points nearby. The distance function calculates the distance between two points, and the "K" value shows the size of the neighborhood. Unknown data points are classified based on simple voting [25].

Logistic Regression (LR)
LR is a binary logistic model. LR is used based on one or more features to estimate the probability of binary response. The author Nadeem describe LR as a discrete choice model, as it is not technically qualified as a classification method. The relationship between the binary variables and the features are clarified through the use of the (Eq. (5)) below. However, for multi-class text classifications, a multinomial extension must be used [10].
where "F(X)" expresses the dependent variable probabilities, and "β0 + β1X" expresses the perfect fit parameters for the sake of success cases.

Performance Measurement
The evaluation of classifiers was measured with accuracy, precision, recall, and the use of an F1score. We calculated the performance measurement for the classifiers with the following equations, where "TP" represents a true positive, "TN" represents a true negative, "FP" represents a false positive, and "FN" represents a false negative: Precision was measured by calculating the false positives of the classifier.
Recall was measured by calculating the false negatives of the classifier.
The F1-score was calculated by taking the weighted harmonic average of the recall and precision measurements.

Experimental Results and Discussion
In this section, a comparison of all classifiers is presented and a discussion of the effect of TF-IDF on text classification. In Tab. 4 This comparison of all classifiers is based on the results of the accuracy, recall, precision, and F1-score model evaluation results. These results were obtained from the optimal parameters of the grid. After collecting and pre-processing the dataset, the dataset was split: 80% of the data was designated for training, and 20% was designated for testing in addition to using a 10-fold cross-validation. By comparing the performance of the six classifiers through the use of N-grams and TF-IDF, RF obtained the highest accuracy rate at 82.39%, and scored 82.40%, 82.67%, and 82.53% for precision, recall, and F-score, respectively. The classifier that showed the least accuracy was KNN, with a rate of 79.75%.
Since social media provides a wide platform for personal expression, people tend to interact with each other. In contrast, those who are depressed are less active and are often more isolated from others in social media. One method of viewing the online interactions of depressed Arabic users is through their mentions count. The more mentions users have, the more interactive they get. To find out how much depressed, non-depressed and neutral Twitter users are interactive with mentions, we counted the numbers of times a user has mentioned or replied to other people with a counter using a Python script. Based on the results of the counters shown in Fig. 3, it is obvious that depressed Twitter users are less interactive with other users. This result provides evidence that depressed Twitter users are more isolated than non-depressed Twitter users.

Figure 3: Mention interaction
Another avenue for examining Arabic Twitter users is evaluating the emojis they use within their Tweets. Emojis show emotions and can sometimes be better than words. To analyze the emojis used by users we counted the most frequent emojis used by all users in the dataset. Based on the emoji count results, Fig. 4 illustrates that depressed users chose negative emojis more frequently, such as the broken heart emoji or the crying face emoji. It was also shown that these users tend to use less emojis than non-depressed users.

Figure 4: Emojis interaction
Figs. 5 and 6 show the word cloud for depressed and non-depressed classes. The most frequently used Arabic words appear in the bigger font, and the less frequently used Arabic words appear in the smaller font. So, this indicates the most used words by depressed people to express their emotions for the depressed class, and it is the same for the non-depressed class.
The feature extraction was tested in the dataset with N-grams with count vectorizer only and Ngrams with count vectorizer and TF-IDF vectorizer. Fig. 7 illustrates that the TF-IDF method that integrates N-grams improves classifier performance.   Based on the results discussed above, Arabic people do share their feelings on Twitter. The results show that depressed people demonstrate specific behaviors within their tweets. Users do not show a lot of interaction with others based on their mentions count, and they often use negative words to describe their symptoms, like suicidal thoughts or sleeping disorders. Further, though these users do not extensively use emojis, they do use them in a negative way. This study evaluated the consistent behavior of Tweets to train a supervised machine learning model. Arabic Tweets were collected and labeled manually into ("depressed," "non-depressed," and "neutral") categories, a label ("neutral") was added to expand the diversity of user tweets. After pre-processing the dataset, six different supervised machine learning text classifiers were used to predict which Twitter users were depressed. The optimal performance of this research was obtained by RF with an accuracy of 82.39%.
In terms of future work, the performance of the machine learning model might be improved by increasing the size of the collected dataset, testing more classifiers, and including additional features in the dataset such as the time of Tweets.


MultiMediate '22: Backchannel Detection and Agreement Estimation in Group Interactions

Backchannels, i.e. short interjections of the listener, serve important meta-conversational purposes like signifying attention or indicating agreement. Despite their key role, automatic analysis of backchannels in group interactions has been largely neglected so far. The MultiMediate challenge addresses, for the first time, the tasks of backchannel detection and agreement estimation from backchannels in group conversations. This paper describes the MultiMediate challenge and presents a novel set of annotations consisting of 7234 backchannel instances for the MPIIGroupInteraction dataset. Each backchannel was additionally annotated with the extent by which it expresses agreement towards the current speaker. In addition to a an analysis of the collected annotations, we present baseline results for both challenge tasks.

INTRODUCTION
Backchannels, i.e. short interjections made by listeners, are at the core of the bilateral nature of dialogue [24]. That is, dialogue in which a listener's responses affect the speaker's speech acts. They may consist of a verbal phrase like "oh yes", a sound (e.g. "hm"), or bodily gestures, such as head nods or hand movements, or combinations thereof. Backchanneling serves the important functions of confirming listeners' attention and comprehension [33], as well as regulating turn-taking [60]. In addition, it is used to communicate agreement or disagreement with the current speaker and is therefore an important indicator of participants' opinions and intra-group relations [12]. The inability to appropriately perform backchanneling can have severe impact on dialogue, e.g. by distracting the speaker [10,51].
As a result, automatic analysis of backchanneling behaviour is crucial and has significant potential for artificial systems designed to passively monitor or actively mediate human dialogue [6,52,53,61]. For example, artificial mediators [14,17,25] could analyse the frequency of backchannels to gauge participants' engagement and encourage those who are disengaged from the discussion. Agreement or disagreement expressed in backchannels could also help artificial mediators to better understand the opinions of participants. This may enable mediators to give a voice to participants who find it difficult to express diverging opinions. Despite this potential, automatic analysis of backchannels in group interactions remains largely unexplored due to the lack of suitable datasets.
With MultiMediate '22, we present the first challenge for automatic backchannel analysis in group interactions. To this end, we introduce the first publicly available dataset of backchanneling behaviour in group discussions. We fully annotated the MPI-IGroupInteraction dataset [49] with 7234 backchannel instances. Furthermore, each backchannel instance was rated with the extent to which it expresses agreement with the current speaker. We present analyses of our novel annotations as well as evaluations of baseline approaches for MultiMediate '22. All collected annotations, baseline implementations, and raw feature representations are made publicly available for further use. 1

PREVIOUS WORK 2.1 Backchannel and Agreement Datasets
While a number of different datasets were recorded for the purpose of backchannel analysis [10,22,44,55], they are rarely public and not sufficient to ensure progress on automatic backchannel analysis in group interactions. In Table 1 we list recent publicly available datasets that were used in BC research, the majority of which consists of spoken dialogues. Note that while these datasets are publicly available, the backchannel annotations collected on them are often not (column "Pub." in Table 1). Furthermore, all existing datasets (except the Canal9 Corpus [54,67] for which no publicly available backchannel annotations exist) consist of dyadic interactions and are thus not fitting for group behaviour analysis.
In terms of annotations of backchannels, some studies use semiautomatic labeling of potential events [9,18], or query a dataset for specific BC keywords [50]. Backchannel annotations are most often based on a set list of backchannel events ("is there a "uh-huh" here?") [18,27,45,50,63], and only rarely on the holistic perception of a backchannel ("is there a backchannel here?") [5,11,35].
While a relationship between backchannels and agreement has been discussed [60], few studies have systematically investigated it. [55] included agreement in their annotations of backchannels, but did not analyse it; [30,36] distinguish backchannels from agreement signals. Some studies of agreement include backchannels [19,20], but to the best of our knowledge, no study of backchanneling investigated agreement transmitted through the backchannels themselves.
In our work, we present the first publicly available annotations of backchannel occurances and agreements expressed via backchannels in group interactions. Our full dataset is manually annotated and with over 33 hours of annotated human behaviour across training, validation and test sets, its size is equal to the largest dyadic BC dataset currently available (see Table 1).

Computational Models
The prediction (i.e. anticipation) of backchannels is a highly active area of research in social signal processing. One major motivation is the goal to generate natural backchanneling behaviour in artificial agents. There is a variety of traditional machine learning methods used for predicting backchannels [27,29,31,40,44,64]. Recently, LSTM networks became the most frequent choice [1,2,34,58], along with residual networks [31]. Multitask learning also seems to be a particularly successful approach [34,37,39]. The most common features used in BC prediction are prosodic [2,34,57,58] while some research also makes use of linguistic features such as word embeddings [1,2,57,58] or syntactic (part of speech tags), semantic (concreteness, valence) and discourse features [16,18,40]. Research shows that adding lexical features to accoustic ones improves results [50,57]. [33] offers a review of the most important backchannel-inviting cues which may also be used as features. In contrast to backchannel prediction, we define the task of backchannel detection as categorizing observed behaviour as to whether a backchannel was shown or not. While to the best of our knowledge, this task was not studied in isolation in previous work, backchannel detection can appear as a part of the multi-class problem of dialogue act classification (DAC) [3,59]. DAC is primarily addressed by text analysis [32,42,56] and only few works incorporated multi-modal information like emotional expressions [21,59]. Importantly, no previous work addressed the task of backchannel detection in group interactions from multi-modal behaviour.
With MultiMediate '22 we aim to attract researchers to two challenging problems centered around backchannels in group discussions: backchannel detection and estimation of the amount of agreement expressed in a backchannel. In line with human studies that underline the importance of non-verbal backchannel cues [5,41], we provide annotations of backchannel behaviour that can take place both via speech and in the visual domain. We contribute to improved comparability between approaches by using an unpublished test set for evaluation.

CHALLENGE DESCRIPTION
As in MultiMediate '21 [47], our challenge is based on the MPI-IGroupInteraction dataset [48,49]. This dataset has been used for diverse tasks, including low rapport detection [49], emergent leadership detection [46], eye contact detection [28,48], next speaker prediction [15], and body language detection [7]. For MultiMediate '22 we collected novel backchannel annotations on the whole dataset. Test samples (excluding ground truth) are released to participants before the challenge deadline. Participants in turn submit their predictions for evaluation by the challenge organisers. We first describe MPIIGroupInteraction and subsequently discuss annotation procedures and task definitions for backchannel detection and agreement estimation.

Dataset
Training data. We use the publicly available recordings of MPI-IGroupInteraction [49] as training data. The dataset comprises 22 conversations between three to four people on a maximally controversial topic, lasting 20 minutes each. While discussing the topic, interactants were recorded with four microphones and eight framesynchronised video cameras (see Figure 1).
Evaluation data. We follow MultiMediate '21 [47] and use six yet unpublished discussions that were recorded during the creation of MPIIGroupInteraction [49] for testing. These six discussions followed the same procedure as the training discussions with the only exception that the topic was not chosen to be maximally controversial. Instead, a topic was randomly selected for each group and participants were asked to take on opposing views for themselves.

Backchannel Detection Task
Backchannel Annotations. In a first step annotators where asked to label the occurrences of backchanneling behaviour with respect to different modalities: auditive, and visual. Visual annotations take only the aspects observable in the video into account (e.g. nodding or head shaking). Auditive backchanneling behaviour, in turn, relies only on cues from the audio signal, encompassing verbal (e.g. "yes", "oh really?") and paraverbal behaviour (e.g. "uhm hum", "aha"). Each of three annotators labeled a specific portion of the dataset with respect to backchannels for each modality. Figure 2 (left) shows the average of annotated backchannel events annotated per participant in each modality. On average, we observed 28 annotations per participant for the auditive modality, and 60 for the visual modality. Figure 2 (middle) shows the average duration of annotated backchannel events for each modality. Visual backchannel events are on average longer (1.8 seconds) compared to auditive backchannel events (0.56 seconds). The modality specific labels were subsequently joined using the logical OR operator. Hence, overlapping backchannels in different modalities (e.g. a person nods while simultaneously saying "yes") are merged to a new, modality independent backchannel label. To provide negative examples we also calculated an equal amount of non-backchannel samples per session. Overall this results in 14468 labels partitioned into the following splits: 6716 Train, 2854 Val, 4898 Test.
Task definition. Given an observation window of 10 seconds participants have to detect if there is a backchannel present in the sample. Every sample has been created such that the end of an annotated backchannel has been used as the end the sample or that a sample consists of 10 seconds with no annotated backchannels. The performance metric for the task is accuracy.

Backchannel Agreement Estimation Task
Agreement Annotations. In the second step, all backchannel instances that where annotated for the backchannel detection test were labeled with respect to their level of expressed agreement on a scale from -1 (total disagreement) to 1 (total agreement) using a step size of 0.1. In case annotators found that the instance was erroneously labeled as a backchannel, they indicated this fact with an extra label. When all annotators agreed that an instance was wrongly labeled as a backchannel the respective sample has been removed from the dataset during the sampling process. This annotation step has been performed by each annotator for all backchannel labels (see section 3.2) The groundtruth labels were then created by averaging all three annotations per sample. Overall this results in 7234 labels partitioned into the following splits: 3358 Train, 1427 Val, 2449 Test. To quantify the reliability of agreement annotations, we compute the average Spearman when comparing a left-out annotator to the average of the two remaining ones is. We observe a of 0.62, indicating substantial agreement. Figure 2 (right) shows the distribution of aggregated agreement annotations. The distribution is centered in the positive range between 0 and 0.5. The detection of disagreement represents a special challenge due to the small number of samples below 0.
Task definition. For the backchannel agreement estimation tasks participants have to predict the average expressed agreement per sample on scale from -1 to 1. The performance metric for the task is mean squared error.

Features
We extracted the same set of features for backchannel detection and agreement estimation from backchannels. Features were extracted from the last second of the 10 second input window and aggregated by computing the mean or the mean of absolute differences ("mean delta") of adjacent frames over this second.

Head Features.
We extracted features from participants' head and face using OpenFace 2.0 [8]. These include mean and mean delta of AU intensity estimates, mean delta of head orientation and translation, as well as mean delta of gaze angles for both eyes. In total, we extracted 46 features based on OpenFace 2.0.

Pose Features.
We extract body pose estimates using Open-Pose [23] and employ a set of angular features that proved successful for group interaction analysis [13]. These features consist of angles between body parts, e.g. the angle defined by the line between left shoulder and left elbow and the line between left shoulder and right shoulder. We limit ourselves to the "Upper Body" and "Head" features described in [13], as lower body pose estimates tend to be unreliable on the MPIIGroupInteraction dataset. We compute the mean delta of these features, resulting in 8 body pose features.

Voice Features.
We extracted the extended Geneva Minimalistic Acoustic Parameter Set (eGeMAPS) [26] on the last second of the input window. This set consists of 88 acoustic parameters that are commonly applied to tasks like depression, mood, and emotion recognition [66], or Alzheimer's Dementia recognition [43].

Prediction Approach
For backchannel detection, we trained a binary Support Vector Classifier (SVC) with rbf kernel. For the agreement estimation task, we trained a Support Vector Regressor (SVR) with rbf kernel. We employed 10-fold cross-validation on the training set to choose and parameters of the SVC/SVR. For test set evaluations, we trained on training and validation sets; for evaluations on the validation set we only trained on the MultiMediate '22 training set.

Results
We present evaluation results for different combinations of feature sets in Table 2. For fairness reasons, we only evaluate two feature sets for each task on the training set: the best performing feature set on the validation set, as well as all included features.

Backchannel Detection.
The best feature set on the validation set consisted of a combination of head and pose features, reaching 0.639 accuracy on validation and 0.596 accuracy on the test set. This clearly outperformed the trivial baseline of a random predictor at 0.5 accuracy and was marginally better than all featuresets combined. Notably, our experiments on the validation set revealed that each individual feature set achieved above-random performance on backchannel detection, even though OpenFace 2.0 based head features were clearly leading. An ablation of the head feature set revealed that head pose alone (i.e. mean delta of translation and rotation of the head) reaches 0.636 accuracy. This is likely due to the strong association between backchanneling and nodding.

Agreement Estimation.
For agreement estimation, head pose features performed best on the validation set (0.075 MSE) and reached 0.061 MSE on the test set. In both cases this improves above the trivial baseline of using the mean on the training set as a predictor. However, featuresets that do not include head pose or  gaze features fail to outperform the trivial baseline. This is in contrast to the backchannel detection task and indicates the difficulty of backchannel agreement estimation.

CONCLUSION
We introduced MultiMediate '22, the first challenge addressing backchannel detection and agreement estimation from backchannels in well-defined conditions and evaluated baseline approaches for each task. In addition we introduced a novel publicly available dataset of backchannel annotations in group interactions that is a valuable resource for research on backchannel detection and agreement estimation, even beyond the MultiMediate challenge.


extfieldsofstudy:
Computer Science
Engineering
Mathematics

text:
Optimal Precoding Design for Monostatic ISAC Systems: MSE Lower Bound and DoF Completion

In this letter, we study the parameter estimation performance for monostatic downlink integrated sensing and communications (ISAC) systems. In particular, we analyze the mean squared error (MSE) lower bound for target sensing in the downlink ISAC system that reveals the suboptimality in re-using the conventional communication waveform for sensing. To realize a practical dual-functional waveform, we propose a waveform augmentation strategy that imposes an extra signal structure, namely the degrees-of-freedom (DoF) completion method. The proposed approach is capable of improving the parameter estimation performance of the ISAC system and achieving the derived MSE lower bound. To improve the performance of the proposed strategy, we formulate an MSE minimization problem to design the ISAC precoder, subject to the communication users' signal-interference-plus-noise-ratio (SINR) constraints. Despite the non-convexity of the waveform design problem, we obtain its globally optimal solution via semi-definite relaxation (SDR) and the proposed constructive method. Simulation results validate the proposed DoF completion technology could achieve the derived MSE lower bound and the effectiveness of the MSE-based ISAC waveform design.

I. INTRODUCTION
B ENEFITING from the improved spectral-, energy-and hardware efficiency, and the ability to deploy sensing functionality into the current communication networks, the research interest for integrated sensing and communication (ISAC) has arisen in the design of the sixth-generation (6G) systems [1]. Depending on the geographical configurations, current ISAC systems are categorized into three classical configurations [2], i.e., 1) the monostatic deployment that transmits communication signals and then captures the target echoes via the co-located receiver, such as enabling a base station (BS) as a sensor; 2) the bistatic deployment that collects the reflected and scattered echoes from a separated receiver, such as various Wi-Fi sensing applications; 3) the distributed deployment that characterizes a target via signals collected from widely distributed receivers. In particular, the monostatic ISAC systems are appealing for practical implementation as it promises a pilot-free signaling strategy [1].
To unlock the potential integration gain promised by ISAC signaling [3], one of the major challenges is to design a fully unified waveform design via jointly considering both the sensing and the communication performances. As such, several ISAC signaling strategies have been proposed to strike a balance Y. Cui between communications and sensing [4], e.g., embedding communication data into a sensing waveform [5] and the direct use of standard-compatible communication waveforms for sensing [6]. Yet, there are limited discussions regarding the estimation error for sensing functionality in ISAC systems, which is also a fundamental performance metric [7].
In this letter, we focus on the downlink monostatic ISAC system where an ISAC BS serves multiple users and simultaneously sensing the surrounding environment. In particular, we analyze the parameter estimation performance of the re-used communication waveform, by formulating a transmit waveform design problem that minimizes the MSE for the target sensing. To this end, a MSE lower bound is derived that paves the way for establishing a bound-achieving waveform augmentation strategy, namely degrees-of-freedom (DoF) completion, that facilitates the transmit waveform to achieve the MSE lower bound. The simulation results validate the performance of proposed DoF completion strategy and the effectiveness of the MSEbased ISAC waveform design. We highlight our contributions as following: • A MSE lower bound is derived for the monostatic downlink ISAC system, which indicates that re-using conventional communication waveforms without further processing could not achieve the lower bound even if the signalinterference-plus-noise-ratio (SINR) asymptotically high. • A waveform augmentation strategy, i.e., DoF completion, is systematically introduced. Comparing to other pioneer work [8], [9], we confirm that the proposed strategy can achieve the derived MSE lower bound, and then, apply it to a newly proposed MSE-based ISAC waveform design problem. Finally, we obtain its globally optimal solution via semidefinite relaxation (SDR) by constructively showing that the adopted SDR is tight. Notations: Matrices are denoted by bold uppercase letters, vectors are represented by bold lowercase letters; Tr(·) and vec(·) denote the trace and the vectorization operations, (·) T , (·) H , rank(·), and (·) −1 stand for the transpose, the Hermitian transpose, the rank, and the inverse of the input matrices, respectively. ⊗, and E(·) denotes the Kronecker product and the expectation operation, respectively. I represents the identity matrix.

1) Communication Model:
We consider an integrated sensing and communication BS equipped with N T transmitting antennas and N R receiving antennas, as shown in Fig. 1. The BS transmits data to K single-antenna communication users and simultaneously extracting environmental information from reflected echoes such that K ≤ N T . The channel matrix from the BS to K = {1, · · · , K} users is represented by denoting the channel vector from the BS to user k ∈ K.
We further assume that the communication channel experiences block fading, in which the data symbol s k,l ∈ C transmitted to user k over time slot l ∈ L = {1, · · · , L} is precoded by a spatial precoding vector p k ∈ C K×1 , where L is the channel coherence time interval with K ≤ N T ≤ L [4], [9]. Taking the inter-user interference into account, the received signal at the k-th user is given by where n k,l ∼ CN (0, δ 2 C ) is the additive complex Gaussian white noise at user k. Without loss of generality, we assume that the channel knowledge is perfectly known at the BS. Then, the achievable rate at user k can be written as C k (P) = log 2 (1 + γ k (P)), where P ∈ C N T ×K = [p 1 , · · · , p K ] denotes the overall precoding matrix at the BS transmitter and γ k (P) is the received SINR at user k. By stacking the data symbols transmitted at slot l into a vector s l = [s 1,l , · · · , s K,l ] T , we assume that the transmitted data symbols are independent with unitary power, i.e., E(s l s H l ) = 1, such that the instantaneous per-user SINR can be given by 2) Sensing Model: The BS re-uses the transmitted waveform for sensing the surrounding environment. The corresponding sensing waveform X = PS is an N T × L dimensional matrix that jointly governed by the precoder and transmitted data symbol matrices, where S = [s 1 , · · · , s L ] is a K × L dimensional data matrix with orthogonal data streams 1 L SS H = I. Thus, the sample covariance matrix of X can be written as Then, the reflected sensing echo Y S ∈ C N R ×L received at the BS can be written in a matrix form as where G ∈ C N R ×N T is the target response matrix and N S ∈ C N R ×L is the noise matrix. For sensing, we aim for estimating all target responses, i.e., elements of matrix G. For the ease of presentation, we adopt the following operator, vec(AB) = (B T ⊗ I)vec(A), to vectorize (5) as where g = vec(G), n S = vec(N S ), andX = X T ⊗ I denotes the N T N R dimensional vectorized target response matrix, noise vector, and the equivalent sensing waveform matrix, respectively. Typically, the target information is unknown to the sensing system. Hence, we assume that g is a complex Gaussian random vector g ∼ CN (0, R g ), and is independent of the noise vector n S ∼ CN (0, δ 2 S I). 3) Target Model: Here, we consider an extended target model, where a sensory target (e.g. a human being or a vehicle) is located near to the BS, such that the sensing waveforms are reflected by various target components (scatters). Therefore, in a rich scattering environment, R g is typically of full rank. Recalling that the transmitting and the receiving antennas employed for sensing are colocated at the BS, thus, R g can be modeled as [10] (7) where the scalar α m is the radar cross section (RCS) of scatter m, a(θ m ) and b(θ m ) are the corresponding transmit and receive steering vectors to echo's direction of arrival θ m , respectively. Since we are only interested in the MSE-based waveform design at the transmitter side, the receive steering vectors are assumed to orthogonal with each other. Therefore, by introducing the transmit covariance matrix R g T = m α 2 m a(θ m )a H (θ m ) and the receive covariance matrix R g R = I NR , we can simplify R g as

4) MSE-Based Transmit Waveform Design:
From the sensing perspective, we consider an estimation-oriented waveform design that minimizes the MSE for target sensing. Since the received signal y S and the target responses g are jointly Gaussian distributed, the MMSE estimator is indeed a linear MMSE (LMMSE) estimator. Recall the fact that the sensing waveform X is known to the BS. In particular, the MMSE estimator is expressed as Therefore, the MSE matrix E is To bridge the connection between the sensing waveform design and communication precoder, we further observe that X HX = (XX H ) ⊗ I NR = R P ⊗ I NR . By taking (8) into account, the to-be-designed precoding matrix P appears in the MMSE sensing waveform design criteria. Exploiting the identity Tr(A⊗I NR ) = N R Tr(A), the MMSE transmit waveform design problem is given by Therefore, when the transmitted communication signal is reused for sensing, the waveform design problem that minimizes the target estimation error under the maximum power budget can be formulated as Obviously, the optimization problem (12) is convex w.r.t R P . Its optimal point can be found by the well-known Karush-Kuhn-Tucker (KKT) conditions [11]. Yet, in the next section, our analysis on the optimal solutions of (10) will unveil the parameter estimation performance of the downlink waveform in the monostatic ISAC system.

III. A MSE LOWER BOUND
In this section, we analyze the parameter estimation performance by examining the above MSE problem. Here, we start with a MSE lower bound inspired from (12).
Theorem 1: Let R P be the covariance matrix of the BS precoder and R g T be the target response matrix, which are both N T × N T positive-definite full-rank Hermitian matrices. Let σ i,P and σ i,g T be the i-th large eigenvalue of R P and R g T , respectively. Then, the MSE is lower-bounded by The equality holds if and only if the eigenvector matrix U P = U g T , where U P and U g T consist of eigenvectors corresponding to descending ordered eigenvalues of R P and R g T , respectively.
Proof. It can be proved by following the similar procedure as in [12] and is omitted here due to the page limitation.
The MSE matrix evaluates the parameter estimation performance of sensing functionality. Typically, if R g T is known, Theorem 1 can be employed to find the optimal eigenvalues {σ * i,P } N T i=1 that minimize the MSE. In (12), with the maximum power budget P T at BS, the optimal eigenvalues is the waterfilling solution given by Consequently, the optimal precoder can be designed as P = U P diag{σ * 1 2 i,P , · · · , σ * 1 2 N T ,P }. If the to-be-designed P is unconstrained, it is obvious that the optimal solution is determined by R g T 's eigenvalues and the corresponding eigenvectors. However, in a practical downlink monostatic ISAC sensing system, the transmitted waveform matrix is often rank-deficient.
Typically, in the communication BS, the precoder P delivers K data streams, stacking as S, to its corresponding users via N T transmitting antennas. If we re-use the communication waveform X for sensing, the rank of the transmitted sensing waveform is given by rank(R X ) (4) = rank(R P ) ≤ min{K, N T }.
By examining (13) and (15), we further observe that (13) still holds when K = N T . However, when K < N T , there are N T −K eigenvalues {σ i,P } N T i=K+1 of R P that are zero, inspiring the following corollary.
Corollary 1: Let R P be the rank-deficient covariance matrix of the BS precoder, where rank(R P ) = K and K < N T . Then, the MSE is given by The equality holds if and only if the eigenvector matrix U P = U g T , where U P and U g T consist of eigenvectors corresponding to descending ordered eigenvalues of R P and R g T , respectively.
Proof. The proof is given as Corollary 1 indicates that in the downlink monostatic ISAC system, if the transmitted communication waveform is re-used for sensing, the MSE matrix is governed by the rank of the transmitted waveform, which is indeed, controlled by the number of served communication users. Moreover, the estimation performance attains its MSE bound when the user number K is equal to the antenna numbers N T . On the other hand, when the transmitted signal power is unlimited, which implies that all σ i,P of R P are sufficiently large, (13) and (16) can then be characterized as and lim inf respectively, it can be seen that the E(R P ) is lower bounded by 0 when R X is full rank, however, E(R P ) will always be positive because the eigenvalues of the target echoes N T i=K+1 σ i,g T are always positive. In other words, the sensing MSE of the downlink monostatic ISAC system could not always achieve the lower bound 0 for K < N T , if the transmitted communication waveform is re-used without any further processing.

IV. DOF COMPLETION : A SCHEME TO ACHIEVE MSE
LOWER BOUND In this section, we introduce a waveform augmentation strategy to improve the sensing performance of the downlink monostatic ISAC system, namely DoF completion. The main idea is to complete the transmitted signal from a rank-deficient matrix to a full rank one, by embedding an additional signal structure into the transmitted communication signals. Therefore, the resultant waveform dedicated for sensing contains not only the re-used communication signal, but also an augmented waveform matrix to extend the spatial DoFs. To this end, we express the augmented waveformX aŝ where P A and S A are the N T ×(N T −K) and (N T −K)×L additional sensing precoding matrix and data matrix, respectively. We assume that the additional data matrix is a unitary matrix such that S A S H A = I N T −K . Therefore, the covariance matrix of X can be written as where R PA ∈ C N T ×N T is the covariance matrix dedicated to the additional precoder P A and RP denotes the covariance matrix of the augmented precoder matrix. Hence, when the augmented signal waveform is employed, the resulting MSE matrix that measures the sensing performance can then be written as It is also worth noting that the to-be-designed matrix is now RP instead of the previsou R P for DoF completion. Recalling Corollary 1, the MSE matrix E(RP) is also lower bounded by the eigenvalues of RP and R −1 g T as Consequently, the proposed strategy is able to achieve the MSE lower bound of 0 when the transmitted signal power is unlimited, i.e., lim inf Remark: In general, the idea behind DoF completion strategy is to generate a transmit waveform that is full-rank. In the radar context, a full-rank waveform matrix is able to guarantee the feasibility of unbiased estimation [8], as well as to realize omnidirectional beampattern without additional mechanical rotation systems or phased-array beam scanning. In the above analysis, we proved that the full-rank waveform could achieve the MSE lower bound of 0 for a sufficient large transmit power, which implies that the parameter estimation errors from target echoes can be minimized. However, the above strategy still brings several drawbacks. On the one hand, the emission of the additional signal structure will always take additional costs of the transmission power. Intuitively, these additional power is dedicated for sensing but may not necessarily be beneficial for the communication functionality. On the other hand, while the additional signal structure from the DoF completion strategy can be removed by the successive interference cancellation (SIC) technology, this requires high computation capability at the communication users' side. Therefore, we introduce the following MMSE waveform design approach to deal with the dual-functional waveform design problem.

V. MMSE WAVEFORM DESIGNS VIA DOF COMPLETION
In this section, we apply the DoF completion strategy into a monostatic downlink ISAC waveform design problem. Our objective function is to minimize the MSE of the sensing performance, under the per-user SINR constraint for communication functionality. Note that this is a typical waveform design problem in the monostatic downlink ISAC scenario. Therefore, the waveform design can be formulated as Tr{RP} ≤ P T , RP 0.
where the received SINR of each communication user is required to be larger than the threshold γ 0 given by (25) and (26) limits the overall transmit power at the BS. When (25) is removed or γ 0 = 0, the waveform design concerns only the target estimation performance that is equivalent to (12). Due to the SINR constraint (25) for communication functionality, the overall problem is non-convex. Fortunately, (25) can be reformulated by employing SDR technology, ∀k ∈ K, and R P = k∈K Q k . We then recast constraint (25) as Then, the optimization problem can be reformulated as By dropping the rank-one constraint rank(Q k ) = 1, i.e., the relaxed problem is a convex one that can be solved by the well-known CVX toolbox. However, in this case, the resulting optimal solution is a relaxed solution of (29), which is typically not guaranteed to yield a rank-one optimal point. By employing Theorem 4 in [8], we could construct a tight rank-one solution.
Denoting the relaxed optimal solution as {Q k } N T k=1 with convariance matrixRP , the rank-one beamformers p k in P can be constructed by and P A can always be obtained from Note (30) and (31) yield an optimal solution of (24). Therefore, the above MMSE waveform design problem is solved optimally.

VI. SIMULATIONS
In this section, we provide numerical results to verify the advantages of our proposed DoF completion strategy, as well as to show the performance of the MSE-based sensing waveform design. Without loss of generality, the ISAC BS is equipped with N R = 10 receive antennas, under the maximun power budget of P T = 40 dBm. The transmission frame length L = 30, the carrier frequency is set as 2.4 GHz, the noise powers are  [13]. The communication channel experiences Rayleigh fading, where each entry of the channel matrix follows the standard complex Gaussian distribution. The path loss exponent is 2.2 according to the 3GPP path loss model. Besides, K users are randomly and uniformly distributed in a circle centered at (40 m, 0 m) with a radius of 10 m. Moreover, we assume that the entries of the target response matrix G follows Swerling 2 model with Gaussian distributed complex amplitude [8], [14].
In Fig. 2, we compare the estimation performance of the sensing functionality in terms of MSE versus with the growth of K with and without our proposed DoF completion strategy. The number of antennas are set to be N T = 16 and N T = 21, respectively. Without DoF completion strategy, the increase of K leads to a lower MSE. This observation is due to the fact that the rank of the transmit waveform grows with the number of users that facilitates the design to achieve the MSE lower bound in (16). On the contrary, for the schemes without DoF completion, the resulting MSE performance is not affected by the number of users, which is indeed the MSE lower bound. More interestingly, when K = N T = 16, the MSE performance without DoF completion also achieves the lower bound. This observation validates the correctness of our MSE analysis in Section IV, where the performance gap between the MSE lower bound and performance of optimal MSE solution vanishes at K = N T .
In Fig. 3, we evaluate the performance of SINR constrained MMSE waveform design for the downlink monostatic ISAC system (29). The Cramer-Rao Bound (CRB) based waveform design in [8] acts as a baseline. We observe that without the DoF completion strategy, the MMSE waveform design can achieve the CRB at K = N T = 16. More interestingly, by employing the propsoed DoF completion strategy, the performance of MMSE waveform design is no longer affected by the number of served communication users. The comparison between the MMSE waveform designs with and without DoF completion strategy shows the performance gain of the proposed strategy.

VII. CONCLUSION
In this letter, the estimation performance of downlink monostatic ISAC system was investigated. We derived a parameter estimation lower bound that facilitated the derivation of the downlink monostatic ISAC waveform, and then proposed a waveform augmentation strategy to improve the parameter estimation performance. The proposed strategy was adopted in the MMSE ISAC waveform design constrained by the per-user SINR of communication functionality. The MSE lower bound and the effectiveness of the proposed waveform design were revealed by the simulations.


Optimizing Artificial Neural Network-Based Models to Predict Rice Blast Epidemics in Korea

To predict rice blast, many machine learning methods have been proposed. As the quality and quantity of input data are essential for machine learning techniques, this study develops three artificial neural network (ANN)-based rice blast prediction models by combining two ANN models, the feed-forward neural network (FFNN) and long short-term memory (LSTM), with diverse input datasets, and compares their performance. The Blast_Weather_FFNN model had the highest recall score (66.3%) for rice blast prediction. This model requires two types of input data: blast occurrence data for the last 3 years and weather data (daily maximum temperature, relative humidity, and precipitation) between January and July of the prediction year. This study showed that the performance of an ANN-based disease prediction model was improved by applying suitable machine learning techniques together with the optimization of hyperparameter tuning involving input data. Moreover, we highlight the importance of the systematic collection of long-term disease data.

Rice blast disease, caused by Pyricularia oryzae Cavara, is one of the major constraints to rice production, causing significant yield losses worldwide (Katsantonis et al., 2017;Wang et al., 2015). Many studies have been conducted to understand the primary elements of the rice blast epidemic and predict the occurrence of this disease (Chung et al., 2020;Jeon, 2019;Kim et al., 2015). It is well known that low temperature, high humidity, and excessive use of nitrogen fertilizers promote the occurrence of rice blast disease. An integrated disease management (IDM) program based on a proper understanding of rice blast epidemiology is unequivocally the most effective and efficient method for managing rice blast disease in the long term.
IDM is designed to minimize the impact of plant diseases below the level that could cause significant economic damage by deploying all available methods that are optimal and realistic in the context of the rice-growing environments and population dynamics of pathogens. To properly apply various management techniques, it is essential to develop methods that can predict the risk of temporal and spatial occurrences using a plant disease prediction model. In other words, knowing when and to what extent plant diseases occur is crucial for effective plant disease management. This helps determine the optimal timing and order of applying proper disease management methods. In particular, owing to the acceleration of climate change and increasing occurrence of abnormal climate conditions, it is becoming more difficult to predict plant diseases. Therefore, various innovative methods have been proposed to cope with this situation (Juroszek and von Tiedemann, 2011).
Computer modeling has been developed and employed to predict plant disease epidemics using weather, environmental, and agronomic data. Traditional plant disease prediction models find statistical, empirical, and/or mechanical relationships between these data and the occurrence of plant diseases, and simulate the key infection process based on them. These types of prediction models with ei-ther observed or predicted input data provide information on when and to what extent plant diseases would occur to determine optimal management measures. In particular, when the overall production cost increases due to unnecessary control activities or the control effect decreases owing to ill-timed control, an IDM based on accurate prediction is urgently required. These prediction models are utilized as an important component of IDM along with various plant disease management methods.
Recently, as the amount of agricultural data has exponentially increased, many attempts have been made to model plant diseases using machine learning (Kim and Lee, 2020). Different algorithms, such as support vector machines (SVMs), artificial neural networks (ANNs), and random forest, have been used to create plant disease prediction models based on meteorological variables, including maximum and minimum temperatures, humidity, rainfall, and wind speed. Fenu and Malloci (2019) trained two different models using the ANN and SVM techniques to predict late blight in potatoes. They used 4-year meteorological data (hourly temperature, humidity, rainfall, wind speed, and solar radiation) as input, and classified corresponding disease occurrence data in Southern Sardinia into three risk levels. The SVM model showed better performance for low-and high-risk levels, whereas the ANN model outperformed at the medium-risk level. Interestingly, the ANN model incorrectly classified 40 out of 49 high-risk cases as mediumrisk cases, although its overall accuracy was 96%. This was due to the imbalance of the training dataset, where most data were classified as low risk; thus, the overall accuracy was determined by the classification performance for the major class without profoundly considering the minor class. Bhatia et al. (2020) adopted the Extreme Learning Machine algorithm and found proper resampling techniques can solve the problem of highly imbalanced dataset in plant disease occurrence.
Similarly, researchers have attempted to predict rice blast disease using machine learning. Kaundal et al. (2006) developed and compared rice blast prediction models using multiple regression, SVMs, and two ANN algorithms. Malicdem and Fernandez (2015) developed rice blast prediction models using the feed-forward neural network (FFNN) model, the simplest ANN structure that unidirectionally connects the input and output layers without a loop. Recent studies have used the long-short term memory (LSTM) structure to predict rice blast disease occurrence (Kim et al., 2018;Nettleton et al., 2019). LSTM was developed by Hochreiter and Schmidhuber (1996), and is considered to be a high-performance model among recurrent neural networks (RNNs). This is because LSTM solved the prob-lem of long-term dependency in RNNs by using a variable called the cell state to selectively store information through the input, forget, and output gates (Hochreiter and Schmidhuber, 1996). Using LSTM, Kim et al. (2018) developed region-specific models for the prediction of rice blast in Korea. Three-year rice blast occurrence and weather data (average temperature, relative humidity, and sunshine duration) were used as input data to train the model for rice blast prediction over the next few years. In another study, two machine learning-based rice blast prediction models (M5Rules and LSTM) showed comparable performance to process-based models (Yoshino and WARM) (Nettleton et al., 2019).
Developing a plant disease prediction model using machine learning has many hurdles such as the low quality and quantity of disease occurrence data for training and validation, relatively low performance of the models developed in previous studies, and lack of information on optimal machine learning techniques for plant disease prediction. Although the LSTM sits in the center of interest for plant disease prediction these days, it is well known that LSTM generally performs well for sequences of up to 250-500 timesteps to solve long-term dependency problems (Chemali et al., 2017), indicating it may not outperform the FFNN for rice blast epidemics with a sequence of much shorter timesteps. Therefore, the objectives of this study are to develop ANN-based rice blast prediction models using the limited quality and quantity of rice blast occurrence data available in Korea, and to compare the performance of the FFNN and LSTM models after optimizing hyperparameters of both models. In previous studies, historical rice blast occurrence and weather variables were used as inputs for model training without considering the performance variation depending on hyperparameters (Kim et al., 2018;Nettleton et al., 2019). In machine learning, hyperparameters determine the structure of the learning model, such as the learning rate, number of nodes and layers, and batch size; thus, optimizing them is crucial for model performance (Probst et al., 2019). In this study, the number of observed years for rice blast occurrence, range of observed months, number of timesteps for weather variables, and type and combination of weather variables were included as hyperparameters to examine the variance in model performance.
In the study, historical rice blast occurrence data and weather observation data were used as input data to train rice blast prediction models (Fig. 1). Historical rice blast occurrence data were obtained from the National Crop Pest Management System (NCPMS; https://ncpms.rda.go.kr) of the Rural Development Administration (RDA) of Korea.
For weather observation data, we obtained the daily maximum air temperature (°C), minimum air temperature (°C), precipitation (mm), relative humidity (%), and wind speed (m/s) data from 89 weather stations of the Korea Meteorological Administration from 2002 to 2020 (Fig. 1). To avoid biased learning toward specific data due to the difference in scale between input data (Sola and Sevilla, 1997), weather data were min-max normalized when creating the training datasets. To match the weather observation data with the rice blast occurrence data, the nearest weather stations were selected using the haversine formula (Eq. 1). The haversine formula is used to determine the distance between two points using their coordinates (longitude and latitude), assuming the Earth as a sphere (Yang et al., 2019), as follows: , where d is the distance between sites 1 and 2, r is the radius of the earth, ϕ 1 is the latitude of site 1, ϕ 2 is the latitude of site 2, λ 1 is the longitude of site 1, and λ 2 is the longitude of site 2. Among 2,473 data points, seven outliers with distances of more than 36 km between the rice monitoring plot and weather station were eliminated from the sample data.
The models were developed in the following order. First, we constructed a Blast_FFNN model that uses only historical rice blast occurrence data as input for training and conducted hyperparameter tuning for model optimization. Subsequently, we introduced an additional parallel layer using weather input data into the optimized Blast_FFNN model. Consequently, two new models, a Blast_Weather_ FFNN model in which weather data go through FFNN layers and a Blast_Weather_LSTM model in which weather data go through LSTM layers, were created and then went through sequential hyperparameter tuning processes. The hyperparameters considered in the optimization process for each model are listed in Table 1.
In the Blast_Weather_FFNN and Blast_Weather_LSTM models, hyperparameter tuning first determines the number of nodes (units) and activation functions of the parallel layers. Second, the input data of months, period, and weather_ variables, referring to the selected months to be included, the number of timesteps during the selected months, and combinations of weather variables, respectively, are determined based on the hyperparameter tuning. For example, in the case of period '4' with the weather_variables of 'tmax' and 'prec' for the months of 'June-July,' the 15-day (61 days for June-July divided by period 4) average values of daily maximum temperature and daily precipitation were used as input data. Selecting target months as input data allowed us to examine whether weather conditions until July (approximately up to 50-70 days after transplanting in most rice cultivation areas in South Korea) have a significant effect on model performance. Additionally, selecting an appropriate period was important for model performance.
The period of few sections may dilute the characteristics of the weather conditions affecting rice blast occurrence during the period, whereas the period of too many sections may result in outlier conditions that misrepresent the favorable weather conditions for rice blast. The development processes for the three models are illustrated in Fig. 2.
Since the number of class 0 (no occurrence) was approximately three times more than class 1 (occurrence), the training and test sets were split according to the same ratio using stratified k-fold cross-validation with k = 10. Additionally, as the rice blast occurrence data used in the study were significantly imbalanced, we increased the number of class 1 samples using the random oversampling method (Batista et al., 2004), which randomly replicates the minority class dataset to a size comparable to that of the major class dataset. Focal loss was used as the loss function (Lin et al., 2017), and Adam optimizer was used as the optimizer setting, with a learning rate of 10 -3 (Kingma and Ba, 2014). An appropriate number of epochs (100) was determined in the preliminary test. The models were developed using TensorFlow version 2.6.0, an open-source machinelearning library developed by Google (Abadi et al., 2016).
Both accuracy and recall were used as measures to evaluate the performance of the proposed models and optimize the hyperparameters. The prediction results of the classifier were expressed in a confusion matrix, and four classes were defined: true-positive (TP) for correctly predicted class 1 data, false-positive (FP) for incorrectly predicted class 1 data, true-negative (TN) for correctly predicted class 0 data, and false-negative (FN) for incorrectly predicted class 0 data. Accuracy indicates how often the classifier is The number of timesteps during the selected months.
Optimizing ANN-Based Rice Blast Models 399 correct and is calculated as the ratio of (TP + TN)/(TP + FP + FN + TN). Accuracy has some constraints as a performance indicator because model training is biased toward the major class when an ANN model is trained using an imbalanced dataset. In this case, even if the minor class is not predicted well, the overall accuracy can be high, since the major class is generally predicted well. Therefore, we used the recall indicator to evaluate model performance.
Recall, also called sensitivity or the TP rate, indicates how often the classifier predicts the actual disease occurrence, calculated as the ratio of TP/(TP + FN). When it comes to plant disease management, it is necessary to predict the actual disease occurrence to inform farmers to implement appropriate disease control measures to reduce potential yield loss. Therefore, we selected hyperparameters with the maximum recall values, as shown in Table 2. Validation of the performance of each model was repeated 10 times, then the average value was obtained. Experiments for selecting hyperparameters for each of the three models (i.e., Blast_FFNN, Blast_Weather_FFNN, and Blast_Weather_LSTM) verified that the performance of the models depends on the hyperparameters (Table 2). Recall increased as the year_size of the Blast_FFNN model increased from 1 to 3, indicating that a record of rice blast occurrence in recent years helps predict future occurrences. This is because the amount of initial inoculum of a year results from the overwintered inocula from the epidemics of the previous year(s). Moreover, local specific conditions, such as cultivars, climate, and soil, might influence the inherent disease proneness; thus, rice blast is more likely to occur where it normally occurs. Kim et al. (2018) used data from the past 3 years to examine the feasibility of predicting the occurrence of rice blast. Larger year_size over 3 years in our study reduced the number of training samples due to the presence of missing values. Additionally, unnecessarily old data beyond three years in the past had little impact on prediction and became disruptive to learning. After tuning the remaining hyperparameters with 16 nodes for the hidden layers using the tanh and sigmoid activation functions, the Blast_FFNN model showed the maximum performance with a recall of 55.99%.
Using weather data in addition to rice blast occurrence data as input, the Blast_Weather_FFNN and Blast_ Weather_LSTM models showed higher performance with 66.33% and 64.50% recall scores, respectively, compared to the Blast_FFNN model. As shown in previous studies (Fenu and Malloci, 2021;Kim et al., 2018), weather data is necessary to improve the prediction performance of ANNbased rice blast models. We found that the months and periods, based on which the weather data are applied as input, are important determining factors of model performance. The Blast_Weather_FFNN model showed the highest performance with the months between January and July and a period of 20 (approximately 10-day averages), while the Blast_Weather_LSTM model was best parameterized with the months between March and July and a period of 24 (approximately 6-day averages). Both models performed better when the weather data before planting were included as input, probably because it is related to the survival rate of the overwintered inocula from previous epidemics and thus determines the amount of initial inoculum of the prediction year. Among the five weather variables used in this study, both models showed the highest performance when using daily maximum temperature, precipitation, and relative humidity. The optimal numbers of nodes were 8 and 16 for Blast_Weather_FFNN and Blast_Weather_LSTM, respectively. In addition, the rectified linear unit (ReLU) activation function was selected for both models.
As a result, Blast_Weather_FFNN had higher performance (a recall score of 66.33%) than Blast_Weather_ LSTM (a recall score of 64.50%). Since LSTM has a complex structure and more parameters compared to other ANN models, the relatively small quantity of NCPMS data for training likely affected the LSTM model to be underfitted. In addition, as there was no clear time-series pattern appearing in weather data for less than a year, there might not be an added value of using LSTM. Thus, we concluded that it is more appropriate to use FFNN with a limited amount of data and a weak time-series pattern. Furthermore, considering that LSTM requires more computing resources and a longer training time owing to its complex structure and process, other ANN models might be a better starting point to consider.
In this study, long-term NCPMS data were used to develop an ANN-based rice blast prediction model. Government-led data collection in more than 80 locations across the country for two decades has resulted in quality data that are eligible for various machine learning-based studies. Considering that Fenu and Malloci (2021) used only 2-5 years of disease occurrence data, the new models developed in the current study using 19 years of data may show more robust performance in predicting interannual disease variation. Another promising fact is that the amount of data continues to increase with time, as data collection continues even at the moment. The quantity and quality of data are important in data-driven modeling research, such as machine learning-based studies. However, a chronic shortage of disease survey data in most countries has led to very few studies assessing the amount of data required for the reliable prediction of rice blast occurrence using machinelearning approaches. Examining this data requirement aspect requires a significant amount of data that exceeds what we used in the study. One way of securing sufficient data for such analyses is to generate artificial disease occurrence data using process-based disease epidemiological models considering various environmental, agronomic, and host plant and pathogen factors as input.
Another hurdle in developing ANN-based disease prediction models is imbalanced datasets, which could result in biased model training toward the major class. In particular, for plant disease survey data collected from designated monitoring plots, observations of disease symptoms are very rare. To solve this problem, we used random oversampling and focal loss to increase the number of class 1 samples compared to class 0 samples for model training (Liu et al., 2007). This is because we emphasize reducing false-negative errors over false-positive errors to avoid severe yield losses resulting from no action over actual disease epidemics.
In Korea, unmanned aerial vehicles are commonly utilized in most rice paddies for collaborative disease control (Kim and Jung, 2020). Disease early warnings that use seasonal climate forecasts (SCFs) with a lead time of a few months support collaborative disease controls requiring at least a month before the decision-making of scheduling and preparing the control activities. The Blast_Weather_ FFNN model, which showed the best performance in this study, requires weather data from January to July of the prediction year. Considering that rice leaf blast normally occurs between June and August, the model should rely on forecasted weather information from the SCFs. If the model can generate a rice blast alert sometime in May using the SCFs for June to August, the alert information can be applied for planning collaborative disease controls in South Korea. Therefore, follow-up studies should verify the performance of the rice blast prediction model using the SCFs. Unlike observational data, the reliability of the model prediction depends significantly on the predictability of the SCFs. This problem can be overcome by utilizing machine learning techniques, where SCFs are used as input variables to train the prediction model. Thereby, the inherent uncertainty of the SCFs is considered in hyperparameter tuning while training the model.

Conflicts of Interest
No potential conflict of interest relevant to this article was reported.