extfieldsofstudy:
Computer Science
Medicine

text:
Pleasantness Recognition Induced by Different Odor Concentrations Using Olfactory Electroencephalogram Signals

Olfactory-induced emotion plays an important role in communication, decision-making, multimedia, and disorder treatment. Using electroencephalogram (EEG) technology, this paper focuses on (1) exploring the possibility of recognizing pleasantness induced by different concentrations of odors, (2) finding the EEG rhythm wave that is most suitable for the recognition of different odor concentrations, (3) analyzing recognition accuracies with concentration changes, and (4) selecting a suitable classifier for this classification task. To explore these issues, first, emotions induced by five different concentrations of rose or rotten odors are divided into five kinds of pleasantness by averaging subjective evaluation scores. Then, the power spectral density features of EEG signals and support vector machine (SVM) are used for classification tasks. Classification results on the EEG signals collected from 13 participants show that for pleasantness recognition induced by pleasant or disgusting odor concentrations, considerable average classification accuracies of 93.5% or 92.2% are obtained, respectively. The results indicate that (1) using EEG technology, pleasantness recognition induced by different odor concentrations is possible; (2) gamma frequency band outperformed other EEG rhythm-based frequency bands in terms of classification accuracy, and as the maximum frequency of the EEG spectrum increases, the pleasantness classification accuracy gradually increases; (3) for both rose and rotten odors, the highest concentration obtains the best classification accuracy, followed by the lowest concentration.

Introduction
Emotion is an ever-changing psycho-physiological phenomenon that represents the body's adaptation to environments [1]. As a high-level function of brain activity, emotions affect people's communication, learning, memory, decision-making, and other mental aspects [2][3][4]. The human exploration of one's own emotions has never stopped, and emotion has always been the focus of psychology and physiology [5]. Nowadays, with the rapid development of neuroscience, psychology, brain science, computer science, and artificial intelligence, people have deepened their understanding and application of emotions.
At present, for the induction of emotions, researchers generally use visual, auditory, or a combination of visual and auditory stimuli [6][7][8]. However, as one of the most important and original senses, olfaction is naturally emotional, mainly because the olfactory cortex in the brain is directly connected to the limbic system, and the limbic system plays a key role in human emotion and memory [9][10][11]. In fact, olfaction can provide a unique channel for humans to understand their own perceptual world and social-emotional experience, and olfactory stimulation can trigger pleasant or unpleasant emotions, affecting people's behaviors and decision making. The research of emotion recognition based on olfactory stimulation has important practical significance. For example, in multimedia systems,

Experimental Procedures
Two kinds of odors in the T&T olfactometer (from Daiichi Yakuhin Sangyo Co., Ltd., Tokyo, Japan) were used; one smells like roses, termed rose odor, and the other is a foul, unpleasant odor, termed rotten odor, and each includes five different concentrations. The five concentrations of each odor type from low to high are described as A10 −5.0 , A10 −4.5 , A10 −4.0 , A10 −3.5 , and A10 −3.0 and C10 −6.0 , C10 −5.5 , C10 −5.0 , C10 −4.5 , and C10 −4.0 , respectively, where the values 10 −3.0 , 10 −3.5 , . . . , 10 −6.0 indicate the dilution coefficients of the concentration. For different odor types, the central concentration, e.g., A10 −4.0 or C10 −5.0 , which corresponds to the detection threshold of normal olfactory sensation, may be different. If someone can smell the central concentration, it means that their sense of smell is normal; otherwise, it is abnormal. Because of the different central concentrations of different odor types, the magnitudes of the exponents (e.g., C10 −6.0 and A10 −5.0 ) of rotten and rose odors are different.
During the experiment, the smells were randomly selected and placed 1-2 cm below the participant's nose through test papers. Subsequently, a Cerebus system (Blackrock Microsystems, Salt Lake City, UT, USA, Figure 1) was used to collect EEG signals with a 1 kHz sampling rate and a 32-channel EEG cap (according to the international 10-20 system layout). Among the 32 electrodes, 2 electrodes were used as references, and the remaining 30 electrodes were used for data analysis.
tion. For different odor types, the central concentration, e.g., A10 −4.0 or C10 responds to the detection threshold of normal olfactory sensation, may b someone can smell the central concentration, it means that their sense of sm otherwise, it is abnormal. Because of the different central concentrations of types, the magnitudes of the exponents (e.g., C10 −6.0 and A10 −5.0 ) of rotten a are different.
During the experiment, the smells were randomly selected and placed the participant's nose through test papers. Subsequently, a Cerebus syste Microsystems, USA, Figure 1) was used to collect EEG signals with a 1 kHz and a 32-channel EEG cap (according to the international 10-20 system la the 32 electrodes, 2 electrodes were used as references, and the remaining were used for data analysis.  There were 13 participants, aged 24.9 ± 2.25 years, who were informed the experimental protocol. All the participants in this study are with no smell, no smell-related diseases, right handedness, and no smoking and dr For each odor concentration for each participant, 35 stimulations were per tain 35 trials. In each trial, the collection duration of the EEG signal was 1 were a 4-s interval for two adjacent trials and a 2-min interval for two odor c Using a 2-min interval, the last odor concentration could be excluded fro mental room with an exhaust fan, and the pleasantness induced by each was subjectively evaluated. Within each participant, the olfactory EEG data to a fixed concentration of a certain odor were assigned to one category, w gory containing 35 EEG samples. Thus, for each participant, we obtained odors ×5 concentrations) categories and 350 (10 × 35) EEG samples. In total, EEG samples were collected.

Feature Extraction
The high-frequency noise of EEG signals was removed using a 0.5-70 filter, and then the power spectral density (PSD) features were calculated five typical EEG bands, namely, delta (δ): 0.5-3 Hz, theta (θ): 4-7 Hz, alpha beta (β): 14-30 Hz, and gamma (γ): 31-70 Hz. Figure 2 shows the PSD feature extraction process. First, for each elect There were 13 participants, aged 24.9 ± 2.25 years, who were informed and agreed to the experimental protocol. All the participants in this study are with normal sense of smell, no smell-related diseases, right handedness, and no smoking and drinking habits. For each odor concentration for each participant, 35 stimulations were performed to obtain 35 trials. In each trial, the collection duration of the EEG signal was 10 s, and there were a 4-s interval for two adjacent trials and a 2-min interval for two odor concentrations. Using a 2-min interval, the last odor concentration could be excluded from the experimental room with an exhaust fan, and the pleasantness induced by each concentration was subjectively evaluated. Within each participant, the olfactory EEG data corresponded to a fixed concentration of a certain odor were assigned to one category, with each category containing 35 EEG samples. Thus, for each participant, we obtained 10 (2 kinds of odors ×5 concentrations) categories and 350 (10 × 35) EEG samples. In total, 4550 (350 × 13) EEG samples were collected.

Feature Extraction
The high-frequency noise of EEG signals was removed using a 0.5-70 Hz bandpass filter, and then the power spectral density (PSD) features were calculated in each of the five typical EEG bands, namely, delta (δ): 0.5-3 Hz, theta (θ): 4-7 Hz, alpha (α): 8-13 Hz, beta (β): 14-30 Hz, and gamma (γ): 31-70 Hz. Figure 2 shows the PSD feature extraction process. First, for each electrode, a Welch periodogram with 100 sample Hamming windows was used to estimate the PSD amplitude. Then, the PSD curves were divided into five segments based on the EEG rhythm.

Subjective Evaluation Results
In the experiment, each participant performed a subjective evaluation of th level induced by each odor concentration. For each concentration of each odor jective score ranged from -10 to 10. Positive scores represent pleasant emotio greater values represent more pleasant odors; the negative scores represent dis tions, where greater absolute values indicated more disgusting odors. The subje uation scores are shown in Table 1.
The results show that different concentrations of odors can induce differen of pleasure or disgust. To obtain emotional categories, the average subjective e scores of 13 participants were calculated. As a result, five different levels of plea tions can be induced by the five different concentrations of rose odor, while five levels of disgust emotions can be induced by the five different concentrations odor.
To validate the claim of varying levels of emotions with concentrations, Figure 2. PSD feature extraction of a single participant based on EEG rhythm, which corresponds to a particular concentration of the rose odor, i.e., the A10 −5.0 . Each color corresponds to an EEG electrode. EEG, electroencephalogram; PSD, power spectral density.

Subjective Evaluation Results
In the experiment, each participant performed a subjective evaluation of the pleasure level induced by each odor concentration. For each concentration of each odor, the subjective score ranged from −10 to 10. Positive scores represent pleasant emotions, where greater values represent more pleasant odors; the negative scores represent disgust emotions, where greater absolute values indicated more disgusting odors. The subjective evaluation scores are shown in Table 1.
The results show that different concentrations of odors can induce different degrees of pleasure or disgust. To obtain emotional categories, the average subjective evaluation scores of 13 participants were calculated. As a result, five different levels of pleasant emotions can be induced by the five different concentrations of rose odor, while five different levels of disgust emotions can be induced by the five different concentrations of rotten odor.
To validate the claim of varying levels of emotions with concentrations, statistical tests, specifically, the Friedman significance test, for the subjective evaluation scores of five concentrations of each odor type have been performed, and the significance test results (i.e., p-values) for the rose and rotten odors are 4.7 × 10 −3 and 6.2 × 10 −8 , respectively. Both of these are less than the significance level of 0.05, proving that for the subjective evaluation scores of 13 participants used in this study, the differences between the five concentrations of each odor type can reflect the overall differences between the odor concentrations. In other words, the significance test results prove that the levels of emotion vary with concentrations.

Pleasantness Recognition Induced by Rose Odor Concentrations
In this part, we classified five kinds of pleasant emotions with different levels of pleasure. From Table 1, the average jury scores vary with the five different concentrations of the rose odor, indicating that the degrees of pleasantness induced by the five different concentrations are different. In the training process for pleasantness recognition induced by the rose odor concentrations, five classes correspond to the five concentrations. To verify and compare several classic recognition methods, all the recognition algorithms were trained and tested 10 times by processing the randomly divided training samples (20 samples in each concentration class for each participant) and testing samples (15 samples in each concentration class for each participant).
Quantitatively, the corresponding average classification accuracies across the concentrations of all participants using five classifiers are illustrated in Figure 3, which shows that the SVM with gamma frequency band yields the best classification accuracy, obviously higher than other classification methods. It is also better than the classification accuracy obtained using the full frequency bands (FFB), i.e., using delta, theta, alpha, beta, and gamma frequency bands together. These results suggest that the EEG gamma band is closely related to the olfactory stimulus. Furthermore, Figure 3 shows that as the frequency increases, the accuracy of most classifiers increases, indicating that the olfactory stimulus is highly correlated with the high-frequency band of EEG signals. Specifically, the gamma frequency band yields the best classification accuracy. The accuracy is higher than other frequency bands and outperforms the FFB, indicating the excellent classification ability of the gamma band for different degrees of rose odor-induced pleasantness.
To explore the influence of concentration on recognition accuracy, the average classification accuracies across 13 participants for five different concentrations of the rose odor using the SVM classifier, are shown in Figure 4. For all frequency bands, the highest concentration, i.e., the concentration of A10 −3.0 , obtains the best classification accuracy, followed by the lowest concentration, i.e., the concentration of A10 −5.0 . These results suggest that the highest and lowest concentrations are relatively easy to recognize. The reason may be that the EEG signals induced by the highest and lowest concentrations are different from that induced by other concentrations.

Pleasantness Recognition Induced by Rotten Odor Concentrations
Furthermore, we conducted pleasantness classification induced by the disgu odor, and we also compared the presented classification methods. In the training pr for pleasantness recognition induced by the five concentrations of the rotten odor

Pleasantness Recognition Induced by Rotten Odor Concentrations
Furthermore, we conducted pleasantness classification induced by the disgusting odor, and we also compared the presented classification methods. In the training process for pleasantness recognition induced by the five concentrations of the rotten odor, five classes correspond to the five concentrations of the rotten odor. Each concentration class contains 35 EEG samples, where 20 samples were randomly selected as the training samples, and the other 15 samples were used to test the training algorithms. The random division of the samples was performed 10 times, and the following results are the average results of the corresponding 10 tests of classifiers.
The comparison results of the five classifiers and five frequency bands are shown in Figure 5. For the classification of disgust emotions induced by the rotten odor, the SVM still produces the best classification accuracy, and the classification accuracy of most classifiers generally increases as the frequency increases. Additionally, similar to the conclusion drawn from Figure 3, Figure 5 shows that for the recognition of rotten odor-induced emotions, the accuracy of the gamma frequency band on each participant is higher than the other frequency bands and outperforms the FFB, indicating the excellent classification ability of the gamma band for different degrees of rotten odor-induced disgust emotions. The comparison results of the five classifiers and five frequency bands are Figure 5. For the classification of disgust emotions induced by the rotten odor, still produces the best classification accuracy, and the classification accuracy of m sifiers generally increases as the frequency increases. Additionally, similar to th sion drawn from Figure 3, Figure 5 shows that for the recognition of rotten odor emotions, the accuracy of the gamma frequency band on each participant is hig the other frequency bands and outperforms the FFB, indicating the excellent clas ability of the gamma band for different degrees of rotten odor-induced disgust e Using the SVM classifier, average classification accuracies across 13 partici five different concentrations of the rotten odor are shown in Figure 6. Similar to the results in Figure 6 suggest that the highest and lowest concentrations are r easy to recognize. Similar to the rose odor, the EEG signals induced by the hig lowest concentrations of the rotten odor are relatively different from those ind other concentrations. Using the SVM classifier, average classification accuracies across 13 participants for five different concentrations of the rotten odor are shown in Figure 6. Similar to Figure 4, the results in Figure 6 suggest that the highest and lowest concentrations are relatively easy to recognize. Similar to the rose odor, the EEG signals induced by the highest and lowest concentrations of the rotten odor are relatively different from those induced by other concentrations.

Emotion Classifications of Pleasure and Disgust
According to the subjective evaluation rules, positive scores represent pleasant emotions, and negative scores represent disgust emotions. We divide emotions into two types: pleasure and disgust. As shown in Table 1, the rose odor corresponds to pleasant emotions, and the rotten odor corresponds to disgust emotions. In other words, five concentrations of the rose odor correspond to pleasant emotion, while five concentrations of the rotten odor correspond to disgust emotion. Subsequently, the corresponding olfactory EEG signals are also classified into two categories for recognition. In the training and testing process for recognition of pleasure and disgust emotions, two classes correspond to two odor types, and each odor type has five concentrations. For each odor type, there are 175 = 35 × 5 EEG samples. The classification results are shown in Figure 7.
Comparing Figure 7 with Figure 3 or Figure 5, it can be seen that the classification accuracy is significantly improved in the two types of emotional classifications of pleasure and disgust, indicating that the emotional classifications of pleasure and disgust are easier to recognize than the different degrees of pleasant or disgust emotions.
In addition, we used the polygon area metric [35] to evaluate the performance of the classifier for emotion classifications of pleasure and disgust. According to [35], we can obtain the calculation formula of the polygon area metric (PAM) as follows: Considering that Gamma band outperforms other EEG rhythms in the olfactory-induced emotion recognition. Here, we take Gamma band as an example to test PAM performance of each recognition method, and the results are shown in the Table 2, which again illustrates that recognizing the pleasure and disgust emotions is ease and feasible.

Emotion Classifications of Pleasure and Disgust
According to the subjective evaluation rules, positive scores represent pleasant emotions, and negative scores represent disgust emotions. We divide emotions into two types: pleasure and disgust. As shown in Table 1, the rose odor corresponds to pleasant emotions, and the rotten odor corresponds to disgust emotions. In other words, five concentrations of the rose odor correspond to pleasant emotion, while five concentrations of the rotten odor correspond to disgust emotion. Subsequently, the corresponding olfactory EEG signals are also classified into two categories for recognition. In the training and testing process for recognition of pleasure and disgust emotions, two classes correspond to two odor types, and each odor type has five concentrations. For each odor type, there are 175 = 35 × 5 EEG samples. The classification results are shown in Figure 7.

Analysis of Outperformance of the Gamma Band and SVM Algorithm
The reason for the outperformance of the gamma band and not the FFB may be that   Figure 5, it can be seen that the classification accuracy is significantly improved in the two types of emotional classifications of pleasure and disgust, indicating that the emotional classifications of pleasure and disgust are easier to recognize than the different degrees of pleasant or disgust emotions.
In addition, we used the polygon area metric [35] to evaluate the performance of the classifier for emotion classifications of pleasure and disgust. According to [35], we can obtain the calculation formula of the polygon area metric (PAM) as follows: Considering that Gamma band outperforms other EEG rhythms in the olfactory-induced emotion recognition. Here, we take Gamma band as an example to test PAM performance of each recognition method, and the results are shown in the Table 2, which again illustrates that recognizing the pleasure and disgust emotions is ease and feasible.

Analysis of Outperformance of the Gamma Band and SVM Algorithm
The reason for the outperformance of the gamma band and not the FFB may be that there are redundant features in the FFB. The features extracted from FFB contain the features of delta, theta, alpha, and beta, where the classification ability of these features is weaker than that of the gamma frequency band. More weak classification features may bring redundant features, thereby reducing the classification accuracy.
The reason why SVM outperforms four other classifiers, i.e., NB, k-NN, ELM, and BPNN, may be that for the non-massive data, such as the data used in this study, the classification ability of SVM is strong. This is also proved by the experimental results from other researchers [23][24][25][26].
Notably, the analysis of gamma-band in (visual, auditory, etc.) EEG usually has limitations and it is better suited to MEG studies. However, for olfactory EEG, from the perspective of signal recognition, we found that the features of gamma band are better than those of other EEG rhythms, which has also been proved by other scholars [28]. Moreover, we will further analyze the classification value of gamma band, such as extracting and testing different types of features from gamma band, and testing the value of gamma band with more datasets.

Method Design and Novelty
For the recognition methods, the current research only adopted and verified the feasibility of several classical methods. However, this study does not focus on the method innovations, but the innovation of research content, that is, recognizing emotions induced by different odor concentration through EEG analysis. As far as we know, there are few reports about inducing people's emotions through different odor concentrations and then conducting such emotional recognition through EEG analysis.
Although the experiments in the literature (e.g., [23,24,28]) are similar to those in this study, they are essentially different. For example, most of the experiments in the literature use different odor types to stimulate the participants, rather than stimulating and inducing their emotions with different concentrations of odors.
Considering that the focus of this study is to test the feasibility of recognizing emotions induced by different odor concentrations through EEG, at this stage, we have not compared the adopted methods with those in other papers. In subsequent research, we will design novel recognition algorithms with better performance for this specific recognition tasks, and compare it with other methods in the literature.
Moreover, by considering and comparing our results with previously reported studies in the literature, the novelty of our study is expressed as follows: The detailed contribution points and the main purpose of this study include: (1) to explore the classification possibility of odor concentration-induced emotions using EEG signals, (2) to find the best EEG rhythm wave for this recognition task, (3) to analyze the relationship between recognition accuracy and odor concentrations, and (4) to select the most suitable one among five commonly used classifiers.

Electrodes Corresponding to the Maximum PSD Amplitudes
Taking the gamma frequency band as an example, to explore the maximum of the PSD amplitudes among all electrodes, which is helpful for the determination of the brain regions related to the sense of smell, we tested each odor concentration and obtained the average PSD amplitudes of 35 EEG samples corresponding to each concentration for each participant. The study found that for different concentrations of the same odor type, the electrode corresponding to the maximum of the PSD amplitudes was the same. However, there are differences in the electrodes corresponding to the maximum amplitude of the PSD of the two odor types. Table 3 shows that for most participants, the PSD amplitude of the FP1 or FZ electrodes located in the frontal lobe region of the brain is the maximum. In addition, Kroupi et al. [28] and Alarcao et al. [36] showed that olfactory stimulation is related to frontal and temporal lobes. This may be the reason why the PSD amplitude of the FP1 and FZ electrodes is relatively large under olfactory stimulation.

Limitations
Size of participants: Compared to datasets with a small number of participants, large datasets are more powerful to support the research. The sample size (13 participants) used in our work is small, but we think that the EEG signals collected from 13 participants can also be used to carry out exploratory research of pleasantness recognition induced by different concentrations of odors. Therefore, in this study, the dataset involving 13 participants was presented, which has been openly available on the website [37]. To further analyze the olfactory EEG signals, we are continuing to expand the participants. Next, we will carry out pleasantness recognition induced by odor concentrations on a larger dataset, and look forward to obtaining better results.
Gender non-homogeneity: For the gender ratio of participants, this study involved 13 participants, including 12 men and 1 woman, which is gender non-homogeneity. For this issue, we will increase the number of female participants in the following research, and maintain gender balance while expanding the scale of participants.

Accuracy Analysis with Respect to Sample Size
The training process in our study contained five steps. (1) Randomly select training samples; (2) Extract features from the selected samples; (3) Feed features into the classifier to determine classifier parameters; (4) Use the determined classifier to predict the categories of new EEG samples; (5) Repeat the presented steps 10 times to obtain the average prediction accuracy of 10 times. The results in Figures 3 and 5 were obtained using 20 training samples and 15 testing samples for each odor concentration. To check how the accuracy changes with sample size, we have adjusted the sample size in training and testing processes, and the average classification accuracy of 13 participants is shown in Table 4. The results are obtained using SVM to classify PSD features extracted from gamma band. Table 4. Results of accuracy change with sample size in training and testing processes. Take the average classification accuracy of 13 participants as an example. The results are obtained using the SVM to classify PSD features extracted from the gamma band. The percentages represent the average classification accuracies across the concentrations.  Table 4 shows that as the number of training samples increases and the number of testing samples decreases, both the concentration classification accuracies of rose and rotten odors are improved.

Prediction of Concentrations/Emotions by Analyzing EEG Signals
When given the EEG signals, the concentrations/emotions can be effectively predicted. The accuracies shown in Figures 4 and 6 were calculated by comparing the actual concentration/emotion types with the corresponding prediction results, and the prediction results were obtained by processing EEG signals. Figure 4 shows that using the gamma band to predict the five concentrations/emotions of the rose odor, the lowest accuracy value is 91.0%, corresponding to the concentration of A10 −4.0 ; Figure 6 shows that using the gamma band to predict the five concentrations/emotions of the rotten odor, the lowest accuracy value is 86.8%, corresponding to the concentration of C10 −5.5 , thus proving that the concentrations/emotions can be effectively predicted.

Odor Selection
At present, this study is only a preliminary exploration of emotion recognition induced by different odor concentrations. At this stage, we have tried to analyze the emotions induced by only pleasant and disgusting odors. Through the preliminary exploration, the feasibility of recognizing olfactory-induced emotions through EEG analysis has been verified.
For recognition of emotions induced by different odor concentrations using EEG signals, the T&T olfactometer which contains five odor types, each with different concentrations, provides a very suitable odor material. After the psychological evaluation, we eliminated the intolerable sweat odor and feces odor, and finally screened out two odor types, one being the pleasant rose odor, the other the disgusting rotten odor, each odor with five different concentrations. The next research will add the analysis of neutral odor-induced emotions according to your valuable comments.

Conclusions
This study uses EEG technology to recognize pleasantness induced by different concentrations of odors. Using the support vector machine with power spectral density features of the EEG gamma frequency band, the average classification accuracies of 93.55% and 92.24% were achieved for the pleasantness recognition induced by different concentrations of pleasant rose and disgusting rotten odors, respectively. When classifying the pleasant and disgusting emotions, a considerably high accuracy of 99.9% was obtained. The experimental results reveal that (1) it is possible to classify pleasantness emotions induced by odor concentrations; (2) in the EEG rhythm, the gamma band is most related to odor concentration stimuli; (3) compared to the naive Bayesian, k-nearest neighbor, extreme learning machine, and backpropagation neural networks, the support vector machine demonstrates the feasibility for recognizing the odor-induced pleasantness.


VIDI: A Video Dataset of Incidents

Automatic detection of natural disasters and incidents has become more important as a tool for fast response. There have been many studies to detect incidents using still images and text. However, the number of approaches that exploit temporal information is rather limited. One of the main reasons for this is that a diverse video dataset with various incident types does not exist. To address this need, in this paper we present a video dataset, Video Dataset of Incidents, VIDI, that contains 4,534 video clips corresponding to 43 incident categories. Each incident class has around 100 videos with a duration of ten seconds on average. To increase diversity, the videos have been searched in several languages. To assess the performance of the recent state-of-the-art approaches, Vision Transformer and TimeSformer, as well as to explore the contribution of video-based information for incident classification, we performed benchmark experiments on the VIDI and Incidents Dataset. We have shown that the recent methods improve the incident classification accuracy. We have found that employing video data is very beneficial for the task. By using the video data, the top-1 accuracy is increased to 76.56% from 67.37%, which was obtained using a single frame. VIDI will be made publicly available. Additional materials can be found at the following link: https://github.com/vididataset/VIDI.

Introduction
Our world faces different natural disasters, such as, fires, earthquakes, flooding. It is crucial to take swift action when these types of disasters happen. The human vision system can understand and recognize the type of disaster with a certain confidence. However, it would be hard to use human power in systems that require fast processing. The evaluation by experts is also costly. With the improvement of deep learning, applying computer vision techniques to these challenges has become more popular, less costly, and more effective.
There have been several studies related to incidents [14,19,31], and most of them use information based on text and images. To the best of our knowledge, there is no diverse and large video dataset with various incident types. However, temporal information could be useful for the classification of the incidents. To enable the researchers to explore the benefits of utilizing temporal information, this paper introduces a new video dataset for incidents named Video Dataset of Incidents (VIDI). The video clips have been collected from YouTube 1 . Samples from the VIDI can be seen in Fig. 1. All video clips have been gathered and labeled by our team. In VIDI, when querying YouTube, we used different languages and locations to increase the variety. Some video clips are labeled with more than one label, i.e. multiple class labels are allowed in the dataset. To prevent the unrelated information in the videos that could suppress the actions, the clips were preferred to be taken Figure 1. Samples from VIDI which consists of 4,534 video clips of incidents from a single view. Moreover, the clips with sudden frame changes were discarded. Overall, the dataset contains 4,534 video clips labeled with 43 incidents. Each class has around 100 videos. The average video duration is ten seconds. The included incident categories in VIDI are the same as the ones listed in the Incidents Dataset [31]. It additionally contains temporal information and is collected from a different source, that is, while the Incidents Dataset was collected from Flickr and Twitter, VIDI was collected from YouTube. Considering these aspects, we believe that VIDI complements the Incidents Dataset.
We run benchmark experiments on the presented dataset using the recent state-of-the-art classification approaches, Vision Transformer (ViT) [18] and TimeSformer [4]. Moreover, we also employed these approaches on the Incidents Dataset. We have shown that these models outperform the ResNet-18 [16] based approach presented in [31] on the Incidents Dataset. In addition, the obtained results on the VIDI have indicated that the temporal information contributes to the incident classification performance. Using video data, around 9% absolute increase in classification accuracy has been achieved.
The remainder of the paper is organized as follows. We review and discuss the related work in Section 2. The Video Dataset of Incidents (VIDI) is introduced in Section 3. Employed classification methods and implementation details are explained in Section 4. The experimental setup and results are presented and discussed in Section 5. Finally, Sec-tion 6 concludes the paper and also provides future research directions.

Related work
In this part, we will review the studies under three different categories: incident detection datasets, methods used to classify incidents and transformer-based deep learning methods.

Incident detection datasets
Publicly available incident datasets consist of mostly images or text. They cover a limited number of classes or have a limited number of examples. Only, recently released Incidents Dataset [31] provides 446,684 images with 43 disaster categories and 49 place categories. This dataset consists of images, therefore, it is not possible to build video-based incident classification approaches using it. Another incident dataset with images is xBD [14], which contains satellite images of six different types of disasters -volcanic eruption, hurricane, earthquake, flood, tsunami, fire-from 19 different disaster sites. Even though xBD has 22,068 images, it has a limited number of disaster categories. The last image-based incident dataset that we have examined is ME17-DIRSM [26]. This dataset consists of 6,600 images for flooding incidents as positive and negative. That is, this dataset is collected only for a specific incident type and does not contain other types of disasters.
There is a large-scale annotated video dataset named FIVR-200K [19], which focuses on fine-grained video search. In this study, the authors crawled all events from Wikipedia 2 within the time interval between 2013 and 2017 and filtered the events with two main labels: "armed conflict & attack" and "disaster & accident". The video set was created with the YouTube videos, which were queried with the filtered events. FIVR dataset has 225,960 clips, which are labeled with event headlines and names of the videos.
A summary of some existing incident datasets is listed in Tab. 1. For a comprehensive list of incident datasets please refer to [31]. Similar to the Incidents Dataset [31], we aim to provide a diverse dataset from various types of incidents. Therefore, we utilize the same incident categories that were covered in [31]. As VIDI consists of video clips of the incidents, researchers can benefit from it to develop and assess their incident detection and/or classification systems that incorporate temporal information.

Methods used to classify incidents
Most of the incident classification systems employ deep learning-based approaches to existing social media [1,2,7,10,21,24,25,27,28] and satellite datasets [13,22,26,29]. In one of the studies to detect fires, a dataset was created with publicly shared 114,098 images gathered from Flickr [10]. They used multiple feature detectors --SIFT [23], SURF [3], Color-SIFT [20], Color-SURF [12] --to extract the keypoints of the images and the descriptors at the keypoints. By using the k-means algorithm [15], the descriptors are clustered to create the visual vocabulary. Afterwards, a support vector machine (SVM) [8] classifier is trained and the images are classified with this model.
In another study [24], the authors employed three image classification approaches to assess the severity of the incident damage. The used dataset was collected from social media and it contains 210,000 images from five sources. In the first method, they extracted the features by using Pyramidal Histogram of Visual Words (PHOW) and Bagof-Visual-Words (BoVW) [9] methods, and trained a linear SVM. The second method uses a pre-trained VGG-16 model [30] as a feature extractor. As the final approach, they fine-tuned a pre-trained VGG-16 model to classify the damage severities into three classes: None, mild, severe. They reached the best results on all datasets with the finetuned VGG-16 model. Nogueira et al. proposed a convolutional neural network (CNN) based approach to detect flooding from satellite images [26]. They used ME17-DIRSM dataset which has a total of 6,600 images for flooding incidents as positive and negative. They used two types of models for training. The first one uses dilated convolutions, which do not change the resolution in the learning process. The second one uses deconvolution networks, the encoder part of the network learns to extract the visual features, whereas the decoder part learns to upsample these features. By concatenating the outputs of these models, they trained an SVM classifier.
Finally, in [31], they applied a ResNet-18 [16] model as a backbone and train the model to jointly recognize the incident type and place categories by using the class-negative loss.

VIDI
In this section, we will take a closer look at the created dataset. We will describe how we have collected the videos and checked their quality, as well as we will share the dataset statistics.

Overview
The Video Dataset of Incidents (VIDI) 3 consists of videos containing natural disasters such as earthquakes, floods, landslides, vehicle accidents such as car, truck, and motorcycle crashes, and the results of these events whether they are damaged, burned, and/or collapsed. The same 43 incident categories as listed in [31] are included. In [31], the images were also labeled with 49 different places for diversity. Although we did not label the places, we aimed to collect the videos from different locations to provide a similar diversity in our dataset.
Another approach to increase diversity is searching videos in various languages to get the styles of different cultures and include the region-specific incident events. For example, the appearance of buildings may differ from each other depending on culture, location, and material availability. Wood is commonly used as a building material in America, whereas limestone is more common in the Middle East. Additionally, the precautions and consequences of different climatic events are vastly different in countries where they are expected and those in which they are not.
Mainly, six different common languages have been used in video queries. These languages are English, Turkish, French, Spanish, Simplified Chinese, and Standard Arabic. Apart from these languages, when the amount of collected videos is not sufficient, to collect more videos, Hindi, German, and several other languages have also been used in the dataset. The language statistics of the collected video clips are shown in Tab. 2. However, it may not be possible for these languages to be equally distributed for each class due to the occurrences of specific disasters in some regions. For example, a derecho is an event that exists mostly in North America. Thus, the derecho video clips were mainly collected in English. To overcome this point, we found the list of countries where this disaster occurs and searched in different languages according to these regions. Another issue that we encountered during the data collection was the translation of the incident keywords between languages. For some of the languages, we had to use neural machine translators. In some cases, the translation of the corresponding incident was not meaning the same in the respective country. The tropical cyclone could be given as an example. When the label "tropical cyclone" is translated to the Arabic language, it is seen that translated word has the same meaning as a tornado. These problems were solved by communicating with native speakers when possible.
Besides the translation issues, we had difficulty in finding videos for some labels, such as nuclear explosion. The existing videos for nuclear explosions were recorded between 1945 to 1963 in the US and USSR. Since atmospheric testing was prohibited after that year, there were a limited number of unique videos with this label. Therefore, the gathered videos mostly contain near duplicates in the positive examples: the same scene has been included multiple times in the dataset from different channels.
Finally, we allowed multiple labels in our dataset. Some videos show multiple incident types, for example, a burned house is also labeled with the damaged label and a car crash is labeled with both car accident and blocked.

Annotation and quality assurance
All videos have been obtained manually from YouTube by our team members. While querying, the incident labels and locations were combined. For example, we combined the word "fire" with different places for the "on fire" label and queried on YouTube for words similar to "fire in the building", "fire in the forest", etc. Another example is the car accident label. We have searched for car accidents that occurred at different places, such as fields, forests, seasides, and streets. For the suitable clips, video ids, starting and ending seconds of the related action, labels, and the language of the video were exported. If possible, all stages of the actions as pre-action/action/post-action were included. Another considered point was that the videos were taken only from a continuous camera view. There are no scene changes in the clips, and almost all of the videos have a one-point perspective. However, we allowed both unsteady and steady shots in the clips.
The video clips which have higher than 720p resolution were downsampled to 720p and downloaded. On the other hand, the videos that have lower resolution were downloaded with their native resolutions. After double-checking the labeled clips, the most common issues we have found were labeling the clip with the wrong label or labeling the clip with a single label, when it contains multiple incidents, such as car accidents under heavy fog. Some labels can be seen as similar to each other, therefore, YouTube users may upload videos with wrong titles, for example, confusing the derecho and tornado events. Furthermore, the users may up-load videos that do not include any action associated with the label within the clip duration. By way of illustration, a video with windy environment may have been shot in a tropical area, but the clip does not contain any information related to the region. In that case, we changed the label from "tropical cyclone" to "derecho". We revised the entire collected data for these kinds of situations.

Statistics
VIDI contains 4,534 unique videos in total. Since some of the videos have more than one label, there are 4,767 labels in the dataset. Each incident class has around 100 video clips. The number of videos per class is shown in Fig. 2.
We kept the video duration between 2 to 60 seconds. The average video duration is ten seconds. The short clips were chosen from news channels' broadcasts as they provide various disasters, however, they generally change the clips in less than two seconds. Therefore, we had to collect the longer clips mostly from documentaries and videos captured by mobile phone cameras.
We left roughly ten clips from each class for testing and validation. Thus, the dataset has 405 and 408 videos in the test and validation sets, respectively. Taking into account the multiple labels, the test set and the validation set contain 434 and 438 labels, respectively. The rest of the set was used for training.

Method
In this section, we will explain the methods and provide implementation details.

Architectures
As the Vision Transformer and TimeSformer are recent state-of-the-art methods for image and video classification, respectively, we decided to use these two architectures in our benchmarks. The ViT model was designed for images, whereas the TimeSformer model was designed for videos. Both architectures were pre-trained on the ImageNet dataset and fine-tuned on the Incidents Dataset and VIDI. Vision Transformer architecture does not use any convolutional network layers and is based on the transformer encoder blocks. TimeSformer does not contain convolutional network layers, either, and it consists of a self-attention mechanism. Similar to Vision Transformer, TimeSformer also adapts transformer architecture for computer vision tasks and video processing.

Implementation details
We used the PyTorch 4 framework for the implementation. We trained both datasets using two different architectures and performed five different experiments. For the multi-frame fine-tuning process of TimeSformer, eight frames are sampled sequentially from each video in VIDI. For the single-frame fine-tuning process of both architectures, the frame in the middle of the video was selected and given to the model as an input. To fine-tune the TimeSformer model on the Incidents Dataset, each image was assumed as a video that includes only one frame.
As the pre-trained model, B 16 imagenet1k 5 which was trained on ImageNet-21k [11] and fine-tuned on ImageNet-1k datasets were used for Vision Transformer. The divided space-time model that was trained on the K400 6 [6] and K600 7 [5] datasets were used for TimeSformer.
The training parameters used in the experiments are listed in Tab. 3. Based on the hyperparameter optimization results on the validation set, Adam optimizer [17] was used for fine-tuning the TimeSformer model on the Incidents Dataset, and the stochastic gradient descent optimization method was used in the rest of the experiments. The crossentropy loss was preferred on Incidents Dataset, whereas binary-cross entropy with logits loss was chosen for the VIDI. Different learning rate schedulers were utilized for training. Specifically, Lambda LR, ReduceLROnPlateau, 5 https://bit.ly/3CMzSkb 6 https://bit.ly/3I9TgZi 7 https://bit.ly/36iq1Xi and multi-step LR methods were used in the training of ViT and TimeSformer models. Lambda LR method updates the learning rate according to the current epoch number. In our training, the learning rate is reduced in every three epochs. In multi-step LR method, the learning rate is decreased in every five epochs. Finally, in the ReduceLROnPlateau method, the learning rate is reduced if loss change is less than a certain threshold during two epochs.

Experiments
We run experiments on both Incidents Dataset and VIDI. Incidents Dataset was already split into training, validation, and test sets. While there are 312,742 samples in the training set, there are 17,357 samples in the validation set and 17,255 samples in the test set. VIDI contains 3721 videos for training, 408 videos for validation, and 405 videos for testing. Please note that both datasets have the same incident categories and can include multiple labels per image/video, that is, an image or video might belong to more than one incident category.
Two architectures were trained using only positive incident samples for both datasets to perform incident classification. In Timesformer, the prediction was done three times in the testing phase with different crops. For each prediction of a clip, we used the sum method to combine the scores Multiple experiments were conducted by using both datasets and architectures. One of our motivations while collecting VIDI was to explore the benefit of using video data instead of single frame for incident classification. To investigate this, we fine-tuned the TimeSformer architecture on VIDI both with a multi-frame and a single frame setup. Multi-frame TimeSformer achieved 76.56% accuracy, whereas single frame version obtained 67.37% accuracy. This indicates that, when available, utilizing video information helps to improve incident classification performance.
In the experiments, we also assessed the performance of the recent state-of-the-art architectures, ViT and TimeSformer, for incident classification in comparison to the proposed approach in [31], which is based on ResNet-18. As can be seen in Tab. 4, ViT and TimeSformer achieved higher accuracies than the published results in [31]. While Resnet-18 based model [31] reaches 77.3% accuracy, ViT and TimeSformer achieved 78.5% and 81.47% top-1 accuracies, respectively, on the Incidents Dataset.
Additionally, we compared the performance of ViT and TimeSformer on both datasets using a single frame as the input. TimeSformer is found to be superior to the ViT on both datasets. The results obtained on the VIDI are lower than the ones obtained on the Incidents Dataset. There could be two main reasons for this: (1) Incidents Dataset con- tains more samples for training, therefore, the systems were able to learn better models. (2) VIDI contains more difficult samples for classification. We further analyzed the wrong predictions on the VIDI test set. Some sample frames from wrong predictions can be seen in Fig. 3. As can be observed, in some cases, the inputs are visually very similar to the samples of the predicted classes. In Fig. 4, we showed the false prediction matrix. From this matrix, one can observe which classes are mostly confused by the model. The most misclassified classes are found to be "ice storm & snow-covered", "flooded & storm surge", and "landslide & rockslide rockfall".
Finally, we have calculated the prediction accuracy per class as shown in Fig. 5. The video clips labeled with "bicycle accident", "nuclear explosion", "dirty contamined",  "airplane accident", "wildfire", "traffic jam", "tornado", "dust sand storm", "dust devil", and "snow-covered" have been predicted with higher accuracies than the other classes. It was observed that the lowest accuracy belongs to the "landslide". According to the results, the classes with lower accuracies are mostly mixed within different vehicle accidents.

Conclusion
In this paper, we presented a video dataset of incidents, named VIDI. VIDI contains 43 different incident categories, for example, earthquake, wildfire, landslide, tornado, ice storm, car accident, nuclear explosion, and has 4,534 human-labeled incident video clips collected from YouTube. The dataset contains the same incident categories as presented in the Incidents Dataset [31]. As it additionally contains temporal information as well as collected from a different source, i.e. Flickr and Twitter vs. YouTube, we believe that the proposed dataset complements the Incidents Dataset well.
We run recent state-of-the-art classification models, namely Vision Transformer and TimeSformer, both on VIDI and the Incidents Dataset to provide benchmark results. We observe that the employed approaches perform better than the ResNet-18 based method proposed in [31] on the Incidents Dataset. We also show that utilizing video data improves the incident classification accuracy.
We will continue to work on the proposed dataset and expand it with class-negative samples. When class-negative samples are included, besides performing incident classification, we will also address the incident detection problem with an aim to have a robust incident detection system that is suitable to be used in real-time scenarios.
Lastly, we have only used visual information. Yet, the audio information could also be included as supportive data for the cases in which the incidents have similar environmental characteristics. Thus, the audio could be a meaningful complementary input to be used to identify the correct situation.


A Systematic Evaluation of Node Embedding Robustness

Node embedding methods map network nodes to low dimensional vectors that can be subsequently used in a variety of downstream prediction tasks. The popularity of these methods has grown significantly in recent years, yet, their robustness to perturbations of the input data is still poorly understood. In this paper, we assess the empirical robustness of node embedding models to random and adversarial poisoning attacks. Our systematic evaluation covers representative embedding methods based on Skip-Gram, matrix factorization, and deep neural networks. We compare edge addition, deletion and rewiring attacks computed using network properties as well as node labels. We also investigate the performance of popular node classification attack baselines that assume full knowledge of the node labels. We report qualitative results via embedding visualization and quantitative results in terms of downstream node classification and network reconstruction performances. We find that node classification results are impacted more than network reconstruction ones, that degree-based and label-based attacks are on average the most damaging and that label heterophily can strongly influence attack performance.

Introduction
In recent years, the design of robust machine learning models has become an important topic and attracted significant amounts of research attention [1][2][3][4]. The term 'robust' refers to the ability of a model to provide consistent and accurate predictions under small perturbations of the input data. These perturbations can appear in the form of random noise, out of distribution (OOD) data, or partially observed inputs [5]. They can affect models at train or evaluation times and be random or adversarial in nature. For a more complete overview of robustness in machine learning we refer the reader to [6]. In this manuscript, we empirically study both random and adversarial attack scenarios where perturbations are either a consequence of noise or specifically crafted to reduce model performance. We further focus our analysis on attacks affecting the models at training time exclusively, also know as the poisoning scenario [7].
Simultaneously, node representation learning or node embedding models have become increasingly popular for bridging the gap between traditional machine learning and network structured data [8][9][10]. These approaches map network nodes to real-valued vectors that can be subsequently used in downstream prediction tasks such as classification [11] and regression [12]. Training of these models can be performed in a semi-supervised or unsupervised fashion. In the former, embeddings are optimized for a particular downstream task while in the latter, general purpose embeddings are obtained. Robustness is an important feature for representation learning models as well. One would generally prefer small changes in the input networks to have a minimal impact on the vector representations learned and on downstream task performance. Moreover, with the deployment of these models in safety-critical environments (e.g., [13]) and on the web (where adversaries are common [14,15]), robustness evaluation has become ever more essential. Unfortunately, the robustness of unsupervised node embedding approaches is poorly understood. Some recent studies have analyzed particular semi-supervised models based on the Graph Neural Network (e.g., [16]) and on shallow models (e.g., [17]). Others, have evaluated specific unsupervised random walk approaches under poison attacks [7]. Methods leveraging unsupervised embedding learning paradigms such as matrix factorization and deep neural networks, have not received much attention yet. Additionally, there is a lack of studies providing broader robustness evaluations and comparing multiple models.
We perform a systematic empirical analysis of the robustness of foundational works in the field of node embeddings. Among the 9 unsupervised approaches evaluated we include Node2vec [18], GraRep [19], and SDNE [20], which have inspired many other methods based on similar principles, e.g., [21][22][23]. The models considered can be categorized into Skip-Gram, matrix factorization, and deep neural networks, and their robustness is compared on two downstream tasks: node classification and network reconstruction. We evaluate robustness under randomized and adversarial attacks targeting the network edges. For adversarial attacks we limit the scope to heuristic-based approaches where edges are targeted based on topological network properties (e.g. assortativity, degree). In contrast, optimization-based attacks (e.g. [7,[24][25][26]) solve a multi-level optimization problem to identify the most promising targets. Heuristic attacks are, thus, simpler and more computationally efficient making them more easily accessible to an attacker. Additionally, they do not require tailoring to specific embedding models and downstream tasks -as many optimization-based approaches do-and provide intuitive and explainable targets. Moreover, the heuristic attacks considered in this manuscript have already shown to effectively lead to structural collapse in networks [27]. The analysis of stronger optimization-based models is left for future work. Lastly, we focus our evaluation on global attack scenarios where changes can be made to the entire graph structure provided a fixed attack budget.
Contributions. Our main contribution is a systematic analysis of node embedding robustness. We evaluate a total of 9 unsupervised node embedding approaches based on three learning paradigms. We employ 6 small and mid-sized networks and compare 14 different poison attack strategies. Further, we investigate differences between randomized and adversarial attacks and compare edge addition, deletion and rewiring strategies. We also investigate attacks leveraging full knowledge of the node labels, commonly used as baselines, and show that network homophily (tendency of nodes with similar labels to be connected) and heterophily (where nodes of different labels are more often connected) have a strong impact on their performance. This work constitutes the first empirical evaluation of its magnitude on node embedding robustness.
The remainder is organized as follows: in Section 2 we present the related work and in Section 3 we introduce the embedding methods and attack strategies evaluated. In Section 4, we discuss the experimental evaluation and results and finally, in Section 5 we outline our main conclusions.

Related Work
A large body of research has shown that traditional machine learning models and more recently deep neural models can be easily misled into providing wrong answers with high confidence [28,29]. Work on identifying and protecting against these adversarial attacks has particularly developed in the field of computer vision [30]. Works in this field, including [31,32], have also shown how changes unperceivable to the human eye can result in dramatic performance drops or misclassifications. Later, adversarial attacks were introduced in the field of network science [5]. In [27], the authors show how structural properties of networks can collapse as a result of attacks. The authors further provide a framework for simulating attacks and defenses on networks. With the popularization of node embedding methods authors have also investigated adversarial attacks on particular semisupervised [16,33] and unsupervised [17] approaches. A survey reviewing various adversarial attacks and defense strategies, specifically designed for semi-supervised representation learning models, is presented in [34]. The authors also collect implementations of representative attack and defense strategies and make them publicly available as a PyTorch toolbox. For unsupervised representation learning models, while there are some empirical studies comparing performance (e.g., [35]), there is little research evaluating robustness. With the present work, our aim is to fill this gap and provide a fist empirical study and overview on the robustness to random and adversarial attacks of unsupervised node embedding approaches. Lastly, akin to the results in [36] for graph classification using GNN models, we also investigate patterns in the attacks and relations to the network structure. This analysis reveals an interesting effect of label heterophily on node classification attacks.

Methods
In this section we introduce the node embedding approaches evaluated and the attack strategies used to poison the input networks. Regarding notation, in what follows we will use G = (V, E) to refer to an undirected graph with vertex set V = {v 1 , . . . , v N }, N = |V| and edge set E ⊆ (V × V), M = |E|. We will represent edges or connected node-pairs as unordered pairs {v i , v j } ∈ E. And refer to pairs {v i , v j } / ∈ E as non-edges or unconnected node-pairs. Node embeddings are denoted as X = (x 1 , x 2 , . . . , x N ), X ∈ IR N×d where x i is the d-dimensional vector representation corresponding to node v i .

Node embedding methods
For our experimental evaluation we have selected 9 representative methods spanning three different embedding learning paradigms, namely Skip-Gram, matrix factorization and deep neural networks. Next, we introduce each paradigm and the corresponding methods.
Skip-Gram. These approaches capture node similarities in the graph through random walks and leverage the Skip-Gram model [37] to obtain node representations that maximize the posterior probability of observing neighboring nodes in the walks. From this category we evaluate: Deepwalk [38], the seminal work that proposed fixed length random walks to capture node similarities and Skip-Gram (approximated via hierarchical softmax) for learning the embedding matrix X; Node2vec [18], which introduced more flexible random walks controlled by in/out and return parameters and approximates Skip-Gram via negative sampling; LINE [39], where the authors leverage first and second order proximities to learn representations; And finally, VERSE [11], which minimizes the KL-divergence between a similarity metric on G (by default Personalized PageRank) and a vector similarity on X.
Matrix Factorization. Factorization methods take as input node similarities encoded in the graph Laplacian, incidence matrices, adjacency matrices (A) and their polynomials, etc. and compute low dimensional embeddings by factorizing the selected matrix. We evaluate the following methods based on this paradigm: GraRep [19], HOPE [40], NetMF [41] and M-NMF [42]. GraRep factorizes high order polynomials of A, HOPE can factorize different similarity matrices provided they can be expressed as a composition of two sparse proximity matrices. NetMF decomposes the Deep-Walk transition matrix via SVD and lastly, M-NMF computes embeddings via non-negative matrix factorization and incorporates community structure in this process.
Deep Neural Networks. Deep neural models, from auto-encoders to Siamese networks or CNNs, have also been used to obtain node representations from a graph's link structure in an unsupervised fashion. Among these types of methods we evaluate SDNE [20], a deep neural model that captures first and second order proximity in the graph. 1 .

Network attacks
We subdivide network attacks into randomized and adversarial and further into three main types based on the changes to the network structure. These changes are edge addition, edge deletion and edge rewiring. Table 1 summarizes all attacks and below we briefly describe each one.
Randomized Attacks. These attacks are designed to simulate random errors or failures in the networks. We consider edge addition (add_rand), deletion (del_rand) and rewiring (rew_rand). In the first case, pairs of nodes, v i , v j ∈ V are selected uniformly at random and added to Here d i and d j represent the degrees of nodes v i and v j , respectively. In rewire attacks we use del_rand to remove a budget of edges {v i , v j } ∈ E and then reconnect each Adversarial Attacks. We also consider a particular type of heuristic-based adversarial attacks which target specific network properties such as node degrees, assortativity, and node labels. Despite their lower effectiveness compared to optimization-based attacks, we evaluate these approaches due to their lower computational complexity, applicability to different embedding methods and downstream tasks, and explainable attack targets. Moreover, they aim to modify key structural properties commonly captured by representation models and can thus lead to worse representations. The heuristics considered have also been successfully used as baselines in previous works e.g., [7,27].
For all edge addition attacks we ensure that newly generated pairs do not already exist in the graph, i.e., {v i , v j } / ∈ E, and they do not form selfloops, i.e., v i = v j . For degree-based (add_deg) and preferential attachment (add_pa) edge addition strategies we sample nodes uniformly and based on degree, respectively, and connect them to destination nodes sampled based on degree. For the degree assortativity (add_da) and disassortativity (add_dd) attacks, we generate edges which increase/decrease this property. We define assortativity (r) akin to [43] and compute it per edge as the product of standard scores of Thus, to increase assortativity we sample nodes v i with probability p i ∝ |d i − µ| and connect them to nodes v j sampled with probability The add_ce strategy applies to attributed graphs only and adds a set of random edges connecting nodes of dissimilar labels, exclusively.
Unless otherwise specified, edge deletion attacks ensure that input networks do not become disconnected after the attack. For del_deg and del_pa we first sort all edges based on the appropriate metric, i.e., d i + d j for del_deg and d i × d j for del_pa, and later remove the top edges that do not disconnect the network. For del_da and del_dd we compute r {vi,vj } and −r {vi,vj } as described above. Then, we sort the edges based on these properties and take the top candidates in each case while avoiding disconnections. The del_di strategy applies exclusively to attributed graphs and randomly selects edges for removal where the incident nodes share the same label.
Finally, DICE [7] is an adversarial attack where edges are removed or added to a network with equal probability. Edges are removed according to the del_di strategy and added following add_ce. It is important to note that all edge deletion attacks with the exception of del_di are deterministic while the remaining addition and rewire attacks are non-deterministic (see Table 1).

Experiments
In this section we present the experimental setup, networks used and the results obtained. All our experiments were carried out on a single machine equipped with two 12 Core Intel(R) Xeon(R) Gold processors, 1TB of RAM and an RTX 3090 GPU.
To ensure reproducibility of results, we have employed and extended the capabilities of the EvalNE toolbox [44]. This Python framework allows users to assess the performance and robustness of network embedding approaches for downstream node classification, network reconstruction, link prediction and sign prediction. In the framework we have integrated a variety of random and adversarial poison attack strategies, including those introduced in Section 3.2 and Table 1. In EvalNE, complete evaluation pipelines and hyperparameters are specified through configuration files which can be used at any time to replicate results. These configuration files together with the rest of our code are available online at https://github.com/aida-ugent/EvalNE-robustness.

Preliminaries and Setup
As pointed out in Section 1, the main goal of this paper is to investigate the robustness of node embedding approaches to poison attacks. To this end we report changes in downstream node classification and network reconstruction performances for different attacks on the input graphs. Next, we summarize the main goals and evaluation pipelines for both tasks and the overall evaluation setup.
Node Classification. Given an input graph and labels for a subset of vertices, the goal in node classification is to infer the labels of the remaining vertices. To evaluate node classification robustness we proceed as follows. (1) We start by attacking an input network G with a specific strategy (from Table 1) and budget b. The budget defines the number of edges an attacker can add, delete or rewire in the network, expressed as a fraction of the total edges. For example, b = 0.1 indicates 10% of all edges in E. (2) The attacked networkĜ = (V,Ê) is then provided as input to a node embedding approach which yields a representation matrix X containing vertex representations as its rows. As shown by Mara et. al. [35], gains from optimizing the hyperparameters of these models are marginal, and thus, we resort to fixed default values. 2 We also fix the embedding dimensionality d = 128. (3) Given a number of training nodes N tr (also defined as a fraction of all nodes in V), a multi-label one-versus-rest Logistic Regression classifier with 5-fold cross validation is trained to predict node labels from node representations. (4) We repeat the previous step 3 times with different node samples and report average results. For some experiments we will report results independent of the value of N tr . In these cases we additionally average results over several values of N tr . (5). Finally, and unless otherwise specified, for the non-deterministic attacks listed in Table 1 we repeat the complete process 3 times with varying random seeds resulting in different sets of edges being removed in step 1). We report node classification performance in terms of f1_micro and f1_macro.
Network Reconstruction. In this task the aim is to investigate how well the link structure of an input network can be recovered from the node representations. To this end node representations are first learned from the input network. Then, node-pair representations are derived by applying a binary operator on the node representations. Finally, a binary classifier is trained to discriminate edges from non-edges. High quality representations are expected to result in the classifier scores of edges being higher than those of non-edges.
We evaluate robustness on this task akin to node classification. (1) We attack the input network G with a given strategy and budget b. (2) We compute node representations forĜ with different methods for which we use fixed default hyperparameters. (3) Representations of node pairs {v i , v j } are combined into node-pair representations using the Hadamard product, i.e., x i,j = x i · x j . (4) A binary Logistic Regression with 5-fold cross validation is trained using representations corresponding to edges and non-edges inĜ. (5) Classification performance is tested using representations of edges and non-edges of the original unattacked graph G. For computational efficiency, we approximate the performance using 5% of all possible node-pairs in G. (6) We again repeat the complete process 3 times for non-deterministic attacks. For this task we report AUC and average precision scores.
Experimental Setup. Our evaluation setup is structured as follows. First, in Section 4.3.1 we investigate the performance of node embedding approaches under random attacks. In this case, we use the add_rand and del_rand strategies and vary the attack budget b ∈ [0.1, 0.2, ..., 0.9]. 3 For node classification specifically, we report average results over N tr ∈ [0.1, 0.5, 0.9], 3 node shuffles for each N tr value, and 3 experiment repetitions for non-deterministic attacks. For network reconstruction we only perform the 3 experiment repetitions for non-deterministic attacks. We then also investigate the effect of the number of labeled nodes for node classification by comparing the results obtained for N tr = 0.1 to N tr = 0.5 and N tr = 0.9. Second, in Section 4.3.2 we evaluate adversarial robustness. We use a similar setup with the following exceptions: we compare all attacks from Table 1 (random attacks are used as baselines) and the budget is fixed to b = 0.2. Third, in Section 4.3.3 we compare addition, deletion and rewiring attacks. For both downstream tasks we compare add_rand, del_rand and rew_rand and for node classification we additionally compare add_ce, del_di and DICE. Other parameters are set as for the adversarial attack experiment. In this section we also investigate differences between deletion attacks that disconnect and those that do not

Data
To conduct our experiments we use a total of 6 small and mid sized networks from different domains. Specifically, for node classification we use Citeseer [45] and Cora [46], two citation networks where nodes denote publications, edges represent citations between them and node labels indicate the main research field of each paper. For network reconstruction we use PolBlogs [47], a network of political blogs connected to each other via hyperlinks, and Facebook [48], a network of individuals and their social relations on the platform. Lastly, we perform qualitative and visualization experiments on the internet industry partnership (IIP) [49] and the StudentDB [50] networks. In the former, nodes represent companies, edges represent relations such as alliance or partnership and node labels indicate the company's main business area, i.e., user content, infrastructure or commerce. The latter, StudentDB, is a k-partite network representing a snapshot of the Antwerp University relational database. Nodes represent entities such as students, courses, tracks, etc., and edges are binary relations, e.g., student-in-track, course-in-track, etc. Node labels indicate the type of each node (see Appendix A.1 for more details). In Table 2 we summarize the main statistics of the networks used.

Randomized attacks
We start in Figure 1a with node classification performance under random edge attacks and varying attack budgets. In the chart, negative budgets indicate edge deletion and positives indicate edge addition. In this case we allow edge deletions to disconnect the original networks. We report f1_micro scores for the Citeseer network (f1_macro results as well as those for the Cora network are similar and provided in Appendix A.2). From the figure we first note different general behaviors for edge deletion and addition attacks. Deletions cause a consistent performance degradation until complete network collapse at b = 0.9. Additions cause a sharper loss in performance for relatively low budget values (b ≤ 0.2) which become less severe around b = 0.4. Thus, in the low budget regime commonly analyzed in the literature (−0.2 < b < 0.2), edge addition attacks are superior to edge deletion. Outside of this range, however, edge deletions are more damaging. This observation is reasonable given the asymmetry in the attack budgets. Removing 90% of the graph edges leaves significantly less information to learn an embedding from than adding 90% of spurious edges. We also observe from Figure 1a that NetMF and M-NMF are slightly more robust to edge additions than other approaches while edge deletion performance is similar across the board.
In Figure 1b we present the AUC scores for reconstructing the original Facebook network G, from an attacked graphĜ. The plot indicates high edge recovery with AUCs 1 despite the random attacks. Most methods maintain high robustness for a wide range of budget values. Some notable exceptions are Node2vec, LINE, and SDNE which consistently lose performance the more adversarial edges are added. One possible explanation is that these methods are not only affected by the addition of spurious edges but also by the removal of potentially informative negative samples, used by all three approaches to learn embeddings. For the PolBlogs dataset presented in Appendix A.2, we observe similar patterns. An exception in both networks is HOPE, which significantly degrades performance for strong edge deletion attacks (b ≤ −0.6). This indicates the method is less suited to learning embeddings of highly sparse networks. The high robustness exhibited by the evaluated  Figure 1a presents f1_micro scores for node classification on Citeseer. Figure 1b shows AUCs for network reconstruction on Facebook. In Figures 1c and 1d we show average node classification f1_micro scores for different fractions of labeled nodes N tr on Citeseer and Cora, respectively. Shaded areas denote 95% confidence intervals.
approaches on this task is particularly interesting given the double impact of the attacks. Unlike in node classification, attacks on network reconstruction affect the models both at embedding learning time and binary classifier training (edge and non-edge train labels are obtained from the attackedĜ).
We now focus our attention on the impact of the number of train labels available for node classification (N tr ). In Figures 1c and 1d we compare the average performance over all methods and experiment repetitions for N tr ∈ [0.1, 0.5, 0.9] on Citeseer and Cora, respectively. For both networks we observe equally low performances when few labeled nodes are available i.e., N tr = 0.1. For larger values (N tr ≥ 0.5) the performances are very similar. We also observe that as networks become denser (as we move right on the x-axis in each plot) the difference between low and high values of N tr become more significant. This indicates that node embedding methods will generally not provide robust predictions when few labeled nodes are available and this situation will worsen the denser the network is.

Adversarial attacks
We now compare the effect of different heuristic-based adversarial attacks on node classification. Figures 2a and 2b summarize the results on the Citeseer network for edge deletion and addition attacks, respectively. In both cases we present decreases in f1_micro caused by different attacks with budget b = 0.2, as compared to the performance on the non-attacked graph. Firstly, if we compare across graphs we observe that edge additions decrease performance more than deletions across all methods for this particular budget value. This is also consistent with our observations from Figure 1a for random attacks on node classification. Among the edge deletion attacks we see that del_dd is, from an adversarial perspective, the most effective strategy. With this attack, we are targeting edges from high degree to low degree nodes further increasing the uncertainty regarding the latter. On the other hand, for edge addition the most effective strategies are connecting edges with different labels together (add_ce) or connecting nodes with similar degrees to each other (add_deg). It is interesting to note that attacks with full knowledge of the node labels del_di and add_ce are not significantly stronger than others e.g., degree based attacks. The colors in both figures indicate different fractions of labeled nodes. We observe that most of the variance in performance comes from the experiments with N tr = 0.1 (blue points) and that these are also mostly concentrated in the lower ends of the boxplots. The variances for N tr ≥ 0.5 are very similar across different attack strategies and networks.
In Figures 2c and 2d we present similar results for network reconstruction. In these cases we show the combined performances for both Facebook and PolBlogs datasets. The experiments reveal that edge deletion attacks are marginally stronger than edge addition. In particular, deleting edges based on degree is the most effective adversarial technique of the ones we have evaluated. Overall, we also observe much less variance in performance compared to the results on node classification.

Addition, deletion and rewiring attacks
In Figure 3 we compare edge addition, rewiring and deletion attacks on both downstream tasks. The attack budget is fixed to b = 0.2 and we show combined results for the two networks used in each task (marker color denotes the data used). We observe that for node classification rewiring attacks perform best (central boxes in the left and middle plots in Figure 3). This is also the case if we look at each individual dataset with results for Cora (orange dots) being significantly higher than those on Citeseer (blue dots). For network reconstruction we have much less data available, considering that we do not need to test different train sizes and shuffles per size. In this case the results indicate similar performances for all attack types. We further observe that results on the Facebook network are overall higher than on PolBlogs. The f1_macro and average precision scores for each task also corroborate this findings and are presented in Appendix A.3.
We further investigate how strong a role network connectivity plays in adversarial attacks. We compare random and degree attacks constrained to not disconnecting the input networks and their unconstrained counterparts. We find that constrained attacks are on average, over all methods and networks 5% less effective. Specifically, for random attacks the f1_micro performance without disconnections is 0.651±0.166 (mean and standard deviation) and with disconnections 0.612±0.161.
Similarly, for degree based attacks average performance reaches 0.637 ± 0.163 when disconnections are prevented and 0.606 ± 0.164 when they are not.

Attacks exploiting node labels
In this section we investigate adversarial attacks on node classification where the attacker has full access to the node labels. These types of attacks e.g., DICE, are commonly used as baselines under the assumption that access to the node labels inevitably leads to stronger attacks. Here, we demonstrate that the above assumption does not always hold. Specifically, we find that node label homophily/heterophily has a strong impact on the performance of these types of attacks.
In this experiment we use the IIP and StudentDB datasets. The former is an example of a homophilic network where 70.9% of edges connect nodes of the same label. On the other hand, StudentDB is a strongly heterophilic network where no edges connect nodes sharing the same label. We summarize the results for node2vec, although our findings apply to other methods capturing high order proximities in graphs. We use DICE as an attack strategy.
We start our evaluation by attacking both networks with budgets b ∈ [0.0, 0.2, 0.6]. We then learn node embeddings and perform downstream node classification for each network and attack budget. Correctly and incorrectly classified nodes at validation time are recorded for each case. Figure 4 presents a spring-layout representations of the IIP network for each attack budget where nodes are colored based on their prediction status, correct (blue) or incorrect (orange). From the figure we can visually confirm that, as the attack strength increases, the misclassification rate (mr in the figure) also increases. This is also confirmed numerically by the mr value presented above each plot.
In Figure 5 we present the same information for the StudentDB network. In this case, as the attack strength increases the misclassification rate decreases (as can be seen visually and through the mr values). This seemingly counter intuitive behavior can be explained by the fact that DICE introduces additional information in the network reinforcing the heterophily schema (similar nodes remain unconnected while dissimilar ones are more connected). This dilutes the local network structure and makes nodes of the same type more similar to each other. Methods such as node2vec able to capture high order proximities between nodes can capitalize on this additional information to provide embeddings more suitable for node classification.

Conclusions
In this paper we have demonstrated that node embedding approaches, regardless of their underlying representation mechanisms, are sensitive to random and adversarial poison attacks. We have shown that results on downstream node classification are significantly less robust compared to those on network reconstruction. Our experiments also revealed that for low attacks budgets (below 20% of edges in the graph) edge addition attacks are generally stronger than edge deletions. Outside of this range, the opposite is true. Surprisingly, our empirical evaluation showed no significant differences  between the heuristic-based adversarial attacks evaluated. Leveraging full knowledge of the node labels when attacking node classification does also not yield significantly stronger attacks. Finally, we have also shown that the number of labeled nodes plays a fundamental role in node classification robustness, that rewiring attacks are generally stronger than addition or deletion independently, and that attacks leveraging node label information can result in improved representations of heterophilic networks. With this work and our extension to robustness evaluation for the EvalNE software, we hope to lay the foundations for further research in this area.

A.1 Further dataset details
The IIP network represents a set of companies competing in the internet industry between 1998 and 2001. Nodes in the graph denote companies and edges represent business relations such as joint venture, strategic alliance or other type of partnership. The associated node labels denote the company's main business area i.e., content, infrastructure of commerce.
The StudentDB network represents a snapshot of Antwerp University's relational student database. Nodes in the network represent entities, more specifically: students, professors, tracks, programs, courses and rooms. Edges constitute binary relations between them, that is, student-in-track, studentin-program, student-takes-course, professor-teaches-course, and course-in-room. Numerical node labels are assigned according to each node's type.

A.2 Randomized attacks: additional results
In this section we present our additional experiments regarding randomized attacks on node embeddings. We start in Figures 6 and 7 by presenting the node classification f1_micro results for the Cora dataset and the network reconstruction AUC scores for PolBlogs. In Figures 8 and 9 we summarize the f1_macro scores for both Citeseer and Cora and Figures 10 and 11 present the average precision on Facebook and PolBlogs.

A.3 Other attacks: additional results
We also compare the performance of edge addition, rewiring and deletion on both downstream tasks in terms of f1_micro and average precision. These results support our conclusions in Section 4.3.3 (see Figure 12).