With the increasing popularity of posting multimodal messages online, many recent studies have been carried out utilizing both textual and visual information for multi-modal sarcasm detection. In this paper, we investigate multimodal sarcasm detection from a novel perspective by constructing a cross-modal graph for each instance to explicitly draw the ironic relations between textual and visual modalities. Specifically, we first detect the objects paired with descriptions of the image modality, enabling the learning of important visual information. Then, the descriptions of the objects are served as a bridge to determine the importance of the association between the objects of image modality and the contextual words of text modality, so as to build a cross-modal graph for each multi-modal instance. Furthermore, we devise a cross-modal graph convolutional network to make sense of the incongruity relations between modalities for multi-modal sarcasm detection. Extensive experimental results and in-depth analysis show that our model achieves state-of-the-art performance in multi-modal sarcasm detection 1 . Sarcasm is a peculiar form of sentiment expressions, allowing individuals to express contempt sentiment or intention that is converse to the authen-tic/apparent sentiment information (Gibbs, 1986; Dews and Winner, 1995; Gibbs, 2007). As such, accurately detecting satirical/ironic expression could  The first two authors contribute equally to this work. potentially improve the performance of sentiment analysis and opinion mining (Pang and Lee, 2008; Kumar Jena et al., 2020; Pan et al., 2020). In today's fast growing social media platforms, it is common to post multi-modal messages. Therefore, in addition to developing sarcasm detection models for textual data (Riloff et al., 2013; Joshi et al., 2015), it is increasingly popular to explore sarcasm detection in multi-modal data such as text and images (Schifanella et al., 2016; Cai et al., 2019). Dealing with multimodal data requires an understanding of the information presented in different modalities. As the sarcastic example shown in Figure 1 (a), text-only approaches may erroneously identify it as a positive sentiment expression due to the phrase  wonderful weather . This post however contains a sarcastic expression with negative sentiment, because it is accompanied by an image with  thunderstorm clouds . The key of effective multi-modal sarcasm detection is to accurately extract the incongruent sentiment cues from 1767 different modalities, allowing the detection of the true sentiment conveyed in the message. To perform multi-modal sarcasm detection on data composed of text and image, several related research efforts attempt to concatenate the textual and visual features to fuse sarcastic information (Schi-fanella et al., 2016), employ attention mechanism to implicitly fuse the features of different modalities based on external knowledge (Cai et al., 2019; Xu et al., 2020; Pan et al., 2020), or build interactive graphs to model the relations of different modalities (Liang et al., 2021a). Despite promising progress made by existing models, they still suffer from the following limitations: 1) Simply considering the whole image does not produce good results, mostly due to the intricate visual information presented in an image; not to mention that only particular visual patches are related to the text. As in the examples shown in Figure 1, the correct results can be easily obtained by only tracking the visual information in the bounding boxes. Therefore, discriminating key visual objects from the irrelevant ones could lead to improved learning of visual information. 2) Crucial visual information that relates to the sarcastic cues of text modality may be scattered in an image (Figure 1 (b)). As such, it is essential to focus on drawing the intricate sentiment connections between text and image modalities, allowing a good exploitation of the contradictory sentiment information between modalities for learning sarcastic clues. To this end, we propose a novel cross-modal graph convolutional networks (CMGCN) by constructing a cross-modal graph for each instance, where the important visual information and the related textual tokens are explicitly linked. This allows for the extraction of incongruous implications between two modalities in sarcasm detection. Concretely, instead of trying to produce a caption of the whole image, we first detect the objects of the image to capture the important visual regions and the corresponding attribute-object pairs via the approach proposed by Anderson et al. (2018). Then, we explore a novel solution to assign weights to the edges of the cross-modal graph by means of computing the word similarities between the object descriptors of the attribute-object pairs and textual words based on the WordNet (Miller, 1992). Further, to introduce the multi-modal sentiment relations into the cross-modal graphs, inspired by (Lou et al., 2021), we devise a modulating factor of sentiment relation for each edge by retrieving the affective weights of attribute descriptors (usually adjectives with affective information) and textual words from external affective knowledge (SenticNet (Cambria et al., 2020)). As such, the modulating factors can be adopted to refine the edge weights of word similarities, allowing the capture of sentiment incongruities of the cross-modal nodes in the graph. Further, in the light of cross-modal graphs, we deploy a GCN architecture to make sense of the incongruous relations across the modalities for multi-modal sarcasm detection. The main contributions of our work are summarized as follows:  To the best of our knowledge, we are the first to explore the use of the graph model based on auxiliary object detection for modeling the contradictory sentiments between key textual and visual information in multi-modal sarcasm detection. Using the attribute-object pairs of the image objects as the bridge, a novel approach of constructing cross-modal graphs is developed to explicitly link the two modalities by edges with the varying degree of importance. A series of experiments on a publicly available multi-modal sarcasm detection benchmark dataset show that our proposed method achieves the state-of-the-art performance. Previous work of sarcasm detection has been applied to textual utterances information (Zhang et al., 2016; Tay et al., 2018; Babanejad et al., 2020). Different from text-based sarcasm detection, multimodal sarcasm detection aims to identify the sarcastic expression among different modalities (Schi-fanella et al., 2016; Castro et al., 2019). Schifanella et al. (2016) firstly tackled the multi-modal sarcasm detection task with text and image modalities by manually designed features. Cai et al. (2019) created a new dataset and proposed a hierarchical fusion model for multi-modal sarcasm detection. Xu et al. (2020) explored decomposition and relation network to model both cross-modality contrast and semantic association in sarcasm detection. Pan et al. (2020) proposed inter-modality attention and co-attention to learn the contradiction of sarcasm. For 1768 Object detection cloudy white green green green [CLS] what a wonderful weather ! the graph-based methods, Liang et al. (2021a) deployed a heterogeneous graph structure to learn the sarcastic features from both intraand inter-modality perspectives. However, this method tried to grasp the visual information of the whole image, and meanwhile ignore the sentiment expression between different modalities. Therefore, different from (Liang et al., 2021a), we explore a novel cross-modal GCN model based on the important visual information and sentiment cues to leverage the inconsistent implications between different modalities and thus improve the performance of multimodal sarcasm detection. Models based on graph neural networks (GNN), including graph convolutional network (GCN) (Kipf and Welling, 2017) and graph attention network (GAT) (Velickovic et al., 2018), have achieved promising performance in many recent research studies, such as visual representation learning (Wu et al., 2019; Xie et al., 2021), text representation learning (Yao et al., 2019; Lou et al., 2021; Liang et al., 2021b, 2022), and recommendation systems (Ying et al., 2018; Tan et al., 2020). Further, there are also some research studies explored graph models to deal with the multi-modal tasks, such as multi-modal sentiment detection (Yang et al., 2021), multi-modal named entity recognition (Zhang et al., 2021), cross-modal video moment retrieval (Zeng et al., 2021), multi-modal neural machine translation (Yin et al., 2020), and multimodal sarcasm detection (Liang et al., 2021a). In this section, we describe our proposed Cross-Modal Graph Convolutional Networks ( CMGCN ) model for multi-modal sarcasm detection in details. As demonstrated in Figure 2, the architecture of the proposed CMGCN contains four main components: 1) Text-modality representation , which employs the pre-trained uncased BERT-base model (De-vlin et al., 2019) as the text encoder to capture the hidden representation of the text-modality; 2) Image-modality representation , which deploys the pre-trained Vision Transformer (ViT) (Dosovit-skiy et al., 2021) as the image encoder to capture the hidden representation of the image-modality with respect to each bounding box (visual region); 3) Cross-modal graph , which constructs a cross-modal graph for each multi-modal example based on the external affective knowledge source and the hidden representations of text and image modalities; 4) Multi-modal fusion , which fuses the representations from image and text modalities to capture the sarcastic features by means of a GCN structure and an attention mechanism. For text processing, given a sequence of words s = { w i } ni =1 , n is the length of the text s . We first adopt the pre-trained uncased BERT-base 1769 model (Devlin et al., 2019) to map each word w i into a d T -dimensional embedding: XT = [ x 1 , x 2 ,    , x n ] = BERT( [CLS] s [SEP] ) (1) Where XT is the embedding matrix of the input text. Here, the representations of tokens [CLS] and [SEP] are not utilized in constructing the cross-modal graph. Subsequently, to unify the dimensions of representations between different modalities and capture the sequential relations of the context, we utilize a bidirectional LSTM (Bi-LSTM) to learn the text-modality representation of the input text: T = { t 1 , t 2 ,    , t n } = Bi LSTM( XT ) (2) Where t j  R 2 d h denotes the hidden state vector at time step j from the bidirectional LSTM, d h denotes the dimensionality of the text-modality hidden state representation. For image processing, given an image I , we first adopt a trained toolkit proposed by Anderson et al. (2018) to derive a series of bounding boxes (objects) paired with their attribute-object pairs. For each visual region of the bounding box I i  RL h  L w , following (Xu et al., 2020), we first resize it to 224  224 , i.e. L = L h = L w = 224 . Subsequently, following (Dosovitskiy et al., 2021), we reshape the region I i  RL  L into a sequence I i = { p j  R L/p  L/p } rj =1 , where r = p  p is the number of patches. Then, we flatten and map each patch to a d I -dimensional vector with a trainable linear projection: z j = p j E . For each sequence of image patches, a [class] token embedding z [ class ]  R d I is prepended for the sequence of embedded patches, and position embeddings are added to the patch embeddings to retain positional information. The input of each visual region I i is represented as: Z i = [ z [ class ] ; z 1 ; z 2 ;    ; z r ] + E pos (3) Where Z i  R ( r +1)  d I is the input matrix of the image patches, and E pos  R ( r +1)  d I is the position embedding matrix. Then, we feed the input matrix Z i into the ViT encoder to acquire the representation h i of visual region I i : H i = ViT( Z i ) , h i = H i, [ class ] (4) We use the representation of the [class] token embedding to represent the visual region. Finally, the representation of the image I is defined as: XI = { h 1 , h 2 ,    , h m } (5) Where m is the number of visual regions. Subsequently, we employ a trainable Linear Projection to map each v i to a 2 d h -dimensional vector: V = { v 1 , v 2 ,    , v m } = XIWV (6) Where WV  R d I  2 d h is a trainable parameter. In this section, we describe how to construct a cross-modal graph. To leverage the relations between multi-modal features, we employ a graph structure to link the textual words with the associated image objects. Here, the nodes of the cross-modal graph are the representations of text and image modalities. Many GCN-based approaches have demonstrated that the weights of the edges are crucial in graph information aggregation (Liang et al., 2021b; Yang et al., 2021; Lou et al., 2021). As such, constructing a cross-modal graph boils down to the setting of the edge weights in the graph. To this end, we explore a novel approach of setting the weights based on both word similarities and affective clues between textual words and the attribute-object pairs of the image regions, and the dependency tree of the text-modality. The adjacency matrix A  R ( n + m )  ( n + m ) of the cross-modal graph is defined as: A i,j =   1 if D i,j and i < n , j < n  i,j if i < n , j  n 0 otherwise (7)  i,j = Sim ( w i , o j )   i,j + 1 (8)  i,j =    ( w i )  ( a j )  |  ( w i )   ( a j ) | (9) Where D i,j indicates that there is a relation between w i and w j in the dependency tree of the sentence. Sim (  ) represents the computation of word similarity 2 . We set Sim (  ) = 0 if the return value is None . i,j is a modulating factor refers to the sentiment relation (sentiment incongruity) between an image region and a text token. ( w i )  [  1 , 1] represents the affective weight of 2 We employ the NLTK toolkit ( http://www.nltk. org/ ) to compute the similarity of a word pair based on the WordNet. word w i retrieved from SenticNet (Cambria et al., 2020). We set  ( w i ) = 0 if w i cannot be found in SenticNet. || represents absolute value calculation. a j and o j respectively denote the attribute and the object of the bounding box j . Inspired by Kipf and Welling (2017), we construct the cross-modal graph as an undirected graph, A i,j = A j,i , and set a self-loop for each node, A i,i = 1 . The intention of the cross-modal graph construction (Equations 7 and 9) is that: 1) As in the examples shown in Figure 1, the sarcastic information of text-modality may be expressed by multiple words, such as  wonderful weather . Therefore, we incorporate the syntax-aware relations over the dependency tree of the sentence into the cross-modal graph to advance the learning of the contextual dependencies 3 . 2) We devise a coefficient  i,j , which is associated with the affective weights, to modulate the influence of contrary sentiment relations. Here,  > 1 is a tuned hyper-parameter to regulate the bias of inconsistent sentiment relations. That is, if the polarities of  ( w i ) and  ( a j ) are opposite, the value of  is boosted, otherwise the value is shrunk. Especially, the greater the affective weights, the higher the confidence that the value of  is boosted or shrunk. 3) We add 1 to the cross-modal edges to pay more attention to the cross-modal nodes aggregation. For each instance, we explore a graph architecture to extract the crucial sarcastic clues by aggregating the correlation of nodes in the cross-modal graph. Concretely, we feed the adjacency matrix of the cross-modal graph A and the corresponding nodes' representations R of each multi-modal example into a multi-layers GCNs architecture to derive the graph representation. For each graph convolutional operation, each node in the l -th GCN layer is updated according to the hidden representations of its neighborhoods according to the adjacency matrices of the cross-modal graph, which is defined as: G l = ReLU(  AG l  1 W l + b l ) (10) Where  A = D  12 AD  12 is the normalized symmetric adjacency matrix. D is the degree matrix of A , where D ii = (cid:80) j A i,j . G l  1 is the hidden graph representation evolved from the preceding GCN layer. W l  R 2 d h  2 d h , b l  R 2 d h 3 We employ the spaCy toolkit ( https://spacy.io/ ) to derive the dependency tree of a sentence. are the trainable parameters of the l -th GCN layer. The nodes input of the first GCN layer are the concatenation of text-modality and image-modality representations: G 0 = R . Here, R = { r 1 , r 2 ,    , r n + m } = { t 1 ,    , t n , v 1 ,    , v m } . Subsequently, inspired by (Zhang et al., 2019), we employ a retrieval-based attention mechanism to capture the graph-oriented attention information from the concatenation of text and image representations R = { r 1 , r 2 ,    , r n + m } by means of the graph representation g derived from the final GCN layer. The intention is to retrieve crucially associated cross-modal features that are explicitly connected in the cross-modal graph. The attention weights are computed as:  t = exp(  t ) (cid:80) n + m i =1 exp(  i ) ,  t = (cid:88) i C r  t g i (11) Where C denotes a set of indices in which nodes contain cross-modal edges in the graph. represents the matrix transposition. The final sarcastic representation is defined as: f = n + m (cid:88) t =1  t r t (12) Then, the final sarcastic representation is fed into a fully-connected layer with a softmax function to capture a probability distribution  y  R d p in the sarcasm decision space:  y = softmax( W o f + b o ) (13) Where d p is the dimensionality of sarcasm labels. We minimize the cross-entropy loss via the standard gradient descent algorithm to train the model: where N is the training data size. y i and  y i respectively represent the ground-truth and estimated label distribution of instance i . denotes all trainable parameters of the model,  represents the coefficient of L 2 -regularization. We conduct experiments on a publicly available multi-modal sarcasm detection benchmark dataset collected by Cai et al. (2019). This dataset contains English tweets expressing sarcasm as Positive examples and those expressing non-sarcasm as Negative examples. Each example in the dataset consists of a text and an associated image. The statistics of the dataset are shown in Table 1. For a fair comparison, the data preprocessing follows (Cai et al., 2019). We set the maximum number of visual regions as 10 for object detection results. That is, we select the top 10 bounding boxes with highest scores if the objects are greater than 10. We utilize the pre-trained uncased BERT-base (De-vlin et al., 2019) module to embed each word of text-modality as a 768-dimensional embedding and employ the pre-trained ViT 4 (Dosovitskiy et al., 2021) to embed each visual region patch as a 768-dimensional embedding, i.e. d T = d I = 768 . The resolution of visual region patch is set to L p = 32 , correspondingly, p = 7 , r = 49 . 5 The number of GCN layers is set to 2, which is the optimal depth in the pilot experiments. The dimensionality of hidden representations is set to d h = 512 . The coefficient  is set to 0 . 00001 . Adam is utilized as the optimizer with a learning rate of 0 . 00002 , and the mini-batch size is 32. The dropout rate with 0 . 1 is utilized to avoid overfitting. We use early-stopping with patience of 5. We set  = 3 to compute the modulating factor of incongruous multi-modal sentiment relations, which is the optimal hyper-parameter in the pilot experiments. Following (Cai et al., 2019), we use Accuracy , Precision , Recall , and F1-score to measure the model performance. Since the label distribution of the dataset is imbalanced, following (Pan et al., 4 https://github.com/lukemelas/ PyTorch-Pretrained-ViT 5 We also tried other division resolutions, and found that the fluctuation of performance is negligible over different resolutions of image patches. 2020), we also report Macro-average results. The experimental results of our models are averaged over 10 runs with different random seeds to ensure the final reported results are statistically stable. 1) Image-modality methods : These models use only visual information for sarcasm detection, including Image (Cai et al., 2019), which employs ResNet (He et al., 2016) to train a sarcasm classifier; and ViT (Dosovitskiy et al., 2021), which utilizes the  [class] ' token representation of the pre-trained ViT to detect the sarcasm. 2) Text-modality methods : These models use only textual information, including TextCNN (Kim, 2014), a deep learning model based on CNN for text classification; Bi-LSTM , a bidirectional LSTM network for text classification; SIARN (Tay et al., 2018), adopting inner-attention for textual sarcasm detection; SMSD (Xiong et al., 2019), exploring a self-matching network to capture textual incongruity information; and BERT (Devlin et al., 2019), the vanilla pre-trained uncased BERT-base taking  [CLS] text [SEP] ' as input. 3) Multi-modal methods : These models take both textand image-modality information. Including HFM (Cai et al., 2019), a hierarchical multimodal features fusion model for multi-modal sarcasm detection; D&R Net (Xu et al., 2020), a Decomposition and Relation Network modeling both cross-modality contrast and semantic association; Res-BERT (Pan et al., 2020), concatenating image features and BERT-based text features for sarcasm prediction; Att-BERT (Pan et al., 2020), exploring an inter-modality attention and a co-attention to model the incongruity of multi-modal sarcasm detection; and InCrossMGs (Liang et al., 2021a), a graph-based model to leverage the sarcastic relations from both intraand inter-modal perspectives. We also explore several variants of CMGCN to analyze the impact of different components in the ablation study: 1) w/o G denotes without cross-modal graph, which only concatenates the representations of  [class] ' and  [CLS] ' tokens from ViT and BERT for sarcasm detection; 2) w/o O denotes without object detection. The whole image is input into the image encoder, and the edge weights are set to 1 in the cross-modal graphs; 3) w/o S denotes without using external knowledge. All weights of edges are set to 1 in the cross-modal graph. Further, 4) w/o S w represents without using affective knowledge; 5) w/o D denotes without using syntax-aware information of text-modality in graph construction. Further, to investigate the effectiveness of our CMGCN when used with different pre-trained models, we also set the following variants: 1) -GloVe+ResNet : We replace BERT with GloVe (Pennington et al., 2014) to initialize each word into a 300-dimensional embedding and ViT with ResNet-152 (He et al., 2016) to embed each image patch as a 2048-dimensional vector. 2) -GloVe+ViT : We use GloVe as text encoder and use ViT as image encoder. 3) -BERT+ResNet : We use BERT as text encoder and use ResNet-152 as image encoder. We report the comparison results regarding Text-modality , Image-modality , and Text+Image modalities in Table 2. From the results, we can draw the following conclusions. 1) Our proposed CMGCN outperforms existing baselines across all metrics. This verifies the effectiveness of our proposed model in multi-modal sarcasm detection. 2) We conduct significance tests of our CMGCN over the baseline models, the results show that our CMGCN significantly outperforms the baseline models in terms of most of the evaluation metrics (with p  value < 0 . 05 ). 3) Our CMGCN model performs consistently better than the previous graph-based method (InCrossMGs), which MODEL Acc. demonstrates that recognizing significant visual regions and modeling sentiment relations can lead to improved performance. 4) The methods based on text modality achieve consistently better performance than the methods based on image modality, which shows that the expression of sarcastic/non-sarcastic information primarily resides in the text modality. 5) Methods based on both image and text modalities perform better than the unimodal baselines overall. This implies that leveraging the information of both image and text modalities is more effective for multi-modal sarcasm detection. 6) The results of macro metrics are better than other commonly used metrics overall, which indicates that models perform better in the  negative  class due to the imbalanced class distribution. To analyze the impact of different components of our proposed CMGCN , we conduct an ablation study and report the results in Table 3. Note that removal of cross-modal graph (w/o G ) sharply degrades the performance, which verifies the significance of cross-modal in multi-modal features fusion for learning sarcastic expressions in multimodal sarcasm detection. Removal of object de-1773 tection (w/o O ) leads to considerable performance degradation, which demonstrates that adopting object detection to track important visual information is effective for constructing crucial relations between visual and textual information in the cross-modal graphs. From the results of w/o S and S w , we conclude that exploiting the attribute-object pair as a bridge to set edge weights based on word similarity is effective when constructing cross-modal graphs. Further, leveraging affective clues to capture multi-modal sentiment incongruity between textand image-modality is effective in sarcasm detection, and thus leads to improved performance. In addition, removal of syntax-aware information of text-modality leads to slight performance degradation, which indicates that incorporating syntactic information in the graph makes better learning of dependency relations of textual words and thus improves the performance of sarcasm detection. To investigate the generalizability and effectiveness of our proposed cross-modal graph when used with different pre-trained methods, we conduct experiments with five variants of our proposed CMGCN by using different text and image encoders. The experimental results are shown in Figure 3 (a). Note that the proposed cross-modal graph can directly work with various pre-trained models and performs consistently better than that without cross-modal graph (w/o G ). This demonstrates the generalizability and effectiveness of our proposed cross-modal graph in multi-modal sarcasm detection. Further, from the results, we can also conclude that superior performance is obtained when using more powerful pre-trained methods, such as ViT and BERT. In this section, we analyze the impact of the number of GCN layers on the performance of our pro-branches posed CMGCN . We vary the layer number from 1 to 6 and report the results in Figure 3 (b). Note that the 2-layer GCN architecture performs better than others overall, and thus the number of GCN layers is set to 2 in our model. Model with one layer performs worse, which indicates that a shallow graph network structure is not able to learn sarcastic features well. When the number of layers is greater than 2, the performance tends to decline. This shows that further increasing the number of layers beyond 2 degrades the model performance possibly due to the sharp increase of parameters. To qualitatively investigate how the proposed CMGCN works in multi-modal sarcasm detection, we present a visualization of cross-modal graph construction and attention values of a multi-modal sarcasm example. The results are shown in Figure 4. We first show a sarcasm example and its corresponding object detection results in Figure 4 (a). Note that the correct label of this example can be easily inferred if the relations of crucial sarcastic clues of text (marked by the light red color) and the corresponding visual regions are captured by the model. To demonstrate how the proposed CMGCN identifies the important sarcastic clues, we show the adjacency matrix of the cross-modal graph of this example in Figure 4 (b). Note that highly correlated sarcastic clues in different modalities are connected by edges with large weights in the graph. This verifies the effectiveness of the proposed cross-modal graph in learning multi-modal sarcastic information. Further, based on the cross-1774 modal graph, we show the attention visualization of this example in Figure 4 (c). The crucial textual tokens and the related image regions are highly attended by our proposed CMGCN , which helps identify the incongruity among the learned important features for learning sarcastic expressions and thus leads to improved performance of multi-modal sarcasm detection. This paper has proposed a novel cross-modal graph architecture for multi-modal sarcasm detection, in which the crucial visual regions can be explicitly connected to the highly correlated textual tokens for learning the incongruity sentiment of sarcastic expression. Specifically, unlike previous research efforts that simply consider the visual information of the whole image, we attempt to recognize the important visual regions via object detection results, and further devise a novel cross-modal graph to explicitly establish the connections of scattered visual regions and the associated textual tokens. More concretely, owing to the object detection results, the attribute-object pair descriptors of the objects are served as a bridge to track the highly related sarcastic cues between image and text modalities and their connection weights, and then deriving the cross-modal graphs based on external knowledge bases. Afterwards, a GCNs architecture based on a retrieval-based attention mechanism is employed to capture the key incongruity sentiment expressions across different modalities for multi-modal sarcasm detection. To the best of our knowledge, it is the first study of utilizing a cross-modal graph to extract intricate multi-modal sarcastic relations via object detection and sentiment cues from external knowledge bases. Extensive experiments on a public benchmark dataset show that our proposed approach significantly outperforms state-of-the-art baseline methods. As described in Section 3.3, the weights of edges in the cross-modal graph are computed based on both word similarities and affective clues between textual words and the attribute-object pairs of the image regions, and the dependency tree of the text-modality. The approach can be easily generalized to other sentiment-related multi-modal learning scenarios. Nevertheless, the cross-graph solution might not be generalized well to other multi-modal tasks or data genres, if there is a lack of affective knowledge or a difficulty in deriving dependency trees in low-resource settings. Therefore, future research can consider exploiting alternatively approaches to automatically learn the weights of edges in the cross-modal graph without relying on external knowledge sources. This work was partially supported by the National Natural Science Foundation of China (61876053, 62006062, 62176076, 62006060), UK Engineering and Physical Sciences Research Council (grant no. EP/V048597/1, EP/T017112/1), Natural Science Foundation of Guangdong Province of China (No. 2019A1515011705), Shenzhen Foundational Research Funding (JCYJ20200109113441941, JCYJ20210324115614039), Shenzhen Science and Technology Innovation Program (Grant No. KQTD20190929172835662), Joint Lab of Lab of HITSZ and China Merchants Securities. Yulan He is supported by a Turing AI Fellowship funded by the UK Research and Innovation (UKRI) (grant no. EP/V020579/1).
abstain objective objective abstain abstain result abstain objective abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain objective abstain abstain objective method abstain abstain objective objective objective other other other other other other other other other abstain other other abstain other abstain abstain abstain method abstain abstain method abstain abstain abstain method abstain method method method abstain abstain abstain abstain other abstain abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain abstain other method abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain abstain other abstain other abstain abstain abstain abstain abstain method method result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain objective objective abstain abstain objective objective abstain abstain abstain abstain other other