Sarcasm is important to sentiment analysis on social media. Sarcasm Target Identification (STI) deserves further study to understand sarcasm in depth. However, text lacking context or missing sarcasm target makes target identification very difficult. In this paper, we introduce multimodality to STI and present Multimodal Sarcasm Target Identification (MSTI) task. We propose a novel multi-scale cross-modality model that can simultaneously perform textual target labeling and visual target detection. In the model, we extract multi-scale visual features to enrich spatial information for different sized visual sarcasm targets. We design a set of convolution networks to unify multi-scale visual features with textual features for cross-modal attention learning, and correspondingly a set of transposed convolution networks to restore multi-scale visual information. The results show that visual clues can improve the performance of TSTI by a large margin, and VSTI achieves good accuracy. Sarcasm is a type of sentiment in which people express their negative feelings using positive or intensified positive words. It has the power to disguise the hostility of the speaker (Dews and Winner, 1995), thereby enhancing the effect of mockery or humor on the listener. Sarcasm is prevalent on today's social media platforms such as Twitter, and automatic Sarcasm Target Identification (STI) bears great significance in customer service, opinion mining, and online harassment detection. Previous works about STI have focused on text modality and proposed some methods such as rule-based, statistical classifier-based (Joshi et al., 2018), and deep learning models with socio-linguistic features (Patro et al., 2019). For example, in Figure 1(a), we are not sure whether the context conveys positive or negative emotions if we only see the text This guy definitely deserves $15 an hour!. However, the negative information comes from the image. When observing a lazy guy in the picture lying on the chair, we can easily determine that the lazy guy is a sarcasm target and label the text This guy as a sarcasm target (ST). Moreover, the sarcasm target sometimes does not appear explicitly in the text, which is marked as  OUTSIDE '. In ALTA Shared Task (Molla and Joshi, 2019),  OUTSIDE ' cases account for over 30% of the data. For example, in the tweet of Figure 1(b), the author teased that the skirt was too long, similar to a bed sheet; therefore, the sarcasm target should be the long skirt. However, no sarcasm target appears in the text but we can label the long skirt as an ST with a blue bounding box in the picture. The above examples illustrate the necessity of combining images for STI. In this paper, we introduce a novel task called Multimodal Sarcasm Target Identification (MSTI) on social media data. The MSTI task is to extract sarcasm targets (STs) from both texts and images 8164 in tweets. The textual ST is a word or a phrase, and the visual ST is an object labeled by a bounding box, as shown in Figure 1. The challenge of the MSTI task is not only to extract both textual and visual features but also to leverage cross-modality interaction and semantic learning to improve the performance of STI. The contributions of this paper can be summarized as follows:  To the best of our knowledge, this is the first attempt to perform the MSTI task. We build an MSTI dataset and propose a novel cross-modality MSTI framework. Textual and multi-scale visual features are fused in a cross-modality encoder via convolution and transposed convolution. Our model performs textual and visual tasks simultaneously in an end-to-end manner, and it can leverage textual and visual contexts to enhance textual and visual representations for MSTI. We design a cross-modality attention visualization method in terms of text-to-image and image-to-text to illustrate the mutual effects between textual and visual modalities. These results show the image regions and words extracted through cross-modality attention are the keys to sarcasm and explain the improved performance of TSTI and VSTI by cross-modality learning. The comprehensive experimental results are presented. The results indicate that the images in tweets improve the performance of TSTI by a large margin. Comparisons with textual, object detection, and pretrained multimodal baselines show the advanced performance of our model. The existing research on sarcasm analysis mainly focuses on sarcasm detection (SD) and sarcasm target identification (STI). We begin the literature review with textual sarcasm. Textual Sarcasm Detection. Traditional sarcasm detection is defined as a binary classification of sarcastic or non-sarcastic sentiments in text (Guo et al., 2021). Earlier approaches (Joshi et al., 2017) were based on sarcastic pattern rules (Riloff et al., 2013) or statistical models such as SVM (Joshi et al., 2015) or logistic regression (Bamman and Smith, 2015). Recently, deep learning techniques have gained popularity. Word embeddings and LSTM/CNN model were employed in Joshi et al. (2016); Zhang et al. (2016). Furthermore, Peled and Reichart (2017) presented a neural machine translation framework and Tay et al. (2018) proposed an attention-based neural model to interpret and reason with sarcasm. Xiong et al. (2019) proposed a self-matching network to capture incongruity information by exploring word-to-word interactions. Agrawal et al. (2020) formulated sarcasm detection as a sequence classification problem by leveraging the natural shifts in various emotions over the course of a piece of text. Babanejad et al. (2020) extended the architecture of BERT by incorporating both affective and contextual feature embeddings. Guo et al. (2021) proposed a latent-optimized adversarial neural transfer model for cross-domain sarcasm detection. Textual Sarcasm Target Identification. To deepen the field of sarcasm analysis, STI has been well studied recently (Patro et al., 2019; Parameswaran et al., 2021). The goal of STI is to label the subject of mockery or ridicule within sarcastic texts. Patro et al. (2019) showed that the Exact Match (EM) accuracy on tweets is approximately 30%. Joshi et al. (2018) introduced the STI problem and summarized the 2019 ALTA shared task regarding STI (Molla and Joshi, 2019). The evaluation metrics such as EM accuracy and F1 score were presented. Patro et al. (2019) presented a deep learning framework augmented with socio-linguistic features to detect sarcasm targets. Parameswaran et al. (2019) employed an ensemble of classifiers such as SVM, logistic and linear classifiers to classify  OUTSIDE ' and  NOT OUTSIDE ', then used a rule-based approach to extract the target sarcasm words from the  NOT OUTSIDE ' samples. Multimodal Sarcasm Detection. Benefiting from images, multimodal sarcasm detection (MSD) has gained increasing research attention. Schifanella et al. (2016) first tackled this task as a multimodal classification problem. They concatenated the visual and textual features and employed SVM or a neural network consisting of fully connected and softmax layers, to detect sarcasm. Cai et al. (2019) extended the input modalities to a triplet of text, image, and image attributes, and they proposed a hierarchical fusion model for sarcasm detection. Castro et al. (2019) proposed a video-level multimodal sarcasm detection task. Features were ob-8165  This guy definitely deservces $ 15 an hour ! tained from three modalities, i.e., text, speech, and video, and an SVM classifier with RBF kernel was employed. Sangwan et al. (2020) presented an RNN-based model and gating mechanism, which attempted to decide the weight of image modality regarding textual modality. Pan et al. (2020) proposed a BERT-based model that concentrated on both intraand inter-modality incongruity for multimodal sarcasm detection. Xu et al. (2020) constructed the decomposition network to model the discrepancy between image and text and the relation network to model the semantic association in cross-modality context. Figure 2 illustrates the overall architecture of our MSTI model. The model mainly consists of five components: (1) Backbone to MCE converter (B2M), (2) Multi-scale Cross-modality Encoder (MCE), (3) MCE to Neck network converter (M2N), (4) Textual Sarcasm Target Identification (TSTI), and (5) Visual Sarcasm Target Identification (VSTI). We first extract textual and visual features separately. The multi-scale visual features of the last three blocks of the pretrained backbone network are unified to the same dimension by B2M and input to the MCE together with textual features. The MCE outputs cross-modality representations, where the parts corresponding to textual features are fed into BiLSTM-CRF to label the sequence for TSTI and the parts corresponding to multi-scale visual features restored by M2N are connected to the neck and head networks to predict bounding boxes for VSTI. Textual Representation : We obtain contextual word embeddings from pretrained language models (LM) such as BERT (Devlin et al., 2019) to extract linguistic features. Let S = ([ CLS ] , t 1 , ..., t n , [ SEP ]) be the token sequence and e = ( e 1 , ..., e n ) be the contextual word embeddings generated by a pretrained LM, where e i  R d . As shown in Figure 2, the contextual word embeddings e represent the textual input of the next module MCE. such as ResNet (He et al., 2016), VGG (Simonyan and Zisserman, 2014), and CSPDarkNet (Wang et al., 2020). To improve the detection performance of sarcasm targets with various sizes, our model performs VSTI using multi-scale visual features. The multi-scale outputs at the last three blocks of the backbone are denoted as v 1 , v 2 , and v 3 , shown in Figure 2, for the later use of the Neck network. The dimensions of v 1 , v 2 , and v 3 are d s 1  d s 1  d 1 , d s 2  d s 2  d 2 , and d s 3  d s 3  d 3 , respectively, where d s i  d s i represents image scale and d i represents feature map, i = { 1 , 2 , 3 } . The B2M converter aims to unify the dimensions of three visual features with the dimension d of textual feature and lower the scales of three visual features to reduce the computation of the MCE. The B2M has three parts B 1 , B 2 , and B 3 corresponding to the visual features v 1 , v 2 , and v 3 . Each part consists of convolutional layers followed by Rectified Linear Unit (ReLU) and max pooling layer. Table 1 shows the architecture of B2M. The input dimensions of B 1 , B 2 , and B 3 are 19  19  1024 , 38  38  512 , and 76  76  256 , respectively, when the backbone is set to CSPDarkNet53 (Bochkovskiy et al., 2020). We denote the outputs of B 1 , B 2 , and B 3 as v B 1 , v B 2 , and v B 3 , respectively. According to the computation of the convolutional layer, the output scale is (cid:4) I +2 P  K S (cid:5) + 1 , where I is the dimension of input scale, P is padding, S is stride, and K is kernel. The Conv generates feature maps of d , which is the same as the dimension of the word embeddings. Then, we can obtain all v B 1 , v B 2 , and v B 3 with size 5  5  d . Finally, we flatten the shape size 5  5 to 25 visual tokens { v p,qB i } (1  p, q  5) to generate the visual inputs of the MCE. The MCE is based on the Transformer encoder architecture presented in Vaswani et al. (2017) and shown in the left of Figure 2. The Transformer encoder has a multi-head self-attention sub-layer and a fully connected feed-forward sub-layer. A residual connection and layer normalization are employed around two sub-layers. The Transformer encoder adopts scaled dot-product attention, which is defined as follows: Attention ( Q, K, V ) = softmax ( QKT  d k ) V, (1) B2M B 1 Conv [ K = 3  3 ,P = 1 ,S = 2 ] MaxPooling [ 2  2 ] B 2 Conv (cid:20) K = 3  3 ,P = 1 ,S = 2 K = 3  3 ,P = 1 ,S = 2 (cid:21) MaxPooling [ 2  2 ] B 3 Conv (cid:20) K = 5  5 ,P = 2 ,S = 4 K = 3  3 ,P = 1 ,S = 2 (cid:21) MaxPooling [ 2  2 ] M2N C 1 UpSampling [ 2  2 ] ConvT [ K = 3  3 ,P = 1 ,S = 2 ] C 2 UpSampling [ 2  2 ] ConvT (cid:20) K = 3  3 ,P = 1 ,S = 2 K = 3  3 ,P = 1 ,S = 2 (cid:21) C 3 UpSampling [ 2  2 ] ConvT (cid:20) K = 3  3 ,P = 1 ,S = 2 K = 5  5 ,P = 2 ,S = 4 (cid:21) Table 1: Architecture of B2M and M2N converters. where matrices Q , K , and V consist of queries, keys, and values, respectively, and d k is the dimension of keys. In our model, we concatenate the textual and visual features into a sequence G , G = ( e 1 , ..., e n (cid:124) (cid:123)(cid:122) (cid:125) n , v 1 , 1 B 1 , . . . , v 5 , 5 B 1 (cid:124) (cid:123)(cid:122) (cid:125) 25=5  5 , v 1 , 1 B 2 , ..., v 5 , 5 B 2 (cid:124) (cid:123)(cid:122) (cid:125) 25=5  5 , v 1 , 1 B 3 , ..., v 5 , 5 B 3 (cid:124) (cid:123)(cid:122) (cid:125) 25=5  5 ) . We feed G into the MCE, and therefore Q = K = V = G (cid:62) . The outputs of the MCE are divided into two parts: the corresponding textual part e C is used for TSTI, and the corresponding multi-scale visual parts v C 1 , v C 2 , and v C 3 are used for VSTI. The M2N converter is an inverse procedure of the B2M converter. The dimensions of the output v N i of the M2N converter are the same as those of the input v i of the B2M converter, where i = { 1 , 2 , 3 } . The M2N converter has three parts C 1 , C 2 , and C 3 , corresponding to B 1 , B 2 , and B 3 of the B2M converter, respectively. The architecture of M2N is shown in Table 1. Each part consists of transposed convolution (ConvT) (Dumoulin and Visin, 2016) followed by ReLU and upsampling layer. The ConvT is considered as the reverse operation of convolution. If the ConvT's kernel size, padding size, and stride are the same as those carried out on the Conv layer, then the ConvT generates the same spatial dimension as that of the Conv's input. Upsampling reverses the pooling operation by the nearest-neighbor interpolation algorithm. We use the BIO (short for Beginning, Inside, and Outside) schema (Ramshaw and Marcus, 1995) to label textual sarcasm targets. The B-ST' tag indicates the beginning of an ST and the I-ST' tag indicates the inside of an ST. The O' tag indicates that a token does not belong to any ST. We employ a classical sequence tagging model, i.e., BiLSTM-CRF (Huang et al., 2015), to label the textual STs. The bidirectional LSTM (BiLSTM) first processes each sentence token-by-token and produces forward and backward hidden vectors for each token. Then the concatenation of the two hidden vectors is input to a Conditional Random Fields (CRFs) layer (Lafferty et al., 2001). For a sequence of tags y = { y 1 , . . . , y n } , the probability of the label sequence y is defined as follows: p ( y | x ) = e s ( x,y ) (cid:80) y (cid:48)  Y e s ( x,y (cid:48) ) , (3) where Y is all possible tag sequences for the sentence x and s ( x, y ) are feature functions modeling transitions and emissions. The computation details can be found in Lample et al. (2016). The objective of labeling ST is to minimize the negative log-likelihood over the training data D t = { ( x ( i ) , y ( i ) ) } Mi =1 : LTSTI =  M (cid:88) i =1 log ( p ( y ( i ) | x ( i ) )) . 3.7 Visual Sarcasm Target Identification There are two kinds of object detectors, one-stage and two-stage. One-stage object detector such as YOLO (Redmon et al., 2016) is faster and simpler. In this paper, we adopt YOLOv4's Neck and Head networks (Bochkovskiy et al., 2020) to perform VSTI. The multi-scale cross-modality features v N 1 , v N 2 , and v N 3 are connected to the Neck network, which consists of Spatial Pyramid Pooling (SPP) (He et al., 2015) and Path Aggregation Network (PANet) (Liu et al., 2018). The Neck network is to increase the receptive field and preserve spatial information. The Head network is used for predicting bounding boxes at 3 different scales. The output tensor of the Head network of YOLOv4 is d s i  d s i  [3  (4 + 1 + C )] at each scale, predicting 3 boxes per grid cell where each box has 4 bounding offsets ( t x , t y , t w , t h ) , 1 objectness score, and C class scores. The detailed computation of bounding offsets can be found in Redmon and Farhadi (2018). Each grid cell predicts the object probability and C class probabilities. In our model, since there is 1 sarcasm object class, we ablate C class scores. As in YOLO, L b is based on the bounding box priors that are assigned to ground truth objects and computed by mean squared error (MSE). L o is computed by binary cross-entropy (BCE) for classifying the bounding box priors as object or non-object. Finally, combining the TSTI and VSTI tasks, the objective function for MSTI is as follows: LMSTI = LTSTI + LV STI . In this paper, we build an MSTI dataset for pub-lic research 1 . We label textual and visual STs on the dataset collected by Cai et al. (2019) for multimodal sarcasm detection. Each sample is manually annotated by three persons based on their common sense. The agreement between the annotators is measured using a percentage of overlapping choices between the annotators, i.e., above one word overlapping for text phrases and above 50% intersection-over-union (IoU) overlapping for image regions. We ensure the quality of ground truth by keeping the consistency of all annotator's 1 https://github.com/wjq-learning/MSTI 8168 #Tweet #Textual ST #Visual ST Train 3,546 2,501 2,172 Dev 727 542 543 Test 742 573 524 Table 2: Statistics of the MSTI dataset. opinions. The samples with annotations that the three annotators agree on are put into the dataset otherwise removed, making the annotations valid. Figure 3 shows three examples. The statistics of the MSTI dataset are shown in Table 2. The MSTI dataset is split into 3,546/727/742 as Train/Dev/Test in experiments. The number of textual and visual sarcasm targets are also listed. Table 3 shows the proportion of multimodal sarcasm target types, i.e., STs appear both in the text and image, ST only appears in the text, and ST only appears in the image. Table 4 shows the number and percentage of different sized visual STs. We categorize the size of ST as small (area occupation<1.5%), medium (1.5%<area oc-cupation<10%), and large (area occupation>10%). We use CSPDarkNet53 as backbone and BERT-Base ( d =768) or BERT-Large ( d =1024) as LM. All images are shaped to a size of 608  608 . In the VSTI, we use the default settings of YOLOv4, e.g., IoU threshold and object confidence threshold. The weights of neural network are randomly initialized except the pretrained BERT, backbone and Neck networks. We train the model using the Adam (Kingma and Ba, 2014) optimizer with default settings and the learning rate is set to 1e-4. All pretrained models are finetuned with a learning rate of 1e-5. The mini-batch size is set to 8 and dropout rate is 0.5. We use two-layer BiLSTM with 768 hidden states and Transformer encoder with 12 heads. 2018) and F1 score (Molla and Joshi, 2019) as evaluation metrics for TSTI. The EM accuracy is computed as the number of samples that strictly match the boundaries of gold annotations divided by the total number of samples. The F1 score = 2/(1/P+1/R) is calculated from precision P = TP/(TP+FP) and recall R = TP/(TP+FN), where TP is correctly predicted target word, FP is incorrectly predicted target word, and FN is target word but not predicted. Average Precision (AP) is widely used to evaluate object detection (Lin et al., 2014). The COCO-style AP, AP 50 , and AP 75 are evaluated for VSTI. The COCO-style AP averages AP at IoU=[0.5:0.05:0.95]. AP 50 corresponds to AP at IoU=0.5 and AP 75 corresponds to AP at IoU=0.75. IoU measures the amount of overlap between two bounding boxes and here is used as a criterion that determines if a prediction matches ground truth. We train the model on one machine with an NVIDIA RTX 3090 (GPU) and Intel Core i9 10900K (CPU). Our model takes approximately 10 hours for 60 epochs in training. Our model is compared with text baselines such as rule-based & statistical extractors, socio-linguistic features, and BERT, object detection baselines such as Mask R-CNN (He et al., 2020) and YOLOv4, and pretrained multimodal baselines such as VL-BERT (Su et al., 2019) and Unicoder-VL (Li et al., 2020). Rule-based & Statistical Extractors. Joshi et al. (2018) introduced STI and proposed a method based on rules and statistical classification extractors. The sarcasm target was determined based on the results of two extractors. The configuration of R2 and Hybrid AND' performs the best on the MSTI dataset. We test the source code 2 as a baseline of TSTI. Socio-linguistic Features. Patro et al. (2019) presented a deep learning framework augmented with socio-linguistic features to detect textual ST. Sociolinguistic features include the distribution of location (LOC) and organization (ORG) named entities, the distribution of POS tags, and the distribution of LIWC and Empath (Fast et al., 2016) categories. We test the source code 3 as a baseline of TSTI. BERT. We follow the sequence tagging task of Devlin et al. (2019) as a baseline to perform TSTI. The BERT-based model followed by linear and softmax layers is tested to tag textual ST. Object Detection Models. We treat VSTI as a single object detection problem and test two state-of-the-art models, i.e., Mask R-CNN 4 and YOLOv4 5 . We train the models on the MSTI dataset with the default values of parameters in the repository. Pretrained Multimodal Models. Recently, pretrained multimodal models such as VL-BERT, Unicoder-VL, and UNITER (Chen et al., 2020), have been proposed. These models use regions-of-interest (RoIs) produced by object detectors such as Faster R-CNN (Ren et al., 2016) as visual tokens. The inputs of pretrained multimodal models are RoI features and token embeddings; In this paper, we design an MSTI baseline approach based on pretrained multimodal models as follows: Labeling of the textual STs is based on the outputs of token embeddings, the same as in our TSTI method; The VSTI is performed by a binary classification on the outputs of RoI features followed by linear+softmax layers, and it is trained by the RoIs, which are considered as visual STs when the IoU with gold ST is larger than an optimal value of 0.7, otherwise they are considered as non-STs. We finetune the IoU threshold for non-maximum suppression (NMS) to ignore overlapping RoIs and find that IoU NMS = 0 . 2 is optimal. Table 5 shows the performance of our model compared with text, object detection, and pretrained multimodal baselines on the Dev and Test sets. The results show that the BERT-based sequence tagging models are better than the previous works of STI (Joshi et al., 2018; Patro et al., 2019). Fusing visual clues, our model outperforms BERT-based textual models on average 5.3% in F1 score and 4.4% in EM accuracy. The object detection baselines such as Mask R-CNN and YOLOv4 which are directly trained by sarcastic objects are better than the pretrained multimodal baselines with RoIs detected by a traditional object detector, obtaining an increase of approximately 2% in AP metrics. We test state-of-the-art backbones such as ResNet151 and VGG19, in which scale dimensions of the last three blocks are 19  19 , 38  38 , and 76  76 , respectively, the same as in CSPDarkNet53. Therefore, the B2M in Table 1 can be directly used for ResNet151 and VGG19, and the M2N works if the dimensions of the output feature maps of { C 1 , C 2 , C 3 } are set to {2048, 1024, 512} for ResNet151 and {512, 512, 256} for VGG19, respectively. The results show that CSPDarkNet53 achieves the best performance. In addition, Table 6 reports APs (namely, APS , APM , and APL ) by our best model based on small, medium, and large 8170 EM F1 AP AP 50 AP 75 Our model 37.2 47.9 32.6 51.9 34.6 w/o text -27.4 (-5.2) 44.6 (-7.3) 28.1 (-6.5) w/o image 33.1 (-4.1) 42.8 (-5.1) -w/ text and w/o TSTI loss -29.7 (-2.9) 49.1 (-2.8) 31.6 (-3.0) w/ image and w/o VSTI loss 34.6 (-2.6) 43.9 (-4.0) -Table 7: Ablation results of our model on the Test set. We ablate text (w/o text) or image (w/o image) from our multimodal model. Table 7 shows the results of our model (backbone=CSPDarkNet53, LM=BERT-Large). The performance drops by 5.1% in F1 score and 4.1% in EM accuracy when ablating images, indicating that visual clues are very useful for STI. In addition, we ablate TSTI training (w/ text and w/o TSTI loss) or VSTI training (w/ image and w/o VSTI loss). We observe that by only adding text but not training the textual task, our model can greatly improve the VSTI performance, i.e., from 44.6% to 49.1% in AP 50 . However, by only adding image but not training the image task, our model obtains a small increase of 1.5% EM accuracy and 1.1% F1 score for TSTI. These results indicate that texts has more explicit sarcasm information than images and sarcastic message likely comes more from texts, which are consistent with common sense. We visualize the attentions of the MCE in terms of image-to-text and text-to-image in order to illustrate the sarcasm information added from anther modality. The input of the MCE is composed of textual and multi-scale visual embeddings. We abbreviate G , previously defined in Eq. (2), as G = ( e, v B ) where e = ( e 1 , ..., e n ) and v B = ( v 1 , 1 B 1 , . . . , v 5 , 5 B 1 , v 1 , 1 B 2 , ..., v 5 , 5 B 2 , v 1 , 1 B 3 , ..., v 5 , 5 B 3 ) . The attention weight matrix of the h -th head can be divided to four submatrics and denoted as follows: A h = (cid:18) A h ( e, e ) A h ( e, v B ) A h ( v B , e ) A h ( v B , v B ) (cid:19) . Thus, the scaled dot-product attention of Eq. (1) also can be written as A h G (cid:62) . We define the computation of image-to-text and text-to-image attentions as follows: Text-to-image Attentions. The goal of text-to-image attention is to quantify the effect of text on each image block. We compute the average sum of A h ( v B , e ) across all words, H heads, and 3 scales, then obtain the text-to-image attention weights on the image block with the coordinate ( p, q ) as follows: w v p,q = 1 3 H 3 (cid:88) i =1 H (cid:88) h =1 n (cid:88) k =1 A h ( v p,qB i , e k ) . Image-to-text Attentions. The text-to-image attention aims to quantify the effect of image on each word. We compute the average sum of A h ( e, v B ) across 25 image blocks with 3 scales and H heads, then obtain the image-to-text attention weights as follows: w e k = 1 3 H 3 (cid:88) i =1 H (cid:88) h =1 5 (cid:88) p =1 5 (cid:88) q =1 A h ( e k , v p,qB i ) , (9) where k = 1 , . . . , n . Eq. (9). The attention maps show some meaningful cues discovered by the MCE regarding sarcasm. The red color denotes the highest weights. We scale up the text-to-image attention map to image size using interpolation. As expected, the text-to-image attentions in the middle columns of Figure 4 focus on the regions that are highly relevant to sarcasm targets, such as door in (a), chicken nuggets in (b), and orange peel in (c). Surprisingly, the image-to-text attentions in the right columns of Figure 4 point out the key words to well understand sarcasm. Using these red colored words, the tweet authors express their opinions: (a) Can the door through'? (b) Are the chicken nuggets healthy'? (c) Is the train clean' or is train home pleasant'? In this paper, we introduce a new task for identifying both textual and visual sarcasm targets. This work provides a good attempt to detect sarcasm targets on images. Our model integrates multiple components such as sequence labeling, multi-scale cross-modality learning, and object detection. The experimental results not only illustrate that visual clues can improve the performance of TSTI by a large margin, approximately 5% in F1 score, but also prove that it is feasible to detect sarcasm targets in images, obtaining a good accuracy of 51.9% in AP 50 . This work was supported in part by Zhejiang Provincial Natural Science Foundation of China under Grant No. LGN22F020002, the National Natural Science Foundation of China (NSFC) under Grant No. 62072402, and Key Research and Development Program of Zhejiang Province under Grant No. 2022C03037.
abstain abstain abstain abstain objective method method abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain abstain abstain objective abstain abstain objective objective objective method method objective objective objective result other abstain other other other other other other other other other other other other other other other other other other other other other other other other other other other other other method abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain other abstain method method abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain other abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other result abstain result abstain result abstain abstain abstain abstain result abstain result abstain abstain result result abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain other abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective method abstain abstain other other