As an important task in sentiment analysis, Multimodal Aspect-Based Sentiment Analysis (MABSA) has attracted increasing attention in recent years. However, previous approaches either (i) use separately pre-trained visual and textual models, which ignore the cross-modal alignment or (ii) use vision-language models pre-trained with general pre-training tasks, which are inadequate to identify fine-grained aspects, opinions, and their alignments across modalities. To tackle these limitations, we propose a task-specific Vision-Language Pre-training framework for MABSA (VLP-MABSA), which is a unified multimodal encoder-decoder architecture for all the pretraining and downstream tasks. We further design three types of task-specific pre-training tasks from the language, vision, and multimodal modalities, respectively. Experimental results show that our approach generally outperforms the state-of-the-art approaches on three MABSA subtasks. Further analysis demonstrates the effectiveness of each pretraining task. The source code is publicly released at https://github.com/NUSTM/ VLP-MABSA . Recent years have witnessed increasing attention on the Multimodal Aspect-Based Sentiment Analysis (MABSA) task 1 . Previous research mostly focused on its two subtasks, including Multimodal Aspect Term Extraction (MATE) and Multimodal Aspect-oriented Sentiment Classification (MASC). Given a text-image pair as input, MATE aims to extract all the aspect terms mentioned in the text (Zhang et al., 2018; Lu et al., 2018; Wu et al., 2020a,b; Zhang et al., 2021a), whereas MASC  Corresponding authors. aims to classify the sentiment towards each extracted aspect term (Xu et al., 2019; Yu and Jiang, 2019; Khan and Fu, 2021). As the two subtasks are closely related to each other, Ju et al. (2021) recently introduced the Joint Multimodal Aspect-Sentiment Analysis (JMASA) task, aiming to jointly extract the aspect terms and their corresponding sentiments. For example, given the text-image pair in Table. 1, the goal of JMASA is to identify all the aspect-sentiment pairs, i.e., ( Sergio Ramos , Positive ) and ( UCL , Neutral ). Most of the aforementioned studies to MABSA primarily focused on employing pre-trained unimodal models (e.g., BERT for text and ResNet for image) to obtain textual and visual features respectively. The separate pre-training of visual and textual features ignores the alignment between text and image. It is therefore crucial to perform vision-language pre-training to capture such cross-modal alignment. However, for the MABSA task, the studies on vision-language pre-training are still lacking. To the best of our knowledge, there are very few studies focusing on vision-language pre-training for one of the MABSA subtasks, i.e., MATE (Sun et al., 2020, 2021). One major drawback of these studies is that they mainly employ general vision-language understanding tasks (e.g., text-image 2149 matching and masked language modeling) to capture text-image alignments. Such general pretraining is inadequate to identify fine-grained aspects, opinions, and their alignments across the language and vision modalities. Therefore, it is important to design task-specific vision-language pre-training, to model aspects, opinions, and their alignments for the MABSA task. To address this issue, in this paper, we propose a task-specific Vision-Language Pre-training framework for Multimodal Aspect-Based Sentiment Analysis. Specifically, inspired by the recent success of BART-based generative models in text-based ABSA (Yan et al., 2021), we first construct a generative multimodal architecture based on BART (Lewis et al., 2020), for both vision-language pre-training and the downstream MABSA tasks. We then propose three types of vision-language pre-training tasks, including Masked Language Modeling (MLM) and Textual Aspect-Opinion Extraction (AOE) from the language modality, Masked Region Modeling (MRM) and Visual Aspect-Opinion Generation (AOG) from the vision modality, and Multimodal Sentiment Prediction (MSP) across two modalities. Figure 1 illustrates the whole framework of our proposed pre-training approach. Compared with general pre-training methods, our task-specific pretraining approach incorporates multimodal aspect, opinion, and sentiment supervision, which guides pre-trained models to capture important objective and subjective information for the MABSA task. To evaluate the effectiveness of our pre-training approach, we adopt MVSA-Multi, a widely-used Multimodal Twitter dataset for coarse-grained text-image sentiment analysis (Niu et al., 2016), as our pre-training dataset. We then employ several representative pre-trained models and rule-based methods to obtain the aspect and opinion supervision for our AOE and AOG tasks. As the dataset provides sentiment labels for each multimodal tweet, we adopt them as the supervision for our MSP task. Our contributions in this work are as follows:  We introduce a task-specific Vision-Language Pre-training framework for MABSA named VLP-MABSA, which is a unified multimodal encoder-decoder architecture for all the pretraining and downstream tasks. Apart from the general MLM and MRM tasks, we further introduce three task-specific pretraining tasks, including Textual Aspect-Opinion Extraction, Visual Aspect-Opinion Generation, and Multimodal Sentiment Prediction, to identify fine-grained aspect, opinions, and their cross-modal alignments. Experiments on three MABSA subtasks show that our pre-training approach generally obtains significant performance gains over the state-of-the-art methods. Further analysis on supervised and weakly-supervised settings demonstrates the effectiveness of each pre-training task. Vision-Language Pre-training. Inspired by the success of pre-trained language models like BERT (Devlin et al., 2019), many multimodal pre-training models have been proposed (Chen et al., 2020b; Yu et al., 2021; Zhang et al., 2021b) to perform many vision-language tasks which achieve fantastic success. Correspondingly, many general pre-training tasks are proposed, such as Masked Language Modeling (MLM), Masked Region Modeling (MRM) and Image-Text Matching (ITM) (Chen et al., 2020b; Yu et al., 2021). Besides, in order to make the pre-trained models better understand downstream tasks, researchers also design task-specific pre-training models for different downstream tasks (Hao et al., 2020; Xing et al., 2021). In our work, apart from the popular general pre-training tasks, we also design three kinds of task-specific pre-training tasks for the MABSA task. Text-based Joint Aspect-Sentiment Analysis (JASA). JASA aims to extract aspect terms in the text and predict their sentiment polarities. Many approaches have been proposed including pipeline approaches (Zhang et al., 2015; Hu et al., 2019), multi-task learning approaches (He et al., 2019; Hu et al., 2019) and collapsed label-based approaches (Li et al., 2019; Hu et al., 2019; Chen et al., 2020a). Recently, Yan et al. (2021) proposed a unified generative framework which achieves highly competitive performance on several benchmark datasets for JASA. Multimodal Sentiment Analysis. Multimodal Sentiment Analysis (MSA) in social media posts is an important direction of sentiment analysis. Many neural network approaches have been proposed to perform the coarse-grained MSA in the literature, which aim to detect the overall sentiment of each input social post (You et al., 2015, 2016; Luo et al., 2017; Xu et al., 2018; Yang et al., 2021b). Different 2150 Visual Aspect-Opinion Generation <bos> <msp> BART Decoder Positive Multimodal Sentiment Prediction BART Decoder <bos>  <mlm> Best  <eos> Masked Language Modeling concert Best concert Justin BART Decoder <bos> <aoe> Bieber <sep> Textual Aspect-Opinion Extraction 5 6 <sep> <eos> best 1 1 best guy <img> </img> Faster R-CNN Token Embedding Token Embedding  <bos> Best <mask> opening Justin <eos> BART Encoder 1 2 3 4 5 6 Position index: e <img> v 1 v zero v 3  v 36 e </img> e <bos> e Best e <mask> e opening e of e Justin e Bieber e ! of Textual Pre-training <bos> <aog> BART Decoder <bos><mrm> Masked Region Modeling BART Decoder <feat><zero><feat>  handsomeguy handsome <eos> Visual Pre-training Multimodal Pre-training Bieber 7 ! e <eos> Token Embedding Figure 1: Overview of our Vision-Language Pre-Training framework for MABSA from these studies, our work focuses on the fine-grained MABSA task, which aims to identify the sentiments towards all the aspects mentioned in each input social post. Multimodal Aspect-Based Sentiment Analysis. As an important sentiment analysis task, many approaches have been approached to tackle the three subtasks of MABSA, including Multimodal Aspect Term Extraction (Zhang et al., 2018; Yu et al., 2020b; Wu et al., 2020a,b; Sun et al., 2020; Zhang et al., 2021a), Multimodal Aspect Sentiment Classification (Xu et al., 2019; Yu et al., 2020a; Yang et al., 2021a; Khan and Fu, 2021) and Joint Multimodal Aspect-Sentiment Analysis (Ju et al., 2021). In this work, we aim to propose a general pre-training framework to improve the performance of all the three subtasks. Figure 1 shows the overview of our model architecture. The backbone of our model is BART (Lewis et al., 2020), which is a denoising autoencoder for sequence-to-sequence models. We extend BART to encode both textual and visual inputs, and decode pre-training and downstream tasks from different modalities. In the following subsections, we first introduce our feature extractor, and then illustrate the encoder and decoder of our model, followed by describing the details of three types of pre-training tasks and downstream MABSA tasks. Image Representation. Following many existing Vision-Language pre-training models (Chen et al., 2020b; Yu et al., 2021), we employ Faster R-CNN (Anderson et al., 2018) to extract visual features. Specifically, we adopt Faster R-CNN to extract all the candidate regions from an input image. We then only retain 36 regions with the highest confidence. Meanwhile, we also keep the semantic class distribution of each region, which will be used for the Masked Region Modeling task. For the retained regions, we use mean-pooled convolutional features processed by Faster R-CNN as our visual features. Let us use R = { r 1 , ..., r 36 } to denote the visual features, where r i  R 2048 refers to the visual feature of the i -th region. To be consistent with the text representation, we adopt a linear transformation layer to project visual features to d -dimensional vectors, denoted by V  R d  36 . Text Representation. For text input, we first tokenize the text and then feed tokens to the embedding matrix. The embeddings of text tokens are used as text features. Let us use E = { e 1 , ..., e T } to denote the token indexes of text inputs where T denotes the length of the input text, and W = { w 1 , ..., w T } to denote the embeddings of tokens. We employ a BART-based generative framework for both vision-language pre-training and downstream MABSA tasks. Encoder. The encoder of our model is a multilayer bidirectional Transformer. As shown in Figure 1, to distinguish inputs of different modalities, we follow Xing et al. (2021) by using (cid:104) img (cid:105) and (cid:104) /img (cid:105) to indicate the start and the end of visual features, and (cid:104) bos (cid:105) and (cid:104) eos (cid:105) to indicate the textual input. In the following part of the paper, we denote the concatenated multimodal input by X . decoder is unidirectional when generating outputs, while the encoder is bidirectional. Since all pretraining tasks share the same decoder, we insert two special tokens at the beginning of the inputs of the decoder to indicate different pre-training tasks. Following Yan et al. (2021), we insert a special token (cid:104) bos (cid:105) to indicate the beginning of generation, and then insert a task-specific special token to indicate the task type. Specifically, the special tokens for Masked Language Modeling, Textual Aspect-Opinion Extraction, Masked Region Modeling, Visual Aspect-Opinion Generation, and Multimodal Sentiment Prediction are (cid:104) bos (cid:105)(cid:104) mlm (cid:105) , (cid:104) bos (cid:105)(cid:104) aoe (cid:105) , (cid:104) bos (cid:105)(cid:104) mrm (cid:105) , (cid:104) bos (cid:105)(cid:104) aog (cid:105) , and (cid:104) bos (cid:105)(cid:104) msp (cid:105) , respectively. The dataset we use for pre-training is MVSA-Multi (Niu et al., 2016), which is widely used in Multimodal Twitter Sentiment Analysis (Yadav and Vishwakarma, 2020; Yang et al., 2021b). This dataset provides image-text input pairs and coarse-grained sentiments of image-text pairs. Statistics of the dataset are given in Table 2. With the dataset, we design three types of pretraining tasks, including textual, visual, and multimodal pre-training as follows. Textual Pre-training contains two tasks: a general Masked Language Modeling task to build alignment between textual and visual features and a task-specific Textual Aspect-Opinion Extraction task to extract aspects and opinions from text. Masked Language Modeling (MLM). In the MLM pre-training task, we use the same strategy as BERT (Devlin et al., 2019) by randomly masking the input text tokens with a probability of 15%. The goal of the MLM task is to generate the original text based on the image and the masked text, and thus the loss function of the MLM task is: LMLM =  EX  DT (cid:88) i =1 log P ( e i | e < i , X ) , (1) where e i and  X denote the i th token of the input text and the masked multimodal input, respectively. T is the length of input text. Textual Aspect-Opinion Extraction (AOE). The AOE task aims to extract aspect and opinion terms from the text. Since the MVSA-Multi dataset does not provide annotations for aspect and opinion terms, we resort to a pre-trained model for aspect extraction and a rule-based method for opinion extraction. Specifically, for aspect extraction, we employ the pre-trained model from a wellknown Named Entity Recognition (NER) tool for tweets (Ritter et al., 2011) to perform NER on each tweet in the dataset, and regard the recognized entities as aspect terms. For opinion extraction, we utilize a widely-used sentiment lexicon named Senti-WordNet (Esuli and Sebastiani, 2006) to obtain the dictionary of opinion words. Given each tweet, if its sub-sequences (i.e., words or phrases) match the words in the dictionary, we treat them as opinion terms. These extracted aspect and opinion terms are used as the supervision signal of our AOE task. With the textual aspect-opinion supervision, we follow Yan et al. (2021) by formulating the AOE task as an index generation task. Given the input text as the source sequence, the goal is to generate a target index sequence which consists of the start and end indexes of all aspect and opinion terms. Let us use Y = [ a s 1 , a e 1 , ..., a sM , a eM , (cid:104) sep (cid:105) , o s 1 , o e 1 , ..., o sN , o eN , (cid:104) eos (cid:105) ] to denote the target index sequence, where M and N are the number of aspect terms and opinion terms, a s , a e and o s , o e indicate the start and end indexes of an aspect term and an opinion term respectively, (cid:104) sep (cid:105) is used to separate aspect terms and opinion terms, and (cid:104) eos (cid:105) informs the end of extraction. For example, as shown in Figure 1, the extracted aspect and opinion terms are Justin Bieber and best respectively, and the target sequence is Y =[5 , 6 , (cid:104) sep (cid:105) , 1 , 1 , (cid:104) eos (cid:105) ] . For y t in the target sequence Y , it is either a position index or a special token (e.g., (cid:104) sep (cid:105) ). We use C = [ (cid:104) sep (cid:105) , (cid:104) eos (cid:105) ] to denote the set of special tokens, and C d as their embeddings. We assume that H e denotes the encoder output of the concatenated multimodal input, H e T denotes the textual part of H e , and H eV denotes the visual part of H e . The decoder takes the multimodal encoder output H e and the previous decoder output Y <t as inputs, and predicts the token probability 2152 distribution P ( y t ) as follows: h dt = Decoder ( H e ; Y <t ) , (2)  H eT = ( W + H eT ) / 2 , (3) P ( y t ) = Softmax ([  H eT ; C d ] h dt ) , (4) where W denotes the embeddings of input tokens. The loss function of the AOE task is as follows: LAOE =  EX  D O (cid:88) t =1 log P ( y t | Y < t , X ) , (5) where O = 2 M + 2 N + 2 is the length of Y and X denotes the multimodal input. Visual Pre-training contains two tasks: a general Masked Region Modeling task and a task-specific Visual Aspect-Opinion Generation task to capture subjective and objective information in the image. Masked Region Modeling (MRM). Following Xing et al. (2021), our MRM task aims to predict the semantic class distribution of the masked region. As shown in Figure 1, for the input of the encoder, we randomly mask image regions with a probability of 15%, which are replaced with zero vectors. For the input of the decoder, we first add two special tokens (cid:104) bos (cid:105)(cid:104) mrm (cid:105) , and then represent each masked region with (cid:104) zero (cid:105) and each remaining region with (cid:104) feat (cid:105) . After feeding the input to the decoder, an MLP classifier is stacked over the output of each (cid:104) zero (cid:105) to predict the semantic class distribution. Let us use p ( v z ) to denote the predicted class distribution of the z -th masked region, and q ( v z ) to denote the class distribution detected by Faster R-CNN. The loss function for MRM is to minimize the KL divergence of the two class distributions: LMRM = EX  DZ (cid:88) z =1 DKL ( q ( v z ) || p ( v z )) , (6) where Z is the number of masked regions. Visual Aspect-Opinion Generation (AOG). The AOG task aims to generate the aspect-opinion pair detected from the input image. In the field of Computer Vision, Borth et al. (2013) proposed to detect the visual sentiment concept, i.e., Adjective-Noun Pair (ANP) such as smiling man and beautiful landscape in the image. Since the nouns and adjectives of ANP respectively capture the fine-grained aspects and opinions in the image, we regard ANPs as visual aspect-opinion pairs. In order to detect the ANP of each input image, we adopt a pre-trained ANP detector DeepSentiBank 2 (Chen et al., 2014) to predict the class distribution over 2089 pre-defined ANPs. The ANP with the highest probability is selected as the supervision signal of our AOG task. For example, in Figure 1, the ANP detected from the input image is handsome guy , and we regard it as the supervision. With the visual aspect-opinion supervision, we formulate the AOG task as a sequence generation task. Specifically, let us use G = { g 1 , ..., g | G | } to denote the tokens of the target ANP and | G | to denote the number of ANP tokens. The decoder then takes the multimodal encoder output H e and the previous decoder output G <i as inputs, and predicts the token probability distribution P ( g i ) : h di = Decoder ( H e ; G <i ) , (7) P ( g i ) = Softmax ( ET h d i ) , (8) where E denotes the embedding matrix of all tokens in the vocabulary. The loss function of the AOG task is: LAOG =  EX  D | G | (cid:88) i =1 log P ( g i | g < i , X ) . Multimodal Pre-training has one task named Multimodal Sentiment Prediction (MSP). Different from the aforementioned pre-training tasks whose supervision signals only come from one modality, the supervision signals for MSP come from multimodality, which can enhance models to identify the subjective information in both language and vision and capture their rich alignments. Multimodal Sentiment Prediction (MSP). As the MVSA-Multi dataset provides the coarse-grained sentiment labels for all the text-image pairs, we use the sentiment labels as supervision signals of our MSP task. Formally, we model the MSP task as a classification task, where we first feed the two special tokens (cid:104) bos (cid:105)(cid:104) msp (cid:105) to the decoder and then predict the sentiment distribution P ( s ) as follows: h dmsp = Decoder ( H e ; E msp ) , (10) P ( s ) = Softmax ( MLP ( h dmsp )) , (11) where E msp is the embeddings of two special tokens. 2 https://github.com/stephen-pilli/DeepSentiBank 2153 final <img>  </img> <bos> <eos> Position index: Sergio 1 Ramos 2 10 UCL 9 Faster R-CNN Token Embedding Token Embedding BART Encoder BART Decoder <bos> <AESC> 1 2 POS 9 9 NEU <eos> Sergio Ramos POS UCL UCL NEU  Token Embedding Figure 2: An example of downstream task JMASA. We use the cross-entropy loss for the MSP task: LMSP =  EX  D log P ( s | X ) , (12) where s is the golden sentiment annotated in dataset. To optimize all the model parameters, we adopt the alternating optimization strategy to iteratively optimize our five pre-training tasks. The objective function is as follows: L =  1 LMLM +  2 LAOE +  3 LMRM +  4 LAOG +  5 LMSP (13) where  1 ,  2 ,  3 ,  4 , and  5 are tradeoff hyper-parameters to control the contribution of each task. We consider all the three subtasks in MABSA as our downstream tasks, including Joint Multimodal Aspect-Sentiment Analysis (JMASA), Multimodal Aspect Term Extraction (MATE), and Multimodal Aspect-oriented Sentiment Classification (MASC). We model these downstream tasks based on the same BART-based generative framework in vision-language pre-training, so that the downstream task can benefit more from pre-training during the fine-tuning stage. Following Yan et al. (2021), we formulate the outputs of the three subtasks as follows:  JMASA: Y = [ a s 1 , a e 1 , s 1 , ..., a si , a ei , s i , ... ] ,  MATE: Y = [ a s 1 , a e 1 , ..., a si , a ei , ... ] ,  MASC: Y = [ a s 1 , a e 1 , s 1 , ..., a s i , a e i , s i , ... ] , where a si , a ei , and s i inform the start index, end index, and sentiment of an aspect term in the text. The underlined tokens are given during inference. Similar to the AOE task in Section 3.3.1, we formulate all the subtasks as index generation tasks, and use Eqn. (2) to Eqn. (4) to generate the token distribution. The difference is that the special token set is modified as C = [ (cid:104) POS (cid:105) , (cid:104) NEU (cid:105) , (cid:104) NEG (cid:105) , (cid:104) EOS (cid:105) ] by adding the sentiment categories. Figure 2 shows an example for JMASA. Since the aspect-sentiment pairs are ( Sergio Ramos , Positive ) and ( UCL , Neutral ), its target sequence is [1 , 2 , (cid:104) POS (cid:105) , 9 , 9 , (cid:104) NEU (cid:105) , (cid:104) eos (cid:105) ] . Downstream datsets. We adopt two benchmark datasets annotated by Yu and Jiang (2019), namely TWITTER-2015 and TWITTER-2017 to evaluate our model. The statistics of the two datasets are shown in Table 3. Implementation Details. We employ BART-base (Lewis et al., 2020) as our framework. Specifi-cally, the encoder and decoder both have six layers and are initialized with BART-base parameters. We fix all the hyper-parameters after tuning them on the development set. The pre-training tasks were trained for 40 epochs and the downstream tasks were fine-tuned for 35 epochs. The batch sizes are set to 64 and 16, respectively. The learning rate is set to 5e-5. The hidden size of our model is set to 768, which is the same as BART. The tradeoff hyper-parameters  1 ,  2 ,  3 ,  4 , and  5 are all set to 1. Note that for the subtask MASC, different from Ju et al. (2021) evaluating on the correctly predicted aspects, we provide all the golden aspects to the decoder of our framework during the inference stage and evaluate on all the aspects. We implement all the models with PyTorch, and run experiments on a RTX3090 GPU. Evaluation Metrics. We evaluate our model over three subtasks of MABSA and adopt Micro-F1 score (F1), Precision (P) and Recall (R) as the evaluation metrics to measure the performance. For MASC, to fairly compare with other approaches, 2154 TWITTER-2015 TWITTER-2017 P R F1 P R F1 Text-based methods SPAN  53.7 53.9 53.8 59.6 61.7 60.6 D-GCN  58.3 58.8 59.4 64.2 64.1 64.1 BART 62.9 65.0 63.9 65.2 65.6 65.4 Multimodal methods UMT+TomBERT  58.4 61.3 59.8 62.3 62.4 62.4 OSCGA+TomBERT  61.7 63.4 62.5 63.4 64.0 63.7 OSCGA-collapse  63.1 63.7 63.2 63.5 63.5 63.5 RpBERT-collapse  49.3 46.9 48.0 57.0 55.4 56.2 JML  65.0 63.2 64.1 66.5 65.5 66.0 VLP-MABSA 65.1 68.3 66.6 66.9 69.2 68.0 Table 4: Results of different approaches for JMASA. In this section, we introduce four types of compared systems for different tasks. Approaches for Multimodal Aspect Term Extraction (MATE). 1) RAN (Wu et al., 2020a), which aligns text with object regions by a co-attention network. 2) UMT (Yu et al., 2020b), which uses Cross-Modal Transformer to fuse text and image representations for Multimodal Named Entity Recognition (MNER). 3) OSCGA (Wu et al., 2020b), another MNER approach using visual objects as image representations. 4) RpBERT (Sun et al., 2021), which uses a multitask training model for MNER and image-text relation detection. Approaches for Multimodal Aspect Sentiment Classification (MASC). 1) TomBERT (Yu and Jiang, 2019), which tackles the MASC task by employing BERT to capture intra-modality dynamics. 2) CapTrBERT (Khan and Fu, 2021), which translates the image to a caption as an auxiliary sentence for sentiment classification. Text-based approaches for Joint Aspect-Sentiment Analysis (JASA). 1) SPAN (Hu et al., 2019), which formulates the JASA task as a span prediction problem. 2) D-GCN (Chen et al., 2020a), which proposes a directional graph convolutional network to capture the correlation between words. 3) BART (Yan et al., 2021), which adapts the JASA task to BART by formulating it as an index generation problem. Multimodal approaches for Joint Multimodal Aspect-Sentiment Analysis (JMASA). 1) UMT+TomBERT and OSCGA+TomBERT , which are simple pipeline approaches by combining methods for subtasks mentioned above. 2) Methods TWITTER-2015 TWITTER-2017 P R F1 P R F1 RAN  80.5 81.5 81.0 90.7 90.7 90.0 UMT  77.8 81.7 79.7 86.7 86.8 86.7 OSCGA  81.7 82.1 81.9 90.2 90.7 90.4 JML-MATE  83.6 81.2 82.4 92.0 90.7 91.4 VLP-MABSA 83.6 87.9 85.7 90.8 92.6 91.7 Table 5: Results of different approaches for MATE. UMT-collapsed (Yu et al., 2020b), OSCGA-collapsed (Wu et al., 2020b) and RpBERT-collapsed (Sun et al., 2021), which model the JMASA task with collapsed labels such as B-POS and I-POS . 3) JML (Ju et al., 2021), which is a multi-task learning approach proposed recently with the auxiliary cross-modal relation detection task. In this section, we analyze the results of different approaches on three subtasks of MABSA. Results of JMASA. Table 4 shows the results of different methods for JMASA. As we can see from the table, BART achieves the best performance among text-based methods, and it even outperforms some multimodal methods, which proves the superiority of our base framework. For multimodal methods, JML achieves better performance than previous methods mainly due to its auxiliary task about relation detection between image and text. Among all the methods, VLP-MABSA which is the whole model with all the pre-training tasks consistently performs the best across two datasets. Specifically, it significantly outperforms the second best system JML with 2.5 and 2.0 absolute percentage points with respect to F1 on TWITTER-2015 and TWITTER-2017, respectively. This mainly benefits from our task-specific pre-training tasks, which identify aspects and opinions as well as their alignments across the two modalities. spectively. Similar to the trend on the JMASA subtask, we can clearly observe that our proposed approach VLP-MABSA generally achieves the best performance across the two datasets, except on the accuracy metric of TWITTER-2015. These observations further demonstrate the general effectiveness of our proposed pre-training approach. To explore the impact of each pre-training task, we perform a thorough ablation study over the full supervision setting which uses full training dataset and the weak supervision setting which only randomly chooses 200 training samples for fine-tuning. Impact of Each Pre-training Task. As we can see from Table 7, the performance generally improves with respect to most metrics when adding more pre-training tasks. To better analyze the effect of each pre-training task, we take the weak supervision experiments on TWITTER-2015 as an example. When only using MLM to pre-train our model, the performance only gets slight improvements. After adding the AOE task, the result of MATE gets a huge improvement of 9.44% on F1 . This shows that the AOE task greatly enhances our model's ability to recognize the aspect terms. When adding the MRM task, the performance gets slight improvements again. This reflects that general pre-training tasks (e.g., MLM and MRM) are not adequate for our model to tackle downstream tasks which need the model to understand the subjective and objective information from image and text. When adding the AOG task, the performance over three subtasks gets a moderate improvement, which proves the effectiveness of 200 400 600 800 1000 1200 1400 1600 the number of samples 35 40 45 50 55 60 65 70 F 1 TWITTER-2015 no-pretrainafter-pretrain 200 400 600 800 1000 1200 1400 1600 the number of samples 45 50 55 60 65 70 F 1 TWITTER-2017 no-pretrainafter-pretrain Figure 3: The effectiveness of pre-training when using different number of training samples for the downstream task. the AOG task. Finally, adding the MSP task significantly boosts the performance, especially on the MASC task. This shows that the MSP task can enhance our model's understanding of sentiment across language and image modalities. By combining all the pre-training tasks, our full model generally achieves the best results over most of the subtasks whether in both full supervision and weak supervision settings. Impact of pre-training when using different number of downstream training samples. To better understand the impact of pre-training, we compare the results with and without pre-training when adopting different number of samples for downstream training. We use the JMASA task as the example to observe the impact. As shown in Fig. 3, when the sample size is small, pre-training can bring a huge improvement. In contrast, when the sample size becomes larger, pre-training brings relatively small improvements. This further illustrates the robustness and the effectiveness of our pre-training approach, especially in low-resource scenarios. To further demonstrate the effectiveness of our approach, we present four test examples with predictions from different methods. The compared methods are BART , our framework using multimodal inputs without pre-training (denoted by MM ), and our framework using multimodal inputs with full pre-training (denoted by VLP ), respectively. As shown in Table 8, for example (a), both BART and MM extracted the wrong aspect term (i.e., the Faithfull Pearl Jam ) and gave the incorrect sentiment prediction towards Eddie . For example (b), BART only extracted one aspect term Madonna while MM identified an additional aspect term Demelza . However, the sentiment towards Madonna was wrongly predicted by MM . For example (c), BART only 2156 Image Text recognized part of the aspect term Colombia and MM wrongly predicted the sentiment towards Miss Colombia as Neutral . For example (d), both BART and MM failed to recognize the aspect term D-League . Among all the cases, our VLP model with full pre-training correctly extracted all the aspect terms and classified the sentiment , which shows the advantage of our generative framework and task-specific pre-training tasks. In this paper, we proposed a task-specific Vision-Language Pre-training framework for Multimodal Aspect-Based Sentiment Analysis (VLP-MABSA). We further designed three kinds of pre-training tasks from the language, vision, and multi-modal modalities, respectively. Experimental results show that our proposed approach generally outperforms the state-of-the-art methods for three subtasks of MABSA. Our work is a first step towards a unified Vision-Language Pre-training framework for MABSA. In the future, we plan to apply our pretraining approach on a larger dataset and consider the relation between image and text in our pretraining framework. We hope this work can potentially bring new insights and perspectives to the research of MABSA. The authors would like to thank the anonymous reviewers for their insightful comments. This work was supported by the Natural Science Foundation of China (62076133 and 62006117), and the Natural Science Foundation of Jiangsu Province for Young Scholars (BK20200463) and Distinguished Young Scholars (BK20200018).
abstain abstain abstain abstain objective method result abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain objective abstain objective objective objective method result method objective method result objective other other other other method other other other other other other other other other other objective abstain abstain abstain abstain abstain method method abstain method method method method abstain abstain abstain method method abstain abstain abstain method abstain abstain method other method abstain abstain method abstain abstain method other abstain abstain abstain method method abstain abstain method method abstain method abstain abstain method method other abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain method method abstain abstain abstain abstain abstain method other other method method abstain method abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain other abstain abstain result result abstain result result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result objective method objective objective method objective other other