Recently, opinion summarization, which is the generation of a summary from multiple reviews, has been conducted in a self-supervised manner by considering a sampled review as a pseudo summary. However, non-text data such as image and metadata related to reviews have been considered less often. To use the abundant information contained in non-text data, we propose a self-supervised multimodal opinion summarization framework called MultimodalSum. Our framework obtains a representation of each modality using a separate encoder for each modality, and the text decoder generates a summary. To resolve the inherent heterogeneity of multimodal data, we propose a multimodal training pipeline. We first pretrain the text encoderdecoder based solely on text modality data. Subsequently, we pretrain the non-text modality encoders by considering the pretrained text decoder as a pivot for the homogeneous representation of multimodal data. Finally, to fuse multimodal representations, we train the entire framework in an end-to-end manner. We demonstrate the superiority of MultimodalSum by conducting experiments on Yelp and Amazon datasets. Opinion summarization is the task of automatically generating summaries from multiple documents containing users' thoughts on businesses or products. This summarization of users' opinions can provide information that helps other users with their decision-making on consumption. Unlike conventional single-document or multiple-document summarization, where we can obtain the prevalent annotated summaries (Nallapati et al., 2016; See et al., 2017; Paulus et al., 2018; Liu et al., 2018; Liu and Lapata, 2019; Perez-Beltrachini et al., 2019), opinion summarization is challenging; it is difficult to find summarized opinions of users. Accordingly, studies used an unsupervised approach for opinion summarization (Ku et al., 2006; Paul et al., 2010; Carenini et al., 2013; Ganesan et al., 2010; Gerani et al., 2014). Recent studies (Brazinskas and Titov, 2020; Amplayo and Lapata, 2020; Elsahar et al., 2021) used a self-supervised learning framework that creates a synthetic pair of source reviews and a pseudo summary by sampling a review text from a training corpus and considering it as a pseudo summary, as in Figure 1a. Users' opinions are based on their perception of a specific entity and perceptions originate from various characteristics of the entity; therefore, opinion summarization can use such characteristics. For instance, Yelp provides users food or menu images and various metadata about restaurants, as in Figure 1b. This non-text information influences the review text generation process of users (Truong and Lauw, 2019). Therefore, using this additional information can help in opinion summarization, especially under unsupervised settings (Su et al., 2019; Huang et al., 2020). Furthermore, the training process of generating a review text (a pseudo summary) based on the images and metadata for self-supervised learning is consistent with the actual process of writing a review text by a user. This study proposes a self-supervised multimodal opinion summarization framework called MultimodalSum by extending the existing self-supervised opinion summarization framework, as shown in Figure 1. Our framework receives source reviews, images, and a table on the specific business or product as input and generates a pseudo summary as output. Note that images and the table are not aligned with an individual review in the framework, but they correspond to the specific entity. We adopt the encoderdecoder framework and build multiple encoders representing each input modality. However, a fundamental challenge lies in the heterogeneous data of various modalities (Baltrusaitis et al., 2018). To address this challenge, we propose a multimodal training pipeline. The pipeline regards the text modality as a pivot modality. Therefore, we pretrain the text modality encoder and decoder for a specific business or product via the self-supervised opinion summarization framework. Subsequently, we pretrain modality encoders for images and a table to generate review texts belonging to the same business or product using the pretrained text decoder. When pretraining the non-text modality encoders, the pretrained text decoder is frozen so that the image and table modality encoders obtain homogeneous representations with the pretrained text encoder. Finally, after pretraining input modalities, we train the entire model in an end-to-end manner to combine multimodal information. Our contributions can be summarized as follows:  this study is the first work on self-supervised multimodal opinion summarization;  we propose a multimodal training pipeline to resolve the heterogeneity between input modalities;  we verify the effectiveness of our model framework and model training pipeline through various experiments on Yelp and Amazon datasets. Generally, opinion summarization has been conducted in an unsupervised manner, which can be divided into extractive and abstractive approaches. The extractive approach selects the most meaningful texts from input opinion documents, and the abstractive approach generates summarized texts that are not shown in the input documents. Most previous works on unsupervised opinion summarization have focused on extractive approaches. Clustering-based approaches (Carenini et al., 2006; Ku et al., 2006; Paul et al., 2010; Angelidis and Lapata, 2018) were used to cluster opinions regarding the same aspect and extract the text representing each cluster. Graph-based approaches (Erkan and Radev, 2004; Mihalcea and Tarau, 2004; Zheng and Lapata, 2019) were used to construct a graphwhere nodes were sentences, and edges were similarities between sentencesand extract the sentences based on their centrality. Although some abstractive approaches were not based on neural networks (Ganesan et al., 2010; Gerani et al., 2014; Di Fabbrizio et al., 2014), neural network-based approaches have been gaining attention recently. Chu and Liu (2019) generated an abstractive summary from a denoising autoencoder-based model. More recent abstractive approaches have focused on self-supervised learning. Brazinskas and Titov (2020) randomly selected N review texts for each entity and constructed N synthetic pairs by sequentially regarding one review text as a pseudo summary and the others as source reviews. Amplayo and Lapata (2020) sampled a review text as a pseudo summary and generated various noisy versions of it as source reviews. Elsahar et al. (2021) selected review texts similar to the sampled pseudo summary as source reviews, based on TF-IDF cosine similarity. We construct synthetic pairs based on Brazinskas and Titov (2020) and extend the self-supervised opinion summarization to a multimodal version. Multimodal text summarization has been mainly studied in a supervised manner. Text summaries were created by using other modality data as additional input (Li et al., 2018, 2020a), and some studies provided not only a text summary but also other modality information as output (Zhu et al., 2018; Chen and Zhuge, 2018; Zhu et al., 2020; Li et al., 2020b; Fu et al., 2020). Furthermore, most studies summarized a single sentence or document. Although Li et al. (2020a) summarized multiple documents, they used non-subjective documents. Our study is the first unsupervised multimodal text summarization work that summarizes multiple subjective documents. summary from multimodal data. Following existing self-supervised opinion summarization studies, we consider a review text selected from an entire review corpus as a pseudo summary. We extend the formulation of Brazinskas and Titov (2020) to a multimodal version. Let R = { r 1 , r 2 , ..., r N } denote the set of reviews about an entity (e.g., a business or product). Each review, r j , consists of review text, d j , and review rating, s j , that represents the overall sentiment of the review text. We denote images uploaded by a user or provided by a company for the entity as I = { i 1 , i 2 , ..., i M } and a table containing abundant metadata about the entity as T . Here, T consists of several fields, and each field contains its own name and value. We set j -th review text d j as the pseudo summary and let it be generated from R  j , I , and T , where R  j = { r 1 , ..., r j  1 , r j +1 , ..., r N } denotes source reviews. To help the model summarize what stands out overall in the review corpus, we calculate the loss for all N cases of selecting d j from R , and train the model using the average loss. During testing, we generate a summary from R , I , and T . The proposed model framework, MultimodalSum, is designed with an encoderdecoder structure, as in Figure 1b. To address the heterogeneity of three input modalities, we configure each modality encoder to effectively process data in each modality. We set a text decoder to generate summary text by synthesizing encoded representations from the three modality encoders. Details are described in the following subsections. Our text encoder and decoder are based on BART (Lewis et al., 2020). BART is a Transformer (Vaswani et al., 2017) encoderdecoder pretrained model that is particularly effective when fine-tuned for text generation and has high summarization performance. Furthermore, because the pseudo summary of self-supervised multimodal opinion summarization is an individual review text ( d j ), we determine that pretraining BART based on a denoising autoencoder is suitable for our framework. Therefore, we further pretrain BART using the entire training review corpus (Gururangan et al., 2020). Our text encoder obtains e D -dimensional encoded text representations h text from D  j and the text decoder generates d j from h text as follows: h text = BART enc ( D  j ) , d j = BART dec ( h text ) , where D  j = { d 1 , ..., d j  1 , d j +1 , ..., d N } denotes the set of review texts from R  j . Each review text consists of l D tokens and h text  R ( N  1)  l D  e D . We use a convolutional neural network specialized in analyzing visual imagery. In particular, we use ImageNet pretrained ResNet101 (He et al., 2016), which is widely used as a backbone network. We add an additional linear layer in place of the image classification layer to match feature distribution and dimensionality with text modality representations. Our image encoder obtains encoded image representations h img from I as follows: h img = ResNet101( I ) W img , where W img  R e I  e D denotes the additional linear weights. h img obtains RM  l I  e D , where l I represents the size of the flattened image feature map obtained from ResNet101. To effectively encode metadata, we design our table encoder based on the framework of data-to-text research (Puduppully et al., 2019). The input to our table encoder T is a series of field-name and field-value pairs. Each field gets e T -dimensional representations through a multilayer perceptron after concatenating the representations of field-name and field-value. The encoded table representations h table is obtained by stacking each field representation into F and adding a linear layer as follows: f k = ReLU([ n k ; v k ] W f + b f ) , h table = F W table , where n and v denote e T -dimensional representations of field name and value, respectively, and W f  R 2 e T  e T , b f  R e T are parameters. By stacking l T field representations, we obtain F  R 1  l T  e T . The additional linear weights W table  R e T  e D play the same role as in the image encoder, and h table  R 1  l T  e D . To effectively train the model framework, we set a model training pipeline, which consists of three steps, as in Figure 2. The first step is text modality pretraining, in which a model learns unsupervised summarization capabilities using only text modality data. Next, during the pretraining for other modalities, an encoder for each modality is trained using the text modality decoder learned in the previous step as a pivot. The main purpose of this step is that other modalities have representations whose distribution is similar to that of the text modality. In the last step, the entire model framework is trained using all the modality data. Details of each step can be found in the next subsections. In this step, we pretrain the text encoder and decoder for self-supervised opinion summarization. As this was an important step for unsupervised multimodal neural machine translation (Su et al., 2019), we apply it to our framework. For the set of reviews about an entity R , we train the model to generate a pseudo summary d j from source reviews R  j for all N cases as follows: loss = (cid:80) Nj =1 log p ( d j | R  j ) . The text encoder obtains h text  R ( N  1)  l D  e D from D  j , and the text decoder aggregates the encoded representations of N  1 review texts to generate d j . We model the aggregation of multiple encoded representations in the multi-head self-attention layer of the text decoder. To generate a pseudo summary that covers the overall contents of source reviews, we simply average the N  1 single-head attention results for each encoded representation ( R l D  e D ) at each head (Elsahar et al., 2021). The limitation of the self-supervised opinion summarization is that training and inference tasks are different. The model learns a review generation task using a review text as a pseudo summary; however, the model needs to perform a summary generation task at inference. To close this gap, we Figure 3: Text decoder input representations. use a rating deviation between the source reviews and the target as an additional input feature of the text decoder, inspired by Brazinskas et al. (2020). We define the average ratings of the source reviews minus the rating of the target as the rating deviation: sd j = (cid:80) Ni (cid:54) = j s i / ( N  1)  s j . We use sd j to help generate a pseudo summary d j during training and set it as 0 to generate a summary with average semantic of input reviews during inference. To reflect the rating deviation, we modify the way in which a Transformer creates input embeddings, as in Figure 3. We create deviation embeddings with the same dimensionality as token embeddings and add sd j  deviation embeddings to the token embeddings in the same way as positional embeddings. Our methods to close the gap between training and inference tasks do not require additional modeling or training in comparison with previous works. We achieve noising and denoising effects by simply using rating deviation embeddings without variational inference in Brazinskas and Titov (2020). Furthermore, the information that the rating deviation is 0 plays the role of an input prompt for inference, without the need to train a separate clas-sifier for selecting control tokens to be used as input prompts (Elsahar et al., 2021). 5.2 Other Modalities Pretraining As the main modality for summarization is the text modality, we pretrain the image and table encoders by pivoting the text modality. Although the data of the three modalities are heterogeneous, each encoder should be trained to obtain homogeneous representations. We achieve this by using the pretrained text decoder as a pivot. We train the image encoder and the table encoder along with the text decoder to generate a review text of the entity to which images or metadata belong: I or T  d j  R . The image and table encoders obtain h img and h table from I and T , respectively, and the text decoder generates d j from h img or h table . Note that we aggregate M encoded representations of h img as in the text modality pretraining, and the weights of the text decoder are made constant. I or T corresponds to all N reviews, and this means that I or T has multiple references. We convert a multiple-reference setting to a single-reference setting to match the model output with the text modality pretraining. We simply create N single reference pairs from each entity and shuffle pairs from all entities to construct the training dataset (Zheng et al., 2018). As the text decoder was trained for generating a review text from text encoded representations, the image and table encoders are bound to produce similar representations with the text encoder to generate the same review text. In this way, we can maximize the ability to extract the information necessary for generating the review text. We train the entire multimodal framework from the pretrained encoders and decoder. The encoder of each modality obtains an encoded representation for each modality, and the text decoder generates the pseudo summary d j from multimodal encoded representations h text , h img , and h table . To fuse multimodal representations, we aim to meet three requirements. First, the text modality, which is the main modality, is primarily used. Second, the model works even if images or metadata are not available. Third, the model makes the most of the legacy from pretraining. To fulfill the requirements, multi-modality fusion is applied to the multi-head self-attention layer of the text decoder. The text decoder obtains the attention result for each modality at each layer. We fuse the attention results for multiple modalities as follows: ma fused = ma text +  (cid:12) ma img +  (cid:12) ma table , Yelp Train Dev Test #businesses 50,113 100 100 #reviews/business 8 8 8 #summaries/business 1* 1 1 #max images 10 10 10 #max fields 47 47 47 Amazon Train Dev Test #products 60,935 28 32 #reviews/product 8 8 8 #summaries/product 1* 3 3 #max images 1 1 1 #max fields 5+128 5+128 5+128 Table 1: Data statistics; 1 * in Train column indicates that it is a pseudo summary. where ma text , ma img , and ma table denote each modality attention result from h text , h img , and h table , respectively. (cid:12) symbolizes element-wise multiplication and e D -dimensional multimodal gates  and  are calculated as follows:  =  ([ ma text ; ma img ] W  ) and  =  ([ ma text ; ma table ] W  ) . Note that  or  obtains the zero vector when images or metadata do not exist. It is common to use sigmoid as an activation function  . However, it can lead to confusion in the text decoder pretrained using only the text source. Because the values of W are initialized at approximately 0 , the values of  and  are initialized at approximately 0 . 5 when sigmoid is used. To initialize the gate values at approximately 0 , we use ReLU(tanh( x )) as  ( x ) . This enables the continuous use of text information, and images or metadata are used selectively. To evaluate the effectiveness of the model framework and training pipeline on datasets with different domains and characteristics, we performed experiments on two review datasets: Yelp Dataset Challenge 1 and Amazon product reviews (He and McAuley, 2016). The Yelp dataset provides reviews based on personal experiences for a specific business. It also provides numerous images (e.g., food and drinks) uploaded by the users. Note that the maximum number of images, M , was set to 10 based on the 90 th percentile. In addition, the dataset contains abundant metadata of businesses according to the characteristics of each business. On the contrary, the Amazon dataset provides reviews with more objective and specific details about a particular product. It contains a sin-1 https://www.yelp.com/dataset gle image provided by the supplier, and provides relatively limited metadata for the product. For evaluation, we used the data used in previous research (Chu and Liu, 2019; Brazinskas and Titov, 2020). The data were generated by Amazon Mechanical Turk workers who summarized 8 input review texts. Therefore, we set N to 9 so that a pseudo summary is generated from 8 source reviews during training. For the Amazon dataset, 3 summaries are given per product. Simple data statistics are shown in Table 1, and other details can be found in Appendix A.1. All the models 2 were implemented with Py-Torch (Paszke et al., 2019), and we used the Transformers library from Hugging Face (Wolf et al., 2020) as the backbone skeleton. Our text encoder and decoder were initialized using BART-Large and further pretrained using the training review corpus with the same objective as BART. e D , e I , and e T were all set to 1,024. We trained the entire models using the Adam optimizer (Kingma and Ba, 2014) with a linear learning rate decay on NVIDIA V100s. We decayed the model weights with 0 . 1 . For each training pipeline, we set different batch sizes, epochs, learning rates, and warmup steps according to the amount of learning required at each step. We used label smoothing with 0 . 1 and set the maximum norm of gradients as 1 for other modalities pretraining and multiple-modalities training. During testing, we used beam search with early stopping and discarded hypotheses that contain twice the same trigram. Different beam size, length penalty, and max length were set for Yelp and Amazon. The best hyperparameter values and other details are described in Appendix A.2. We compared our model to extractive and abstractive opinion summarization models. For extractive models, we used some simple baseline models (Brazinskas and Titov, 2020). Clustroid selects one review that gets the highest ROUGE-L score with the other reviews of an entity. Lead constructs a summary by extracting and concatenating the lead sentences from all review texts of an entity. Random simply selects one random review from an entity. LexRank (Erkan and Radev, 2004) is an extractive model that selects the most salient 2 Our code is available at https://bit.ly/3bR4yod sentences based on graph centrality. For abstractive models, we used non-neural and neural models. Opinosis (Ganesan et al., 2010) is a non-neural model that uses a graph-based summarizer based on token-level redundancy. MeanSum (Chu and Liu, 2019) is a neural model that is based on a denoising-autoencoder and generates a summary from mean representations of source reviews. We also used three self-supervised abstractive models. DenoiseSum (Amplayo and Lapata, 2020) generates a summary by denoising source reviews. Copycat (Brazinskas and Titov, 2020) uses a hierarchical variational autoencoder model and generates a summary from mean latent codes of the source reviews. Self & Control (Elsahar et al., 2021) generates a summary from Transformer models and uses some control tokens as additional inputs to the text decoder. We evaluated our model framework and model training pipeline. In particular, we evaluated the summarization quality compared to other baseline models in terms of automatic and human evaluation, and conducted ablation studies. To evaluate the summarization quality, we used two automatic measures: ROUGE{ 1,2,L } (Lin, 2004) and BERT-score (Zhang et al., 2020). The former is a token-level measure for comparing 1 , 2 , and adaptive L-gram matching tokens, and the latter is a document-level measure using pretrained BERT (Devlin et al., 2019). Contrary to ROUGE-score, which is based on exact matching between n-gram words, BERT-score is based on the semantic similarity between word embeddings that reflect the context of the document through BERT. It is approved that BERT-score is more robust to adversarial examples and correlates better with human judgments compared to other measures for machine translation and image captioning. We hypothesize that BERT-score is strong in opinion summarization as well, and BERT-score would complement ROUGE-score. The results for opinion summarization on two datasets are shown in Table 2. MultimodalSum showed superior results compared with extractive and abstractive baselines for both token-level and document-level measures. From the results, we Yelp Amazon Model R-1 R-2 R-LFBERT R-1 R-2 R-LFBERTE x t r ac ti v e Clustroid (Brazinskas and Titov, 2020) 26.28 3.48 15.36 85.8 29.27 4.41 17.78 86.4 Lead (Brazinskas and Titov, 2020) 26.34 3.72 13.86 85.1 30.32 5.85 15.96 85.8 Random (Brazinskas and Titov, 2020) 23.04 2.44 13.44 85.1 28.93 4.58 16.76 86.0 LexRank (Erkan and Radev, 2004) 24.90 2.76 14.28 85.4 29.46 5.53 17.74 86.4 A b s t r ac ti v e Opinosis (Ganesan et al., 2010) 20.62 2.18 12.55 84.4 24.04 3.69 14.58 85.2 MeanSum (Chu and Liu, 2019) 28.86 3.66 15.91 86.5 29.20 4.70 18.15 DenoiseSum (Amplayo and Lapata, 2020) 30.14 4.99 17.65 85.9 --Copycat (Brazinskas and Titov, 2020) 29.47 5.26 18.09 87.4 31.97 5.81 20.16 87.7 Self & Control (Elsahar et al., 2021) 32.76 8.65 18.82 86.8 --MultimodalSum (ours) 33.00 6.63 19.84 * 87.7 * 34.19 * 7.05 * 20.81 87.9 Table 2: Opinion summarization results on Yelp and Amazon datasets. Gold Wow, where to start? Some of the best sweet foods I've ever had. I wasn't sure what to try, so I tried a few things, and oh my goodness they were delicious. That's not all though, they serve drinks too so I got a latte and that was good too. There is a lot of variety here to choose from that'll make any sweet tooth salivate. Definitely a good place! If you're looking for a sweet tooth this is the place to go if you want a delicious dessert. I had the lemon meringue pie and it was delicious. The only thing I didn't like was that I could eat half of it, but it was a little pricey for what you get. Self & Control If you're a fan of the Matos Buffet, this is a good place to visit. If you want to have a sweet tooth this is the place to be. The desserts are delicious and they have a good variety of desserts to choose from. The only thing I don't like about this place is that you have to wait in line for a long time to get in. Other than that, you can't really go wrong with any of the desserts in the buffet. The drinks are good and the desserts are yummy too. They also have desserts that are not too sweet. I'm not a huge fan of buffets, but this is one of my favorite buffets. MultimodalSum This is a cute little bakery located in the M resort. I had the chocolate croissant and it was very good. The croissants were soft and moist and the filling was delicious. I also had a chocolate chip cookie which was also good. I would definitely recommend this place if you are in the area. In particular, our model achieved state-of-the-art results on the Amazon dataset and outperformed the comparable model by a large margin in the R-L representing the ROUGE scores on the Yelp dataset. Although Self & Control showed high R-2 score, we attributed their score to the inferred N -gram control tokens used as additional inputs to the text decoder. Sample summaries on the Yelp dataset are shown in Table 3. They were generated from source reviews on Baby Cakes bakery. Copycat misused sweet tooth and generated lemon mernigue pie that was not mentioned in the source reviews. Self & Control generated a summary about a buffet by totally misunderstanding one sentence from source reviews: If you love the desserts in Studio B Buffet in the M Hotel but don't want to wait in the massive buffet line or even eat in the buffet, Baby Cakes in the M Hotel is really nice fix. Furthermore, Matos Buffet is a non-existent word. On the contrary, MultimodalSum generated a good summary with a rich description of chocolate croissants. Although chocolate chip cookie was not found in the source reviews, our model generated it from cookie images. Note that the term can be found in other reviews that were not used as source reviews. Additional sample summaries on two datasets are shown in Appendix A.5. To evaluate the quality of summarization based on human criteria, we conducted a user study. We assessed the quality of summaries using Best-Worst Scaling (BWS; Louviere et al. (2015)). BWS is known to produce more reliable results than raking scales (Kiritchenko and Mohammad, 2017) and is widely used in self-supervised opinion summarization studies. We recruited 10 NLP experts and asked each participant to choose one best and one worst summary from four summaries for three criteria. For each participant's response, the best model received + 1 , the worst model received 1 , and the rest of the models received 0 scores. The final scores were obtained by averaging the scores of all the responses from all participants. For Overall criterion, Self & Control, Copycat, MultimodalSum, and gold summaries scored 0 . 527 , 0 . 113 , + 0 . 260 , and + 0 . 380 on the Yelp dataset, respectively. MultimodalSum showed superior performance in human evaluation as well as automatic evaluation. We note that human judgments correlate better with BERT-score than ROUGE-score. Self & Control achieved a very low human evaluation score despite its high ROUGE-score in automatic evaluation. We analyzed the summaries of Self & Control, and we found several flaws such as redundant words, ungrammatical expressions, and factual hallucinations. It generated a non-existent word by combining several subwords. It was particularly noticeable when a proper noun was generated. Furthermore, Self & Control generated an implausible sentence by copying some words from source reviews. From the results, we conclude that both automatic evaluation and human evaluation performances should be supported to be a good summarization model and BERT-score can complement ROUGE-score in automatic evaluation. Details on human evaluation and full results can be found in Appendix A.3. To analyze the effects of multimodal data on opinion summarization, we analyzed the multimodal gate. Since the multimodal gate is a e D dimensional vector, we averaged it by a scalar value. Furthermore, as multimodal gates exist for each layer of the text decoder, we averaged them to measure the overall influence of a table or images when generating each token in the decoder. An example of aggregated multimodal gates is shown in Figure 4. It shows the table and images used for generating a summary text, and the multimodal gates for a part of the generated summary are expressed as heatmaps. As we intended, table and image information was selectively used to generate a specific word in the summary. The aggregated value of the table was relatively high for generating Red Lobster, which is the name of the restaurants. It was relatively high for images, when generating food that is depicted in two images. Another characteristic of the result is that aggregated values of the table were higher than those of the image: mean values for the table and image in the entire test data were 0.103 and 0.045, respectively. This implies that table information is more used when creating a summary, and this observation is valid in that the table contains a large amount of metadata. Note that the values displayed on the heatmaps are small by and large, as they were aggregated from e D -dimensional vector. For ablation studies, we analyzed the effectiveness of our model framework and model training pipeline in Table 4. To analyze the model framework, we first compared the summarization quality with four versions of unimodal model framework, as in the first block of Table 4. BART denotes the model framework in Figure 1a, whose weights are the weights of BART-Large. It represents the lower bound of our model framework without any training. BART-Review denotes the model framework whose weights are from further pretrained BART using the entire training review corpus. UnimodalSum refers to the results of the text modality pretraining, and we classified it into two frameworks according to the use of the rating deviation. Surprisingly, using only BART achieved comparable or better results than many extractive and abstractive baselines in Table 2. Furthermore, further pretraining using the review corpus brought performance improvements. Qualitatively, BART with further pretraining generated more diverse words and rich expressions from the review corpus. This proved our assumption that denoising autoencoder-based pretraining helps in self-supervised multimodal opinion summarization. Based on the BART-Review, UnimodalSum achieved superior results. Furthermore, the use of rating deviation improved the quality of summarization. We conclude that learning to generate reviews based on wide ranges of rating deviations including 0 during training helps to generate a better summary of the average semantics of the input reviews. To analyze the effect of other modalities in our model framework, we compared the summarization quality with three versions of multimodal model frameworks, as in the second block of Table 4. We removed the image or table modality from MultimodalSum to analyze the contribution of each modality. Results showed that both modalities improved the summarization quality compared with UnimodalSum, and they brought additional improvements when used altogether. This indicates that using non-text information helps in self-supervised opinion summarization. As expected, the utility of the table modality was higher than that of the image modality. The image modality contains detailed information not revealed in the table modality (e.g., appearance of food, in-side/outside mood of business, design of product, and color/texture of product). However, the information is unorganized to the extent that the utility of the image modality depends on the capacity of the image encoder to extract unorganized information. Although MultimodalSum used a representative image encoder because our study is the first work on multimodal opinion summarization, we expect that the utility of the image modality will be greater if unorganized information can be extracted effectively from the image using advanced image encoders. For analyzing the model training pipeline, we removed text modality or/and other modalities pretraining from the pipeline. By removing each of them, the performance of MultimodalSum declined, and removing all of the pretraining steps caused an additional performance drop. Although Multi-Models R-L BART 14.85 BART-Review 15.23 UnimodalSum w/o rating deviation 18.98 UnimodalSum w/ rating deviation 19.40 MultimodalSum 19.84 w/o image modality 19.54 w/o table modality 19.47 w/o other modalities pretraining 19.26 w/o text modality pretraining 19.24 w/o all modalities pretraining 19.14 Table 4: Ablation studies on the Yelp dataset. modalSum without other modalities pretraining has the capability of text summarization, it showed low summarization performance at the beginning of the training due to the heterogeneity of the three modality representations. However, MultimodalSum without text modality pretraining, whose image and table encoders were pretrained using BART-Review as a pivot, showed stable performance from the beginning, but the performance did not improve significantly. From the results, we conclude that both text modality and other modalities pretraining help the training of multimodal framework. For the other modalities pretraining, we conducted a further analysis in the Appendix A.4. We proposed the first self-supervised multimodal opinion summarization framework. Our framework can reflect text, images, and metadata together as an extension of the existing self-supervised opinion summarization framework. To resolve the heterogeneity of multimodal data, we also proposed a multimodal training pipeline. We verified the effectiveness of our multimodal framework and training pipeline with various experiments on real review datasets. Self-supervised multimodal opinion summarization can be used in various ways in the future, such as providing a multimodal summary or enabling a multimodal retrieval. By retrieving reviews related to a specific image or metadata, controlled opinion summarization will be possible. We thank the anonymous reviewers for their insightful comments and suggestions.
abstain abstain objective result objective objective method method objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain method abstain objective abstain method method abstain method objective other other other other other other other other other other other abstain other other other other objective abstain method abstain abstain abstain method abstain method abstain method abstain abstain method abstain abstain abstain abstain method other abstain method method method abstain abstain method abstain abstain other abstain abstain method abstain abstain abstain abstain abstain method method method abstain method abstain abstain abstain abstain abstain method method method abstain abstain abstain method abstain abstain method abstain method abstain method method abstain method method abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain other method abstain method abstain abstain method method method method abstain method method abstain method abstain abstain method method abstain abstain abstain other method abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain objective abstain objective result abstain abstain other