Multimodal summarization becomes increasingly significant as it is the basis for question answering, Web search, and many other downstream tasks. However, its learning materials have been lacking a holistic organization by integrating resources from various modalities, thereby lagging behind the research progress of this field. In this study, we present a full-scale multimodal dataset comprehensively gathering documents, summaries, images, captions, videos, audios, transcripts, and titles in English from CNN and Daily Mail. To our best knowledge, this is the first collection that spans all modalities and nearly comprises all types of materials available in this community. In addition, we devise a baseline model based on the novel dataset, which employs a newly proposed Jump-Attention mechanism based on transcripts. The experimental results validate the important assistance role of the external information for multimodal summarization. Multimodal summarization refines salient information from one or more modalities, including text, image, audio, and video ones (Evangelopou-los et al., 2013; Li et al., 2017). Given the rapid dissemination of multimedia data over the Internet, multimodal summarization has been widely explored in recent years. Meanwhile, some multimodal datasets (Li et al., 2017; Zhu et al., 2018; Sanabria et al., 2018; Li et al., 2020a) have been introduced to advance the development of this research field. However, a majority of them are restricted in scale and too oriented, such as being less than one hundred examples or merely containing Chinese texts. Moreover, the materials from different modalities are rarely collected across the board, especially videos and their accompanying materials that possess abundant external information for multimodal comprehension and fusion. In this work, we introduce a full-scale Multimodal Article and Video Summarization (MM-AVS) dataset 1 with documents, summaries, images, captions, videos, audios, transcripts, and titles in English. The significance of MM-AVS for the multimodal summarization community includes but not limited to: 1) MM-AVS is a large-scale multimodal collection compared with existing video containing dataset and its generation codes 1 has been released, which can be readily extended for existing and future multimodal summarization approaches; 2) MM-AVS is collected from CNN 2 and Daily Mail 3 , which makes it available to more researchers due to English-based and comparable with the popular text-based CNN/Daily Mail corpus; and 3) MM-AVS firstly collects nearly all types of materials from all modalities, inclusively with videos, audios, transcripts, images, captions, and titles that are rarely assembled. In addition, we implement a general multimodal summarization baseline based on transcripts for multimodal summarization on MM-AVS. This method employs a Jump-Attention mechanism to align features between text and video. Further, we use the multi-task learning to simultaneously optimize document and video summarizations. Evaluations on MM-AVS illustrate the benefits of external information such as videos and transcripts for multimodal summarization without alignment. Multi-modal summarization generates a condensed multimedia summary from multi-modal materials, such as texts, images, and videos. For instance, UzZaman et al. (2011) introduced an idea of illustrating complex sentences as multimodal summaries by combining pictures, structures, and sim-1 https://github.com/xiyan524/MM-AVS . Libovick`y et al. (2018) and Palaskar et al. (2019) studied abstractive text summarization for open-domain videos. Li et al. (2017) constructed MMS dataset and developed an extractive multi-modal summarization method that automatically generated a textual summary based on a topic-related set of documents, images, audios, and videos. Zhu et al. (2018, 2020) combined image selection and output to alleviate the modality bias based on the MSMO dataset. Chen and Zhuge (2018) extended Daily Mail with images and captions to E-Daily Mail dataset and employed a hierarchical encoder-decoder model to align sentences and images. Recently, an aspect-aware model and a large-scale Chinese e-commerce product summarization dataset EC-product were introduced to incorporate visual information for e-commerce product summaries (Li et al., 2020a). The above mentioned datasets are rarely constructed comprehensively, which ignore the abundant visual information underlying in videos. The only video-containing work is restricted in scale, which hampers its use for deep-learning based methods. In this study, we will build a full-scale multimodal dataset to address these issues. To facilitate a straightforward comparison for the multimodal summarization approaches with the text-based ones, MM-AVS extends CNN/DM collections to multimodalities. Each example of MM-AVS contains a document accompanying with multi-sentence summary, title, images, captions, videos, and their corresponding audios and transcripts. Table 1 compares MM-AVS with the representative multimodal summarization benchmarks. MM-AVS contains documents and abstractive summaries as most of the benchmarks including, while it ex-Daily Mail CNN Avg. tends visual information that most existing benchmarks ignore (such as MSMO(Zhu et al., 2018), MMSS(Li et al., 2018), E-DailyMail(Chen and Zhuge, 2018), and EC-product(Li et al., 2020b)). MMS(Li et al., 2017) and How2(Sanabria et al., 2018) also take videos into account; however, MMS only contains 50 examples that are too limited for deep learning and How2 excludes documents, which are the most critical materials for summarization. MM-AVS also keeps image captions for deep descriptions of images as well as document titles for the topic extraction. Further, MM-AVS contains extractive labels for training convenience. In the manner of providing abundant multimodal information, MM-AVS is applicable for existing and future multimodal research in different learning tasks. The concrete statistics of MM-AVS are shown in Table 2 4 , incorporating textual and visual modules: Textual module. Following (Nallapati et al., 2016), we have crawled all the summary bullets of each story in the original order to obtain a multi-sentence reference, where each bullet is treated as a sentence. Given that the reference is an abstractive summary written by humans, we construct the label of each sentence as (Nallapati et al., 2017) does. 4 The data scale is determined by its accompanied videos, considering this modality is more space-consuming. The data acquirability code in the project github mentioned above can be used for extension. Sentences in the document are selected to maximize the ROUGE (Lin, 2004) score with respect to the gold summary by a greedy approach. As for the document and title, we keep their original formats as shown in the websites. Visual module. To enrich visual information for multimodal summarization, we collect images and videos for each example. Image caption is preserved to assist further explorations such as feature extraction and alignment to documents. Given long videos, we separate the audios and extract the transcripts 5 to alleviate the pre-process pressure for large-scale or online learning. We utilize the hierarchical bi-directional long short term memory (BiLSTM) (Nallapati et al., 2017) based on word and sentence levels to read tokens and induce a representation for each sentence denoted as s i . Each sentence in a transcript is denoted as t j . In terms of videos, we employ ResNet (He et al., 2016) for feature extraction and BiLSTM to model the sequential pattern in video frames. Each image is represented as m k . Given that the transcript extracted from a video shares the same modality with a document and accurately aligns with a video, we take it as a bridge to deepen the relationship between two modalities. We apply the jump attention based on transcripts to assist modality alignment, which focuses on transcripts to video images and then on documents to 5 We use IBM Watson Speech for the text service https://www.ibm.com/watson/services/speech-to-text/. transcript attention context. The video-aware context cd 2 v i is denoted as cd 2 v i = NT (cid:88) j =1 NM (cid:88) k =1 b j i d kj m k , (1) where NT and NM are the lengths of transcripts and image frames. b ji and d kj are the attention weights and can be calculated as follows (taking d kj for illustration): d kj =  ( VT ( q j (cid:12) r k + q j + r k )) , (2) where V is the training parameter, q j and r k are the feature mappings of each modality that are calculated as q j = tanh ( W m m k + b m ) and r k = tanh ( W t t j + b t ) . The jump attention can be reversed to obtain an article context vector for video summarization. Given that modalities may be not accurately aligned, we employ late+ fusion by fusing unimodal decisions. Inspired by (Liu et al., 2018), we induce noise filters to eliminate noises as F ( W s f ( s i ) , W c g ( cd 2 v i )) , where the filters W s and W c are calculated as follows: W s = [1  g ( cd 2 v i )]  , W c = [1  f ( s i )]  , (3) where  is a smoothed coefficient for penalty intensity, and f (  ) , g (  ) , and F (  ) are feedforward networks. We employ the multi-task training to enhance summarization. The loss function is the weight mix of R-1 R-2 R-L text-only 39.11 16.42 28.56 +video frames 40.86 17.48 30.23 +transcripts 41.26 17.95 30.98 Table 3: Summarizations based on the materials of documents, videos, and transcripts. L =  ts L ts +  vs ( R div + R rep ) , L ts =  1 NSNS (cid:88) n =1 [ y n log  y n + (1  y n ) log where L ts is the training loss for extractive summarization, y n and  y n represent the true and predicted labels, and  ts and  vs are balance parameters. Following (Zhou et al., 2018), we use unsupervised learning by reinforcement learning methods for video summarization whose loss can be separated into the diversity reward R div (measuring frames dissimilarity) and the representativeness reward R rep (measuring similarity between summary and video) as follows: R div = 1 |M| ( |M|  1) (cid:88) j M (cid:88) j (cid:48) M , j (cid:48) (cid:54) = j d (cid:0) m j , m j (cid:48) (cid:1) , R rep = exp    1 NMNM (cid:88) j =1 min j (cid:48) M (cid:13)(cid:13) m j  m j (cid:48) (cid:13)(cid:13) 2   , (5) where M is the set of the selected video frames and d (  ) is the dissimilarity function. We conduct experiments on the MM-AVS dataset and evaluate the performance by ROUGE (Lin, 2004). R-1, R-2, and R-L respectively represent ROUGE-1, ROUGE-2, and ROUGE-L F1-scores, which are widely used to calculate the n-grams overlapping between decoded summaries and references. Videos, audios, or transcripts are less concerned than documents and images, as revealed in Table 1. Accordingly, the multimodal corpus assembling all of them has been absent so far, till MM-AVS is built in this study. To verify the importance of these materials for multimodal summarization, we test a text-only baseline and its two extensions. As for the baseline, we construct a hierarchical framework that concentrates on word and sentence levels with a feedforward classification layer. Its two extensions respectively take videos and transcripts for additional considerations. As shown in Table 3, both videos and transcripts can contribute to improving multimodal summa-rizaions by fusing documents. This validates that the external information complementary for texts can facilitate capturing the core ideas of documents and inducing high-quality summaries. To further investigate the nature of transcripts, we compare them with documents and references. As shown in Table 4, the video transcripts in MM-AVS are distinct from documents with low overlaps, indicating that they are not repeating documents but provide useful assistant information. While Table 4 also illustrates that the transcripts are lowly correlated with references, suggesting that transcripts can assist summary generation but are not enough for the final excellent summaries. The document , video , and document with video summarization results on 200 groups of MM-AVS examples are scored by five computer science graduates in terms of their informativeness (Inform) and satisfaction (Satis). Each summary is scored from 1 to 5, where a higher score denotes more informative or satisfied, and we record the average scores in Table 5. It shows that the summaries induced via documents and videos are more close to human comprehensions, which is in accord with the observations in Section 5.1, verifying the importance of external information such as videos for excellent summaries. In Proceedings of the 2017 Conference on Empirical Methods in Natural Language Processing , pages 10921102. Jindrich Libovick`y, Shruti Palaskar, Spandana Gella, and Florian Metze. 2018. Multimodal abstractive summarization of opendomain videos. In Proceedings of the Workshop on Visually Grounded Interaction and Language (ViGIL). NIPS. Chin-Yew Lin. 2004. Rouge: A package for automatic evaluation of summaries. Text Summarization Branches Out , 8. Kuan Liu, Yanen Li, Ning Xu, and Prem Natarajan. 2018. Learn to combine modalities in multimodal deep learning. arXiv preprint arXiv:1805.11730 . Naushad UzZaman, Jeffrey P. Bigham, and James F. Allen. 2011. Multimodal summarization of complex sentences. In ACM International Conference on Intelligent User Interfaces , pages 4352. In this work, we contribute a full-scale dataset for multimodal summarization, which extensively assembles documents, summaries, images, captions, videos, audios, transcripts, and titles. A novel multimodal summarization framework is proposed based on this dataset to be taken as a baseline for the future research in this community. We sincerely thank Wei Liu for his constructive collaboration during development of this paper and Nayu Liu for the helpful discussions.
abstain abstain method objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain other other other other other other other other other method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result method abstain abstain abstain result abstain method abstain method method abstain method abstain method other abstain abstain other abstain method method method abstain abstain abstain method abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other other other other other other other other other other other other other other other other other other objective abstain other