It is a common practice for recent works in vision language cross-modal reasoning to adopt a binary or multi-choice classification formulation taking as input a set of source image(s) and textual query. In this work, we take a sober look at such an unconditional formulation in the sense that no prior knowledge is specified with respect to the source image(s). Inspired by the designs of both visual commonsense reasoning and natural language inference tasks, we propose a new task termed  P remise-based M ulti-modal R easoning (PMR) where a textual premise is the background presumption on each source image. The PMR dataset contains 15,360 manually annotated samples which are created by a multi-phase crowd-sourcing process. With selected high-quality movie screen-shots and human-curated premise templates from 6 pre-defined categories, we ask crowd-source workers to write one true hypothesis and three distractors (4 choices) given the premise and image through a cross-check procedure. Besides, we generate adversarial samples to alleviate the annotation artifacts and double the size of PMR. We benchmark various state-of-the-art (pretrained) multi-modal inference models on PMR and conduct comprehensive experimental analyses to showcase the utility of our dataset. Cross-modal reasoning between image and text has been recognized as a fundamental and longstanding task in both academia and industry, which has recently attracted intensive attention from both natural language processing and computer vision communities (Su et al., 2019; Yu et al., 2020; Chen et al., 2020). Researchers try to teach machines to perceive, understand and reason with both visual and textual clues, which mimics human cognitive process (Lake et al., 2016). The canonical form for the cross-modal reasoning tasks usually take the source image(s) as the input and request the inference model to perform a multi-choice classification according to the specified textual query, including visual question answering (Antol et al., 2015; Goyal et al., 2017), visual commonsense reasoning (Zellers et al., 2018), visual entailment (Xie et al., 2019) and image-text grounding (Suhr et al., 2017, 2019), etc. We argue the default cross-modal setting is static or unconditional in the sense that no prior presumption or belief is attached to the input images (Ren et al., 2021). The lack of prior presumption would be insignificant while answering factoid questions according to the images, e.g. how many dogs are lying on the grass? (Suhr et al., 2019) or is the bowl to the right of the green apple? (Hud-son and Manning, 2019). However it may not be the case for more sophisticated cross-modal reasoning that involves human-like cognition and commonsense, e.g. what may happen to [per-sonA]?, what will [personA] say to [personB] or what will [personA] do to [personB]. The answers to these questions are ambiguous by merely looking at the source images, instead we offer an extra textual premise, which serves as the prior belief to the source images, such as to [personA], nothing could be more somber, [personA] is the son of [personB]. We name the proposed task as  P remise-based M ulti-modal R easoning (PMR) 1 . In PMR, the inference model should be able to reason with both textual (from the premise) and visual (from images) clues. The motivations for PMR are two-folds. 1) From the social psychology perspective, the design of PMR is inspired by the ABC Theory (Ellis, 1995) , which represents a widely accepted framework for how one's feelings and behavioral patterns are connected. As illustrated in Figure 1, the theory 1 The dataset and baseline models can be found in https: //2030nlp.github.io/PMR/ . claims that human emotions and resulting behavior do not come directly from the events, but from the interpretations (textual premise) we make of those events (depicted in the source images). 2) From the methodology perspective, PMR gains insights from both visual commonsense reasoning (Zellers et al., 2018; Lei et al., 2020; Park et al., 2020) and natural language inference (Dagan et al., 2006; Bowman and Zhu, 2019; Xie et al., 2019) tasks. From the view of commonsense reasoning, the textual premise serves as the prior belief to the image as mentioned before. Meanwhile in the world of natural language inference, the input image can be viewed as the supplementary evidence that supports the textual premise-hypothesis classification. The proposed PMR could be readily seen as a meaningful extension on the joint cross-modal entailment and commonsense reasoning with both visual and textual clues. Given the input premise-image pair, the practitioners are requested to choose the only true hypothetical action from four candidates in PMR. All four hypothetical actions are written by crowd-sourced workers in a multi-phase cross-check annotation pipeline. We encourage the annotators to write challenging distracting actions that are logically correct with respect to either the premise or the image, but are contradicted with the joint premise-image pair while combining the visual and textual clues. In order to succeed on PMR, the reasoning model should excel at both language grounding among premise, image and candidate actions, and cross-modal understanding on commonsense and logical inference. We establish multiple competitive cross-modal pretrained models as baselines. We hope the proposed PMR could pave the way for the conditional cross-modal commonsense and logical reasoning that requires a human-like cognitive process. We describe the proposed PMR task with an example in Figure 2 I&II. Given a source image and a textual premise, the inference model should perceive and understand the image in combination with the premise so as to choose the exclusive correct action among the four hypothetical candidates. The premise would serve as the background knowledge or domain-specific commonsense for the given image. In the running example, the model should be able to recognize what [person2] wears from the image and infer whether [person4] would give his seat to [person2] under the premise [per-son4] is very friendly. The corrected answer is C' according to the visual and textual clues. In total, we collect about 15k instances for PMR. We list the statistics for PMR in Table 2. 3 Data Collection We collect PMR dataset in a multi-step crowd-sourcing pipeline, including 1) image and premise creation, 2) annotator recruitment and instruction, 3) cross-checking annotation. The source images of PMR are selected from the image pool 2 in the VCR (Zellers et al., 2018) dataset. The entire image pool contains 110k high-quality movie screenshots. To make the images fit better with PMR, we screened out those which have low brightness, more than five people, or more than 15 tags, and finally got 29,987 images. We also kept the object anchor information in VCR along with the source images, which was identified using Mask-RCNN (He et al., 2017). As there are so many possible choices while writing premise for an image, to facilitate analysis of model performance by premise type, we constrained the premise to six categories and manually wrote templates for each category (see Appendix D). As Figure 4 shows, the six categories are personality, identity, emotion, relationship, environment and antecedent. To complete templates, we create multiple word sets and fill corresponding words into the slots randomly. And finally, we obtained 30,759 predefined premises, which would be presented to annotators for selection. 2 The images originate from Fandango MovieClips ( https://youtube.com/user/movieclips ) and Large Scale Movie Description Challenge (Rohrbach et al., 2017). Actions: A. [person2] with a cap wants to sit by the window but [person4] refuses him without any hesitance. B. [person2] in a white coat wants to sit by the window but [person4] refuses him without any hesitance. C. [person2] with a cap wants to sit by the window and [person4] exchanges his seat with him generously. D. [person2] in a white coat wants to sit by the window and [person4] exchanges his seat with him generously. Through job descriptions posted on the online forums, we invite more than a hundred applicants who hold a bachelor of arts or higher degree to attend the online pre-annotating instruction and qualification test. In the pre-annotating instruction, we organized a two-hour training session, which covers the basics and goals of PMR, to better instruct the annotators, we extracted the text from VCR as the reference actions. Specifically, we filtered out questions about the following behavior of persons and retrieved the correct answers as the reference actions. After the pre-annotating instruction, we conducted a qualification test by asking each participant to write the true and false hypothetical actions for 10 sampled image-premise pairs, the authors manually assess the annotation quality and eventually select 61 annotators whose educational background covers literature, linguistics, sociology, etc. Among them, 19 annotators who got higher scores in the qualification test were recognized as advanced workers and assigned to the distractor collection phase (phase 2 in Figure 3), while the other annotators were assigned to the hypothetical action creation phase (phase 1 in Figure 3). We divide the annotation process into 3 phases. In phase 1, workers are presented with an image with bounding boxes on it, six predefined premises of different categories and six reference actions. Firstly, they are supposed to choose an appropriate premise. Workers can adjust the person tags to meet with the given image, but they have to assure that the modified premise still belongs to one of the six categories. Secondly, they ought to write two hypothetical actions, which describe what will happen next. Among them, Action-True contains image information and meets the chosen premise. In contrast, Action-False contains image information but does not meet the chosen premise. Thirdly, to make it easier for replacing certain words in phase 2, they need to enclose the words with curly brackets that mention the information of the image in both written actions. annotation in phase 1 conforms to labeling instruction. They can drop it while meeting poor-quality one, and once they accept it, they need to write another two distractors corresponding to Action-True and Action-False respectively. We instruct the annotators to write challenging distractors that are logically correct with respect to either the premise or the image, but contradict with the joint premise-image pair while combining the visual and textual clues. In phase 3, to ensure the quality of the examiners' work, we sample 10% HITs in phase 2 to check if the annotation meets our standard. This work is performed by the authors of this paper, and we feedback to examiners timely. Post-Processing In order to ground objects from images with entities in text, we follow the VCR which substitutes all the tags both in premise and answers with the index in objects list. Besides, since crowd workers can conditionally modify the given premises, which results in the missing of labels of category, we calculate the BLEU (Papineni et al., 2002) score between premise and each template from six categories and labeled it with the most likely category with the highest BLEU score. Annotation Cost We drop all the annotations that are rejected in phase3 and obtained 15,360 items after 60-day work. (See Appendix E for detailed pricing strategy) We split the total dataset into train, valid and test set in 8:1:1. Adversarial Samples Generation To alleviate the possible bias introduced by annotation artifacts, inspired by Zellers et al., for each sample in the crowd-sourcing set of PMR, we pick three actions from other samples but similar to the premise as negatives and generate adversarial samples with the same scale. See Appendix C for detailed methods. To set up the benchmark, we introduce multiple text-only and pretrained multi-modal baseline models as well as human performance on PMR. Text-only Baselines We begin with a blind setting, where only the text is given without access to the image or bounding boxes annotations. In this setting, the models have to choose the correct answers with only textual clues. Specifically we finetune the BERT (Devlin et al., 2018) model in the action-only and premise+action scenarios. (See Appendix 6 for training details) Multi-modal Baselines Throughout the visual reasoning tasks, cross-modal pretrained models have achieved state-of-the-art performance (Zellers et al., 2018; Goyal et al., 2017; Hudson and Manning, 2019). We introduce three powerful pretrained models and test their performance on PMR:  VL-BERT (Su et al., 2019) A dual-stream pretrained model, is extended from BERT by ap-935 pending visual feature embedding along with the subsequent sentences. ERNIE-VL (Yu et al., 2020) is a knowledge-enhanced approach to learning joint representations of vision and language, which introduced structure knowledge with scene graph prediction tasks while pretraining. UNITER (Chen et al., 2020) is also an extension of BERT to the visual domain. It is trained with a conditional masking strategy that allows the model to learn an informative representation of one modality conditioned on the other. Before feeding into BERT, we concatenate the question and each answer as a sequence and replace detection tags with object names in it. As for person detection tags, we substitute them with gender-neutral names to bridge the gap between PMR and pretraining corpus (Zellers et al., 2018). In terms of multimodal baselines, while an annotation is processed as above, we regard region features as visual tokens to be concatenated along with text sequence. Region features are extracted with Faster RCNN. Different from ERNIE-VL and UNITER which freeze Faster RCNN all the time, VL-BERT updates it while pretraining and finetun-ing. Are Premises Critical for Models to Predict? As is shown in Table 2, E-L achieves the highest accuracy while training and testing on the original split, but if trained with only actions as text inputs, the accuracy of it decreases sharply by 23.3%. More details can be found in Table 3. Premises help E-L correct the prediction of Action-False with the ratio from 30.5% to 8.2%. Is Vision Modality Useful? As for the performance of text-only models and multi-modal models in Table 2, considering the information from images improves the accuracy by 10%-14.7%. Besides, Distractor1, as generated by substituting objects in Action-True, contradicts with corresponding images, which can be hard negatives while the visual signal is imperceptible. In Table 3, trained on the original split, E-L predicts Distractor1 with a ratio of 10.7%, nearly half of the prediction of Ori. Does Crowd-sourcing Introduce Annotation Artifacts to Dataset? Table 2 shows that BERT with only actions outperforms random baseline by 14.3 on accuracy, indicating that textual bias is introduced by manual annotations, which is inevitable due to fixed patterns of thinking during labeling but is tolerable. Adding premises as background presumption, BERT can discover the relation between premises and actions with the performance improved a lot. We also conduct a quantitative analysis of bias in data. To check whether labels are significantly associated with certain words, we compute the point-936 Word green red n't fight throw hit No. (Ori.) 1 2 7 11 18 23 PMI (Ori.) 1.620 1.562 1.322 1.318 1.305 1.301 No. (Mix) 1 2 26 16 52 155 PMI (Mix) 1.488 1.408 1.231 1.242 1.212 1.172 Table 4: PMI calculated for the words in Ori. wise mutual information (PMI) (Gururangan et al., 2018). As each sample in PMR has four choices, we consider the premise concatenated respectively with four answers as four binary-class items with label False if the answer is incorrect, and compute PMI between words and False label. As Table 4 shows, among the top 30 words, green and red are the top two with the highest PMI, they are commonly used by crowd workers to modify the objects and generate distractors mismatched with images. Other words such as leave, fight and hit, crowd workers tend to write universal negative containing these words so as to pass the qualification check, which also results in high PMI of these words. Does Training with Adversarial Samples Help to Relieve the Above Issue? Yes, firstly in Table 4, we can find that the PMIs of words decrease, indicating that adversarial samples help to balance the correlation of words and labels. Secondly, Table 2 shows that there is a great gap in the performance of multi-modal models between testing on OT and AT while training on the original set, which also testifies the existence of bias in the original set. Nevertheless, after training on mixed data, multi-modal models are able to achieve high accuracy both on OT and AT, suggesting that adversarial samples are helpful to improve models' robustness. Besides, We can notice that E-L may be misled by the fewer incorrect samples in the adversarial split, and it results in a decline of 2.8% on OT, which can be seen as a compromise between generalization and performance. Thirdly, we find that another huge gap of 39.4% on OT between BERT trained on the original set and mixed set, and it indicates that the adversarial samples help to change the data distribution and alleviate the bias from the text. From Figure 4, we can find that Relationship and Personality account for nearly two thirds of the samples, while the other four categories hold the rest one third, which indicates that the understanding of interpersonal relationships and the correlation between character's personalities and behaviors are test points of PMR. In terms of the reason why there is such a distribution, we maintain that this is because Relationship and Personality are more likely to conform with the instruction of annotation. As presented in Section 2, we expect premises as a supplement of images, so the information implied in premises should not be repeated or contradictory with the content of the image. Choosing the other four premises is more likely to violate the above requirements. For instance, people's mood tends to explicitly show through expression; the dressing of the characters can reflect their identity and occupation. We manually classified 100 samples from the validation set into nine types, which is presented in Table 5. As we can see, the types of substitution are varied ,and most of them are related to PERSON supertype, which reflects the emphasis on the characters' behavior. In terms of the understanding of images, the levels of difficulty vary among different types. For instance, in APPEARANCE, CLOTHING, OBJECTS and COLOR, the words involved in the replacement process are relatively simple, such as long->short, white->red, knives->guns, which are likely to appear as labels in the datasets of object detection. Thus, systems with pre-trained objects detection models are expected to have an excellent performance on the four types. On the contrary, as for the types like EMOTION, BODY STATE and LOCATION, the texts substituted tend to be abstract and mostly are phrases consisting of more than one word. Consequently, these types are more 937 Supertype Type Action-True Distractor1 PERSON Appearance (8.0%) [person2] with long hair will fall on the [couch1] and have a sleep immediately . challenging, and in order to succeed on them, the model is supposed to excel at in-depth image understanding and fine-grained grounding. Comparing the difference before and after the substitution, we may notice that the substitution, within the scope of several consecutive words, can be regarded as a method of disturbance, and correspondingly, the distractors served as adversarial choices are beneficial to counter the possibility of models exploiting unimodal priors. We conduct a qualitative analysis shown in Figure 5, where the first two samples are correctly predicted by ERNIE-VIL. For each of them, premise 1 is the original. To explore the influence of different premises on the model predictions with the same image as background, we write premise 2 for the test, with which the model is expected to choose Action-False as the correct answer. The result shows that ERNIE-VIL predicted correctly both with premise 1 and 2, which indicates the model can distinguish the possible person's behaviors resulting from the two opposite premises. But for case 3, the model makes an incorrect prediction with Action-False chosen, suggesting that the performance is still unsatisfactory due to limited commonsense knowledge. Besides, even if the simple objects substitution can be easily detected by comparing the text with the images directly, case 4 shows that the model is not able to determine the relative position of people, which indicates that a powerful understanding capacity of images is needed to be added. Multimodal Commonsense Reasoning A series of tasks and datasets have been proposed for cross-modal commonsense reasoning. VCR (Zellers et al., 2018) is the most related work to the proposed PMR. It requires machines to understand a image and answer a multi-choice question. Specifically, questions are like what is going to happen next', infer the relationship between [personA] and [personB]' and why is [personA] smiling', as well as the rationale why the answer is true. In the video understanding regime, VLEP (Lei et al., 2020) is a dataset for future event prediction from videos. Given a video with aligned dialogue, and two possible future events, the AI system is required to choose the more likely event from two provided options. In the world of cross-modal com-938 Premises:1.[person3]isindifferent.2.[person3]ishelpful. monsense graph, VisualCOMET (Park et al., 2020) is a repository of Visual Commonsense Graphs that consists of 1.4 million textual descriptions of visual commonsense inferences. The proposed PMR is different from the above-mentioned dataset as we request the models to perceive and understand the source image with the supervision from a specified textual premise, and reason conditionally with joint visual and textual clues. Natural Language Inference Early methods for textual NLI mainly relied on conventional, feature-based methods trained from small-scale datasets (Dagan et al., 2006). The release of larger datasets, such as SNLI (Bowman et al., 2015), MultiNLI (Williams et al., 2018), made neural network methods feasible. In the field of cross-modal entailment, the SNLI-VE (Xie et al., 2019) dataset casts the source image as the visual premise, and asks the inference model judge whether the specified textual hypothesis entails or contradicts with the visual premise. The proposed PMR can also be viewed as a cross-modal entailment task, however difference from SNLI-VE, both premise and hypothetical actions are textual, and the corresponding source images can be viewed as the supplementary evidence while judging the entailment labels. Besides, the existing textual and visual-related NLI tasks are formulated in a three-way classification, while the proposed PMR is model as a 4-choice classification task. We propose a premise-based cross-modal reasoning (PMR) task, in which the textual premise serves as the presumptions to the source images. With selected images and human-curated premise templates, the collected dataset is formulated as a 4-choice classification task, in which we ask human annotators to write one true hypothetical action and three distracting actions. The PMR task gains insights from both the visual commonsense reasoning and the natural language inference tasks. We hope the proposed dataset and task can pave the 939 way for human-like cognitive reasoning from both visual and textual clues. CoRR , abs/1803.02324. Kaiming He, Georgia Gkioxari, Piotr Dollr, and Ross Girshick. 2017. Mask r-cnn. In Proceedings of the IEEE international conference on computer vision , pages 29612969. Drew A Hudson and Christopher D Manning. 2019. Gqa: A new dataset for real-world visual reasoning and compositional question answering. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 67006709. Brenden M Lake, Tomer D Ullman, Joshua B Tenenbaum, and Samuel J Gershman. 2016. Building Machines That Learn and Think Like People. arXiv . Jie Lei, Licheng Yu, Tamara L Berg, and Mohit Bansal. 2020. What is more likely to happen next? video-and-language future event prediction. arXiv preprint arXiv:2010.07999 . Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th annual meeting of the Association for Computational Linguistics , pages 311318. Jae Sung Park, Chandra Bhagavatula, Roozbeh Mot-taghi, Ali Farhadi, and Yejin Choi. 2020. Visualcomet: Reasoning about the dynamic context of a still image. In European Conference on Computer Vision , pages 508524. Springer. Anna Rohrbach, Atousa Torabi, Marcus Rohrbach, Niket Tandon, Christopher Pal, Hugo Larochelle, Aaron Courville, and Bernt Schiele. 2017. Movie description. International Journal of Computer Vision , 123(1):94120. Weijie Su, Xizhou Zhu, Yue Cao, Bin Li, Lewei Lu, Furu Wei, and Jifeng Dai. 2019. Vl-bert: Pre-training of generic visual-linguistic representations. arXiv preprint arXiv:1908.08530 . Alane Suhr, Mike Lewis, James Yeh, and Yoav Artzi. 2017. A corpus of natural language for visual reasoning. In Proceedings of the 55th Annual Meeting of the Association for Computational Linguistics (Vol-ume 2: Short Papers) , pages 217223, Vancouver, Canada. Association for Computational Linguistics. Alane Suhr, Stephanie Zhou, Ally Zhang, Iris Zhang, Huajun Bai, and Yoav Artzi. 2019. A corpus for reasoning about natural language grounded in photographs. In Proceedings of the 57th Annual Meeting of the Association for Computational Linguistics , pages 64186428, Florence, Italy. Association for Computational Linguistics. This paper is supported by the National Key Research and Development Program of China 2020AAA0106700 and NSFC project U19A2065.
abstain method objective abstain method method method abstain abstain other abstain abstain abstain abstain objective abstain abstain abstain other result abstain abstain abstain abstain abstain abstain method abstain method objective abstain abstain abstain abstain abstain method abstain method abstain abstain abstain method abstain abstain method abstain other abstain abstain abstain abstain abstain objective method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method method method method abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain method abstain abstain method abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain other other other other other other other other objective other other other other other objective method abstain objective other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other other