This paper presents a novel crowd-sourced resource for multimodal discourse: our resource characterizes inferences in imagetext contexts in the domain of cooking recipes in the form of coherence relations. Like previous corpora annotating discourse structure between text arguments, such as the Penn Discourse Treebank, our new corpus aids in establishing a better understanding of natural communication and common-sense reasoning, while our findings have implications for a wide range of applications, such as understanding and generation of multimodal documents. Sometimes a picture is worth the proverbial thousand words; sometimes a few well-chosen words are far more effective than a picture   Feiner and McKeown (1991). Modeling how visual and linguistic information can jointly contribute to coherent and effective communication is a longstanding open problem with implications across cognitive science. As Feiner and McKeown (1991) already observe, it is particularly important for automating the understanding and generation of textimage presentations. Theoretical models have suggested that images and text fit together into integrated presentations via coherence relations that are analogous to those that connect text spans in discourse; see Alikhani and Stone (2018a) and Section 2. This paper follows up this theoretical perspective through systematic corpus investigation. We are inspired by research on text discourse, which has led to large-scale corpora with information about discourse structure and discourse semantics. The Penn Discourse Treebank (PDTB) is one of the most well-known examples (Milt-sakaki et al., 2004; Prasad et al., 2008). However, although multimodal corpora increasingly include discourse relations between linguistic and nonlinguistic contributions, particularly for utterances and other events in dialogue (Cuayhuitl et al., 2015; Hunter et al., 2015), to date there has existed no dataset describing the coherence of textimage presentations. In this paper, we describe the construction of an annotated corpus that fills this gap, and report initial analyses of the communicative inferences that connect text and accompanying images in this corpus. As we describe in Section 2, our approach asks annotators to identify the presence of specific inferences linking text and images, rather than to use a taxonomy of coherence relations. This enables us to deal with the distinctive discourse contributions of photographic imagery. We describe our data collection process in Section 3, showing that our annotation scheme allows us to get reliable labels by crowdsourcing. We present analyses in Section 4 that show that our annotation highlights a range of cases where text and images work together in distinctive and theoretically challenging ways, and discuss the implications of our work for the understanding and generation of multimodal documents. We conclude in Section 5 with a number of problems for future research. We begin with an example to motivate our approach and clarify its relationship to previous work. Figure 1 shows two steps in an online recipe for a ravioli casserole from the RecipeQA data set (Yagcioglu et al., 2018). The image of Figure 1a shows a moment towards the end of carrying out the covering action of the accompanying text; that of Figure 1b shows one instance of the result of the spooning actions of the text. Cognitive scientists have argued that such images are much like text contributions in the way their interpretation connects to the broader discourse. In particular, inferences analogous to those used to interpret text seem to be necessary with such images to recognize their spatio-temporal perspective (Cumming et al., 2017), the objects they depict (Abusch, 2013), and their place in the arc of narrative progression (McCloud, 1993; Cohn, 2013). In fact, such inferences seem to be a general feature of multimodal communication, applying also in the coherent relationships of utterance to co-speech gesture (Lascarides and Stone, 2009) or the coherent relationships of elements in diagrams (Alikhani and Stone, 2018b; Hiippala and Orekhova, 2018). In empirical analyses of text corpora, researchers in projects such as the Penn Discourse Treebank (Miltsakaki et al., 2004; Prasad et al., 2008) have been successful at documenting such effects by annotating discourse structure and discourse semantics via coherence relations. We would like to apply a similar strategy to text image documents like that shown in Figure 1. However, existing discourse annotation guidelines depend on the distinctive ways that coherence is signaled in text. In text, we find syntactic devices such as structural parallelism, semantic devices such as negation, and pragmatic elements such as discourse connectives, all of which can help annotators to recognize coherence relations in text. Images lack such features. At the same time, characterizing the communicative role of imagery, particularly photographic imagery, involves a special problem: distinguishing the content that the author specifically aimed to depict from merely incidental details that happen to appear in the scene (Stone and Stojnic, 2015). Thus, rather than start from a taxonomy of discourse relations like that used in PDTB, we characterize the different kinds of inferential relationships involved in interpreting imagery separately. To characterize temporal relationships between imagery and text, we ask if the image gives information about the preparation, execution or results of the accompanying step. To characterize the logical relationship of imagery to text, we ask if the image shows one of several actions described in the text, and if it depicts an action that needs to be repeated. To characterize the significance of incidental detail, we ask a range of further questions (some relevant specifically to our domain of instructions), asking about what the image depicts from the text, what it leaves out from the text, and what it adds to the text. Our approach is designed to elicit judgments that crowd workers can provide quickly and reliably. This approach allows us to highlight a number of common patterns that we can think of as prototypical coherence relations between images and text. Figure 1a, for example, instantiates a natural Depiction relation: the image shows the action described in the text in progress; the mechanics of the action are fully visible in the image, but the significant details in the imagery are all reported in the text as well. Our approach also lets us recognize more sophisticated inferential relationships, like the fact that Figure 1b shows an Example:Result of the accompanying instruction. Many of the relationships that emerge from our annotation effort involve newly-identified features of textimage presentations that deserve further investigation: particularly, the use of loosely-related imagery to provide background and motivation for a multimodal presentation as a whole, and depictions of action that seem simultaneously to give key information about the context, manner and result of an action. Work on text has found that text genre heavily influences both the kinds of discourse relations one finds in a corpus and the way those relations are signalled (Webber, 2009). Since our focus is on developing methodology for consistent annotation, we therefore choose to work within a single genre. We selected instructional text because of its concrete, practical subject matter and because of its step-by-step organization, which makes it possible to automatically group together short segments of related text and imagery. TextImage Pairs. We base our data collection on an existing instructional dataset, RecipeQA (Yagcioglu et al., 2018). This is the only publicly available large-scale dataset of multimodal instructions. It consists of multimodal recipestextual instructions accompanied by one or more images. We excluded documents that either have multiple steps without images or that have multiple images per set. This was so that we could more easily study the direct relationship between an image and the associated text. There are 1,690 documents with this characteristic in the RecipeQA train set. To avoid overwhelming crowd workers, we further filtered those to retain only recipes with 70 or fewer words per step, for a final count of 516 documents (2,047 imagetext pairs). Protocol. We recruit participants through Amazon Mechanical Turk. All subjects were US citizens, agreed to a consent form approved by Rut-gers's institutional review board, and were compensated at an estimated rate of USD 15 an hour. Experiment Interface. Given an image and the corresponding textual instruction from the dataset, participants were requested to answer the following 10 questions. For Question 1, participants were asked to highlight the relevant part of the text. For the others, we solicited True/False responses. 1. Highlight the part of the text that is most related to the image. 2. The image gives visual information about the step described in the text. 1 The dataset and the code for the machine learning experiments are available at https://github.com/malihealikhani/CITE 3. You need to see the image in order to be able to carry out the step properly. 4. The text provides specific quantities (amounts, measurements, etc.) that you would not know just by looking at the picture. 5. The image shows a tool used in the step but not mentioned in the text. 6. The image shows how to prepare before carrying out the step. 7. The image shows the results of the action that is described in the text. 8. The image depicts an action in progress that is described in the text. 9. The text describes several different actions but the image only depicts one. 10. One would have to repeat the action shown in the image many times in order to complete this step. The interface is designed such that if the answer to Question 8 is TRUE , the subject will be prompted with Question 9 and 10. Otherwise, Question 8 is the last question in the list. Agreement. To assess the inter-rater agreement, we determine Cohen's  and Fleiss's  values. For Cohen's  , we randomly selected 150 imagetext pairs and assigned each to two participants, obtaining a Cohen's  of 0.844, which indicates almost perfect agreement. For Fleiss's  (Fleiss and Cohen, 1973; Cocos et al., 2015; Banerjee et al., 1999), we randomly selected 50 textimage pairs, assigned them to five subjects, and computed the average  . We obtain a score of 0.736, which indicates substantial agreement (Viera et al., 2005). Overall Statistics. Table 1 shows the rates of true answers for questions Q2Q10. Subjects reported that in 17% of cases the images did not give any information about the step described in the accompanying text. Such images deserve further investigation to characterize their interpretive relationship to the document as a whole. Our anecdotal experience is that such images sometimes provide context for the recipe, which may suggest that imagery, like real-world events (Hunter et al., 2015), creates more flexible discourse structures than linguistic segments on their own. of cases. This suggests that subjects construe imagery as backgrounded or peripheral to the document, much as speakers regard co-speech iconic gesture as peripheral to speech (Schlenker and Chemla, 2017). Note, by contrast, that subjects characterized 12.7% of images as introducing a new tool: this includes many cases where the same subjects say the image is not required. In other words, subjects' intuitions suggest that coherent imagery typically does not contribute instruction content, but rather serves as a visual signal that facilitates inferences that have to be made to carry out the instruction regardless. Our annotated examples, where imagery is linked to specific kinds of inferences, provide materials to test this idea. TEXT : Top with another layer of ravioli and the remaining sauce not all the ravioli may be needed. Sprinkle with the Parmesan. The Complex Coherence of Imagery. Our annotation reveals cases where a single image does include more information than could be packaged into a single textual discourse unit (the proverbial thousand words). In particular, such imagery participates in more complex coherence relationships than we find between text segments. Multiple temporal relationships show this most clearly: 12% of images that have any temporal relation have more than one. For example, many images depict the action that is described in the text, while also showing preparations that have already been made by displaying the scene in which the action is performed. Figure 2 depicts the action and the result of the action. It also shows how to prepare before carrying out the action. Other images show an action in progress but nearing completion and thereby depict the result. For instance, the image that accompanies mix well until blended can show both late-stage mixing and the blended result. Looking at a few such cases closely, the circumstances and composition of the photos seem staged to invite such overlapping inferences. Such cases testify to the richness of multimodal discourse, and help to justify our research methodology. The True/False questions characterize the relevant features of interpretation without necessarily mapping to single discourse relations. For instance, Q4 and Q5 indicate inferences in line with an Elaboration relation; Q9 and Q10 indicate inferences in line with an Exemplification relation, as information presented in images show just one case of a generalization presented in accompanying text. However, our data shows that these inferences can be combined in productive ways, in keeping with the potentially complex relevant content of images. Information across modalities. We carried out machine learning experiments to assess what information images provide and what textual cues can guide image interpretation. We use SVM classifiers for performance, and Multinomial Naive Bayes classifiers to explain classifier decision making, both with bag-of-words features. Table 2 reports the F1 measure for instance classification with SVMs (with 5-fold cross valida-tion). In many cases, machine learning is able to find cues that reliably help guess inferential pat-Q4. terns. Table 3 looks at two effective Naive Bayes classifiers, for Q4 (text has quantities) and Q8 (im-age depicts action in progress). It shows the features most correlated with the classification decision and their log probability estimates. For Q4, not surprisingly, numbers and units are positive instances. More interestingly, verbs of movement and combination are negative instances, perhaps because such steps normally involve material that has already been measured. For Q8, a range of physical action verbs are associated with actions in progress; negative features correlate with steps involved in actions that don't require ongoing attention (e.g., baking). Table 4 reports top SVM with NB (NBSVM) (Wang and Manning, 2012) features for Q1 that asks subjects to highlight the part of the text that is most related to the image. Action verbs are part of highlighted text, whereas adverbs and quantitative information that cannot be easily depicted in images are part of the remaining segments of the text. Such correlations set a direction for designing or learning strategies to select when to include imagery. In this paper, we have presented the first dataset describing discourse relations across text and imagery. This data affords theoretical insights into the connection between images and instructional text, and can be used to train classifiers to support automated discourse analysis. Another important Q1. contribution of this study is that it presents a discourse annotation scheme for cross-modal data, and establishes that annotations for this scheme can be procured from non-expert contributors via crowd-sourcing. Our paper sets the agenda for a range of future research. One obvious example is to extend the approach to other genres of communication with other coherence relations, such as the distinctive coherence of images and caption text (Alikhani and Stone, 2019). Another is to link coherence relations to the structure of multimodal discourse. For example, our methods have not yet addressed whether imagetext relations have the same kinds of subordinating or coordinating roles that comparable relations have in structuring text discourse (Asher and Lascarides, 2003). Ultimately, of course, we hope to leverage such corpora to build and apply better models of multimodal communication. The research presented here is supported by NSF Award IIS-1526723 and through a fellowship from the Rutgers Discovery Informatics Institute. Thanks to Gabriel Greenberg, Hristiyan Kourtev and the anonymous reviewers for helpful comments. We would also like to thank the Mechanical Turk annotators for their contributions.
objective objective abstain abstain abstain result abstain abstain abstain abstain method objective result result abstain objective abstain abstain abstain abstain abstain abstain abstain result result abstain abstain method result result method method result abstain result objective abstain abstain abstain abstain abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain objective abstain abstain objective abstain abstain abstain abstain method other other other