Understanding manipulated media, from automatically generated deepfakes' to manually edited ones, raises novel research challenges. Because the vast majority of edited or manipulated images are benign, such as photoshopped images for visual enhancements, the key challenge is to understand the complex layers of underlying intents of media edits and their implications with respect to disinformation. In this paper, we study Edited Media Understanding Frames , a new conceptual formalism to understand visual media manipulation as structured annotations with respect to the intents, emotional reactions, e  ects on individuals, and the overall implications of disinformation. We introduce a dataset for our task, EMU, with 56k question-answer pairs written in rich natural language. We evaluate a wide variety of vision-and-language models for our task, and introduce a new model PELICAN, which builds upon recent progress in pretrained multimodal representations. Our model obtains promising results on our dataset, with humans rating its answers as accurate 48.2% of the time. At the same time, there is still much work to be done  and we provide analysis that highlights areas for further progress. The modern ubiquity of powerful image-editing software has led to a variety of new disinformation threats. From AI-enabled deepfakes to low-skilled cheapfakes, attackers edit media to engage in a variety of harmful behaviors, such as spreading disinformation, creating revenge porn, and committing fraud (Paris and Donovan, 2019; Chesney and Citron, 2019; Kietzmann et al., 2020, c.f.). Accordingly, we argue that it is important to develop systems to help spot harmful manipulated media. The rapid growth and virality of social I want to suggest that subject2 has the support of subject1. media requires as such, especially as social media trends towards visual content (Gretzel, 2017). Identifying whether an image or video has been digitally altered (i.e., digital forgery detection) has been a long-standing problem in the computer vision and media forensics communities. This has enabled the development of a suite of detection approaches, such as analyzing pixel-level statistics and compression artifacts (Farid, 2009; Bianchi and Piva, 2012; Bappy et al., 2017) or identifying what the edit was (Tan et al., 2019). However, little work has been done on why an edit is made, which is necessary for identifying harm. Darkening someone's skin in a family photo because background light made them seem quite pale is generally harmless. While such color rebalancing is common, darkening Barack Obama's (or Rafael Warnock's) skin in campaign ads was clearly meant as a harmful edit by the editor that did it. 1 We choose to focus on the why  we define a schema for approaching the problem of intent and provide a rich set of natural language responses. We also make a significant contribution towards the what: we include a physical-change question, provide rationales based in physical changes, and give structured annotations (bounding boxes) on what was changed in the edit. We introduce Edited Media Understanding Frames (EMU), a new conceptual formalism that captures the notions of why and what in image editing for language and vision systems (Figure 1). Following literature on pragmatic frames (Sap et al., 2017, 2020; Forbes et al., 2020)derived from frame semantics (Baker et al., 1998) we formalize EMU frames along six dimensions that cover a diverse range of inferences necessary to fully capture the scope of visual disinformation. We delve into the concept of intention as discussed by the fake news literature (Rashkin et al., 2017; Shu et al., 2017; Zhou and Zafarani, 2020) to capture editor's intent such as motivation for edit and intent to deceive, as well as the resulting implications of the edited content. For every dimension we collect both a classification label and a free-form text explanation. For example, for frame intent , a model must classify the intent of the edit, and describe why this classification is selected. We then introduce a new dataset for our task, EMU, with 56k annotations over 8k image pairs. To kickstart progress on our task, we introduce a new language and vision model, PELICAN, that leverages recent progress in pretrained multimodal representations of images and text (Tan and Bansal, 2019; Lu et al., 2019; Li et al., 2019). We compare our model to a suite of strong baselines, including a standard VLP model (Zhou et al., 2019), and show key improvement in terms of ability to reason about co-referent subjects in the edit. Nevertheless, our task is far from solved: a significant gap remains 1 How Georgia's Senate race pits the Old South against the New South. https: // www.politico.com / news / 2020 / 12 / 05 / georgia-senate-old-new-south-442423 between the best machine and human accuracy. Our contributions are thus as follows. First, we introduce a new task of Edited Media Understanding Frames, which requires a deep understanding of why an image was edited, and a corresponding dataset, EMU, with 56k captions that cover diverse inferences. In addition, we introduce a new model, PELICAN, improving over competitive language-and-vision transformer baselines. Our empirical study demonstrates promising results, but significant headroom remains. We release our dataset at jeffda.com/edited-media-understanding to encourage further study in discovering pragmatic markers of disinformation. Through an edit e on source image i (e.g.  e = x is edited into a room full of drugs), an editor can cause harm to the subject x 's mental state ( mental state :  x is angry about e ) and e  ect x 's image ( effect :  e makes x seem dishonest) (Rashkin et al., 2016). The editor does this through the intention of the edit ( intent :  e intends to harm x 's image) and changing the implications of the image ( implication :  e frames x as a drug cartel member) (Forbes et al., 2020; Sap et al., 2020; Paris and Donovan, 2019). To this end, we collect edits e and source images i from Reddit's r/photoshopbattles community. There is no readily available (large) central database of harmful image edits, but r/photoshopbattles is replete with suitable complex and culturally implicative edits (e.g., reference to politics or pop culture). This provides us with relevant image edits at a reasonable cost without advocation for dangerous training on real harmful image edits. Keeping the source image i in the task allows us to sustain the tractability of the image edit problem (Tan et al., 2019; Jhamtani and Berg-Kirkpatrick, 2018). Given an edit e : IS  IE , we define an edited media understanding frame F (  ) as a collection of typed dimensions and their polarity assignments: (i) physical P ( IS  IE ): the changes from IS  IE , (ii) intent N ( E  IE ): whether the Editor E implied malicious intent in IS  IE , (iii) implication M ( E  IE ): how E might use IE to The Editor creates Image Edit from Image Source Frame N(E  IE): The Editor has Intent Frame M(E  IE): The Image Edit has potential Implications Frame S(IE  s1 ): The Image Edit impacts Mental State of Subject 1 Frame E(IE  s1 ): The Image Edit has Effect on perceptions of Subject 1 P(E  IE): The physical changes between Image Source and Image Edit Subject 1 is edited into a room full of illicit drugs Background Changed Structural changes Intent: to show Subject 1 in illegal behavior Implications: to frame Subject 1 as a member of a drug cartel Mental State: Subject 1 would be angry Effect: makes Subject 1 seem like a dishonest leader Subject 1 is shown next to illegal substances Label: Intent is Harmful Label: Implications is Harmful Label: Mental State of S1 is Negative Label: Effect on S1 is Negative Subject 1 is next to a massive amount of drugs Subject 1 would not want to be seen in front of drugs shows Subject 1 is in charge of illegal drugs Su b j e c t 1 because because because because Image Source EMU Frames Figure 2: An example from EMU. mislead, (iv) mental state S ( IE  s i ): whether the predicate IE impacts the emotion of a role s i , (v) e  ect E ( IE  s i ): the e  ect of IE on s i . We assume frames can be categorized as harmful or not harmful with polarity l  { + , } . Each polarity l can be interpreted with reason y , and that each reason can be supported with rationale r . Technically, a model is given the following as input:  A source image IS , and an edited image IE . A list of important subjects: expressed as bounding boxes b i for each subject. An open-ended question q associated with F (  ); e.g., How might subject3 feel upon seeing this edit?  A list of annotated boxes a i  IE marking the objects in the image that were introduced and modified , and a true / false label denoting if the background was changed. A model must produce the polarity classification l (cid:48)  { + , } , interpretation of the polarity (response y (cid:48) ) and rationale for interpolation r (cid:48) . (For the physical frame, only y needs to be generated). Figure 2 shows an example of our task configuration. The lexicon of the label is fixed for each F (  ) (e.g. for N (  ),   harmful, +  harmless). Sourcing Image Edits We source our image edits from the r/photoshopbattles community on Reddit which hosts regular Photoshop competitions, where given a source photo , members submit a comment with their own edited photo . We collect 8K image edit pairs (source and edited photo pairs) from this community by, first, manually curating a list of more than 100 terms describing people frequently appearing in Photoshop battles posts. Then, we screen over 100k posts for titles that contain one or more of these search terms resulting in 20k collected image pairs. Additionally, we run an object detector (He et al., 2017) to ensure that is at least one person present in each image as a means for ensuring that annotators do not see image pairs without any subjects. Annotating Image Edits We ask a group of vetted crowd workers to identify the main subjects in an image edit and answer open-ended questions in natural language. Each image is annotated by 3 independent crowd workers. Crowd workers are first presented with a numbered set of people bounding boxes (produced by Mask R-CNN (He et al., 2017)) over the edited F rame Notation Related Question P hysical P ( IS  IE ) What changed in this image edit? image and are asked to select subjects that are significant to the edits (as opposed, say, a crowd in the background). Once subjects are selected, the annotators are asked to assign classification labels for each of the five possible question types and provide free-form text answers for each question (when applicable). For the classification label, we retain the majority vote (Fleiss  = 0 . 67). In a separate and final pass, we explicitly identify which portions of the modified image is introduced or altered by asking the workers to to label the most important sections of the modified image and selecting one of the two labels. The statistics of the dataset are shown in Figure 3. 4 Modeling Edited Media Understanding Frames In this section, we present a new model for Edited Media Understanding Frames, with a goal of kick-starting research on this challenging problem. As described in Section 2, our task di  ers from many standard vision-and-language tasks both in terms of format and required reasoning: a model must take as input two images (a source image and its edit), with a significant change of implication added by the editor. A model must be able to answer questions, grounded in the main subjects of the image, describing these changes. The answers are either boolean labels, or open-ended natural language  including explainable rationales. For Edited Media Understanding Frames, not all image regions are created equal . Not only is the subject referred to in the question (e.g. subject1 ) likely important, so too are all of the regions in Figure 3: Statistics for EMU. the image edit that are introduced or altered . We propose to use the annotations that collected for these regions as additional signal for the model to highlight where to attend. 2 Not only should a model likely attend to these important regions, it should prioritize attending to regions nearby (such as objects that an edited person is interacting with). We propose to model the (likely) importance of an image region through graph propagation. We will build a directed graph with all regions of the image, rooted at a subject mentioned by the question (e.g. subject1 ). We will then topologically sort this graph; each region is then given an embedding corresponding to its sorted position  similar to the position embedding in a Transformer. This will allow the model to selectively attend to important image regions in the image edit. We use a di  erent position embedding for the image source, and do not perform the graph propagation here (as we do not have introduced or altered annotations); this separate embedding captures the inductive bias that the edited is more important than the source. 2 These annotations are collected from workers, but in theory, it would be possible to train a model to annotate regions as such. To make our task as accessible and easy-to-study as possible, however, we use the provided labels in place of a separate model however. In this section, we describe integrating our importance embeddings with a multimodal transformer. Let the source image be IS and IE . We use the backbone feature extractor  ( Faster-RCNN feature extractor (Ren et al., 2015; Anderson et al., 2018) to extract N regions of interest for each region: [ s 1 , ... , s N ] =  ( IS ) [ e 1 , ... , e N ] =  ( IE ) . (1) We note that some of these regions in e 1 , ... , e N are provided to the model (as annotated regions in the image); the rest are detected by  . These, plus the language representation of the question, are passed to the Transformer backbone T : [ z 1 ... z N + L ] = T ([ s 1 ... s N ] , [ e 1 , ... , e N ] , [ x 1 ... x L ]) (2) Important for EMU, z 2 N + 1 , ... , z 2 N + L serve as language representations. Training under a left-to-right language modeling objective, we can predict the next next token x L + 1 using the representation z N + L . Transformers require position embeddings to be added to each image region and word  enabling it to distinguish which region is which. We supplement the position embeddings of the regions { e 1 ... e N } in the edited image IE with the result of a topological sort. Graph definition. We define the graph over image regions in the edited image as follows. We begin by sourcing a seed region s  { e 1 ... e N } . Let G = ( V , E ), where each v  V represents metadata of some r i   ( IE ), defined as v i  m ( IE ) for simplicity, s.t.: v i = { x 1 , y 1 , x 2 , y 2 , s i , l i } (3) where x 1 , y 1 , x 2 , y 1 represents the bounding box of r i , s i  { 1 , 0 } denoting if r i is a subject of IE , and l i  { introduced , altered } denoting the label of r i . We build the graph iteratively: for each iteration, we define an edge e = { v , u } ; u  V s.t.:  v  m ( IE ) ,  u  V , E = E  ( u , v )  E (cid:48) (4) We define E (cid:48) as the set of edges ( u , v ) in which u and v are notationally similar . We define three cases in which this is true: if s i  u i  s j  v j , if l i  u i = l j  v j , and if x 1 , y 1 , x 2 , y 2  u i and x 3 , y 3 , x 4 , y 4  u i overlaps, in which the percentage overlap is defined by standard intersection-over-union: min { x 4 , x 2 }  max { x 3 , x 1 } min { y 4 , y 2 }  max { y 3 , y 1 } (5) We cap the number of outgoing edges at 3, and prevent cycles by allowing edges only to unseen image regions. In cases where there are more than three possible edges, we add edges in the order defined in the previous paragraph, and break overlap ties via maximum overlap. To produce embeddings, we run topological sort over the directed graph to assign each image region an embedding, then assign an embedding to each image region based on the ordered index. The embedding is zeroed out for image regions that are missing from the DAG, and from the source image (which are unlabeled). We include bounding box and class labels. To generate text and classification labels, we attach the embeddings onto the input for an encoder-decoder structure. In this section, we evaluate a variety of strong vision-and-language generators on EMU. Similar to past work on VQA, we rebalance our test set split ensuring a 50 / 50 split per question type of maliciously labeled captions. We provide two human evaluation metrics  head-to-head, in which generated responses are compared to human responses, and accuracy, in which humans are asked to label if generated responses are accurate in regards to the given edit. In addition to evaluating PELICAN, we compare and evaluate the performance of various potentially high-performing baselines on our task. a. Retrieval . For a retrieval baseline, which generally performs well for generation-based tasks, we use features from ResNet-158 (He et al., 2016), defined as  , to generate vectors for each IE in the test set. We then find the most similar edited image IT in the training set T via cosine similarity: argmax IT  T  ( IE )   ( IT ) (cid:107)  ( IE ) (cid:107)  (cid:107)  ( IT ) (cid:107) (6) We use the captions associated with the most similar image in the training set. b. GPT-2 (Radford et al., 2019) . As a text-only baseline, we use the 117M parameter model from GPT-2, fine-tuned on the captions from our dataset. Since the images are not taken into consideration, we generate from the seeds associated with each question type and use the same captions for all images in the test set. c. Cross-Modality GPT-2 . We test a unified language-and-vision model on our dataset. Similar to (Alberti et al., 2019), we append the visual features  ( IS ) and  ( IE ) to the beginning of the token embeddings from GPT-2 (117M). For the questions involving a subject , we append an additional vector  ( r ), where r is the region defined by the bounding box for that subject . d. Dynamic Relational Attention (Tan et al., 2019) . We test the best model from previous work on image edits on our task, Dynamic Relational Attention. We train the model from scratch on our dataset, using the same procedure as (Tan et al., 2019). We seed each caption with the relevant question. e. VLP (Zhou et al., 2019) . We test VLP, a pre-trained vision-and-language transformer model. For image captioning, VLP takes a single image as input and uses an o  -the-shelf object detector to extract regions, generation a caption using sequence-to-sequence decoding and treating the regions as a sequence of input tokens. To generate a caption for a particular question type, we fix the first few generated tokens to match the prefix for that question type. We fine-tune VLP starting from weights pre-trained on Conceptual Captions (3.3m image-caption pairs) (Sharma et al., 2018) and then further trained on COCO Captions (413k image-caption pairs) (Lin et al., 2014). We present our results in Table 2. We calculate generative metrics (e.g. METEOR) by appending the rationale to the response. Generations from PELICAN are preferred over human generations 14.0% of the time, with a 0.86 drop in perplexity compared to the next best model. To investigate the performance of the model, we run an ablation study on various modeling attributes, detailed in Table 3. First, we investigate the e  ect of pretraining (on Conceptual Captions (Sharma et al., 2018; Zhou et al., 2019)). We find that performance drops without pretraining (53.47%), but surprisingly still beats other baselines. This suggests that the task requires more pragmatic inferences than the semantic learning typically gained from pre-training tasks. Second, we ablate the importance of including annotated ( a i ) features from the dataset when creating the directed graph, relying on a seed from a random R-CNN region (54.44%). We also ablate our use of topological sort and a directed graph by suggesting a simple (but consistent) order for image regions (54.91%). Finally, we ablate including the visual regions from the source image. The performance is Figure 5: Generation examples from PELICAN, marked with results from human evaluation. similar (55.35%), suggesting that PELICAN would be able to perform in real-world settings in which only the edited image is present (e.g. social media posts). Last, we present qualitative examples in Figure 5. PELICAN is able to correctly understand image pairs which require mostly surface level understanding for example, in the top example, it is able to identify that the gun and action implies negative context, but misunderstands the response with regards to the situation. In the bottom example, we show that PELICAN is able to refer to subject1 correctly, but misinterprets the situation to be non-negative. To study if EMU is helpful in real-world settings, we train a model of PELICAN on EMU with only the edited image. In this setting, the model must hypothesize which parts of the image were edited and discern the main subjects in the image. At test time, we generate captions for each of the 5 intention-based question types. Results of this version of PELICAN is in Table 2. While this evaluation scheme is crude, we find that this version of PELICAN is still able to outperform previous models without usage of the source image. This suggests potential for generations from EMU-trained models in human-assisted settings. In an initial human study (given PELICANREAL captions, classify the edit as disinformation  were the captions helpful in your decision?) we find that annotators label as helpful 71.5% of the time. Additionally, annotators tended more often to pick the gold label (89.1%  95.2%). EMU also helps us understand what current vision-and-language models are missing for use on disinformation , by analyzing the reasons and rationales generated. We ask annotators to compare PELICAN-generated captions marked as worse and human captions. Category details are included in the appendix. Figure 6 shows our results. Overall, current models primarily lack the commonsense (event-based and social) to accurately describe disinformation. Geographical (location-based) and political (e.g. knowledge about the job of a president) external knowledge is also a missing component. PELICAN also still makes mistakes in description-related attributes: describing something other than the important change and an inaccuracy (e.g. wrong color) are the most common. Specific information  such as information relating to a specific person in the image (i.e. requiring a model to identify the person in the image), and information about a past event  are the least critical, suggesting that e  orts should be focused first on general intelligence rather than named-entity lookup. Language-and-Vision Datasets Datasets involving images and languages cover a variety of tasks, including visual question answering (Agrawal et al., 2015; Goyal et al., 2017), image caption generation (Lin et al., 2014; Young et al., 2014; Krishna et al., 2016), visual storytelling (Park and Kim, 2015; Bosselut et al., 2016), machine translation (Elliott et al., 2016), visual reasoning (Johnson et al., 2017; Hudson and Manning, 2019; Suhr et al., 2019), and visual common sense (Zellers et al., 2019). Two-image tasks Though most computer vision tasks involve single images, some work has been done on exploring image pairs. The NLVR2 dataset (Suhr et al., 2019) involves yes-no question answering over image pairs. Neural Naturalist (Forbes et al., 2019) tests fine-grained captioning of bird pairs; (Jhamtani and Berg-Kirkpatrick, 2018) iden-tifies the di  erence between two similar images. Image Edits There has been some computer vision research studying image edits. Unlike our EMU dataset, however, much of this work has focused on modeling lower-level image edits wherein the cultural implications do not change signifi-cantly between images. For example, (Tan et al., 2019) predicts image editing requests (generate change the background to blue' from a pair of images). Past work has also studied learning to perform image adjustments (like colorization and enhancement) from a language query (Chen et al., 2017; Wang et al., 2018). Hateful Meme Challenge (Kiela et al., 2020) is a recent work challenging models to classify a meme as hateful or not. We present Edited Media Understanding Frames a language-and-vision task requiring models to answer open-ended questions that capture the intent and implications of an image edit. Our model, PELICAN, kickstarts progress on our dataset  beating all previous models and with humans rating its answers as accurate 48.2% of the time. At the same time, there is still much work to be done  and we provide analysis that highlights areas for further progress. The authors would like to thank Ryan Qiu for help with analysis, and the Amazon Mechanical Turk community for help with annotation. This material is based upon work supported by the National Science Foundation Graduate Research Fellowship under Grant No. DGE1256082, and in part by NSF (IIS-1714566), DARPA CwC through ARO (W911NF15-1-0543), DARPA MCS program through NIWC Pacific (N66001-19-2-4031), NSF (IIS-1714566), and the Allen Institute for AI. In constructed the EMU dataset, great care was taken to ensure that crowd-workers are compensated fairly for their e  orts. To this end, we monitored median HIT completion times for each published batch, adjusting the monetary reward such that at least 80% of workers always received > $15 / hour, which is roughly double the minimum wage in the United States (the country of residence for most Amazon Mechanical Turk workers). This included the qualification and evaluation rounds. The following data sheet summarized relevant aspects of the data collection process (Bender and Friedman, 2018): A. C uration R ationale : Selection criteria for the edits included in the presented dataset are discussed Section 3. We selected the highest-rated posts on Reddit, and collected metadata data from annotators marking if the edit is NSFW or o  ensive. B. L anguage V ariety : The dataset is available in English, with mainstream US Englishes being the dominant variety, as per the demographic of Amazon Mechanical Turk workers. C. S peaker D emographic : N / A D. A nnotator D emographic : N / A E. S peech S ituation : All frames were collected and validated over a period of about 12 weeks, between November and January 2020, through the Amazon AMT platform. Workers were given regular, detailed feedback regarding the quality of their submissions and were able to address any questions or comments to the study's main author via Email or Slack. F. T ext C haracteristics : In line with the intended purpose of the dataset, the included edits describe social interactions related (but not limited to) platonic and romantic relationships, political situations, as well as cultural and social contexts. G. R ecording Q uality : N / A H. O ther : N / A Lastly, we want to emphasize that our work is strictly scientific in nature, and serves the exploration of machine reasoning alone. It was not developed to o  er guidance on misinformation or to train models to classify social posts as misinformation. Consequently, the inclusion of malicious image edits could allow adversaries to train malicious agents to produce visual misinformation. We are aware of this risk, but also want to emphasize that the utility of these agents allow useful negative training signal for minimizing harm that may be cased by agents operating in visual information. It is, therefore, necessary for future work that uses our dataset to specify how the collected examples of both negative and positive misinformation are used, and for what purpose.
abstain abstain objective abstain objective result result abstain abstain abstain objective abstain abstain abstain abstain abstain abstain abstain method objective objective abstain abstain method abstain objective abstain result objective other objective objective objective objective method abstain abstain method abstain method abstain method abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain method abstain method abstain abstain method method other abstain abstain abstain abstain method abstain other method method method method abstain abstain method abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain result method abstain method abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other other other other other method other other other method abstain result other other other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain