Propaganda can be defined as a form of communication that aims to influence the opinions or the actions of people towards a specific goal; this is achieved by means of well-defined rhetorical and psychological devices. Propaganda, in the form we know it today, can be dated back to the beginning of the 17th cen-tury. However, it is with the advent of the Internet and the social media that it has started to spread on a much larger scale than before, thus becoming major societal and political issue. Nowadays, a large fraction of propaganda in social media is multimodal, mixing textual with visual content. With this in mind, here we propose a new multi-label multimodal task: detecting the type of propaganda techniques used in memes. We further create and release a new corpus of 950 memes, carefully annotated with 22 propaganda techniques, which can appear in the text, in the image, or in both. Our analysis of the corpus shows that understanding both modalities together is essential for detecting these techniques. This is further confirmed in our experiments with several state-of-the-art multimodal models. Social media have become one of the main communication channels for information dissemination and consumption, and nowadays many people rely on them as their primary source of news (Perrin, 2015). Despite the many benefits that social media offer, sporadically they are also used as a tool, by bots or human operators, to manipulate and to mislead unsuspecting users. Propaganda is one such communication tool to influence the opinions and the actions of other people in order to achieve a predetermined goal (IPA, 1938). Propaganda is not new. It can be traced back to the beginning of the 17th century, as reported in (Margolin, 1979; Casey, 1994; Martino et al., 2020), where the manipulation was present at public events such as theaters, festivals, and during games. In the current information ecosystem, it has evolved to computational propaganda (Wool-ley and Howard, 2018; Martino et al., 2020), where information is distributed through technological means to social media platforms, which in turn make it possible to reach well-targeted communities at high velocity. We believe that being aware and able to detect propaganda campaigns would contribute to a healthier online environment. Propaganda appears in various forms and has been studied by different research communities. There has been work on exploring network structure, looking for malicious accounts and coordinated inauthentic behavior (Cresci et al., 2017; Yang et al., 2019; Chetan et al., 2019; Pacheco et al., 2020). In the natural language processing community, propaganda has been studied at the document level (Barron-Cedeno et al., 2019; Rashkin et al., 2017), and at the sentence and the fragment levels (Da San Martino et al., 2019). There have also been notable datasets developed, including ( i ) TSHP-17 (Rashkin et al., 2017), which consists of document-level annotation labeled with four classes (trusted, satire, hoax, and propaganda); ( ii ) QProp (Barron-Cedeno et al., 2019), which uses binary labels (propaganda vs. non-propaganda), and ( iii ) PTC (Da San Martino et al., 2019), which uses fragment-level annotation and an inventory of 18 propaganda techniques. While that work has focused on text, here we aim to detect propaganda techniques from a multimodal perspective. This is a new research direction, even though large part of propagandistic social media content nowadays is multimodal, e.g., in the form of memes. They can easily become viral, and thus it is important to detect malicious ones quickly, and also to understand the nature of propaganda, which can help human moderators, but also journalists, by offering them support for a higher level analysis. Figure 1 shows some examples of memes 1 and propaganda techniques. Example (a) applies transfer , using symbols (hammer and sickle) and colors (red), that are commonly associated with communism, in relation to the two Republicans shown in the image; it also uses Name Calling ( traitors , Moscow Mitch , Moscow's bitch ). The meme in (b) uses both Smears and Glittering Generalities . The one in (c) expresses Smears and suggest that Joe Biden's campaign is only alive because of mainstream media. The examples in the second row show some less common techniques. Example (d) uses Appeal to authority to give credibility to a statement that rich politicians are crooks, and there is also a Thought-terminating clich  e used to discourage critical thought on the statement in the form of the phrase WE KNOW , thus implying that the Clintons are crooks, which is also Smears . 1 In order to avoid potential copyright issues, all memes we show in this paper are our own recreation of existing memes, using images with clear licenses. Then, example (e) uses both Appeal to (Strong) Emotions and Flag-waving as it tries to play on patriotic feelings. Finally, example (f) has Reduction ad hitlerum as Ilhan Omars' actions are related to such of a terrorist (which is also Smears ; moreover, the word HATE expresses Loaded language ). The above examples illustrate that propaganda techniques express shortcuts in the argumentation process, e.g., by leveraging on the emotions of the audience or by using logical fallacies to influence it. Their presence does not necessarily imply that the meme is propagandistic. Thus, we do not annotate whether a meme is propagandistic (just the propaganda techniques it contains), as this would require, among other things, to determine its intent. Our contributions can be summarized as follows:  We formulate a new multimodal task: propaganda detection in memes, and we discuss how it relates and differs from previous work. We develop a multi-modal annotation schema, and we create and release a new dataset for the task, consisting of 950 memes, which we manually annotate with 22 propaganda techniques. 2 2 The corpus and the code used in our experiments are available at https://github.com/di-dimitrov/ propaganda-techniques-in-memes . We perform manual analysis, and we show that both modalities (text and images) are important for the task. We experiment with several state-of-the-art textual, visual, and multimodal models, which further confirm the importance of both modalities, as well as the need for further research. Computational Propaganda Computational propaganda is defined as the use of automatic approaches to intentionally disseminate misleading information over social media platforms (Woolley and Howard, 2018). The information that is distributed over these channels can be textual, visual, or multi-modal. Of particular importance are memes, which can be quite effective at spreading multimodal propaganda on social media platforms (Diresta, 2018). The current information ecosystem and virality tools, such as bots, enable memes to spread easily, jumping from one target group to another. As of present, attempts to limit the spread of such memes have focused on analyzing social networks and looking for fake accounts and bots to reduce the spread of such content (Cresci et al., 2017; Yang et al., 2019; Chetan et al., 2019; Pacheco et al., 2020). Textual Content Most research on propaganda detection has focused on analyzing textual content (Barron-Cedeno et al., 2019; Rashkin et al., 2017; Da San Martino et al., 2019; Martino et al., 2020). Rashkin et al. (2017) developed the TSHP-17 corpus, which uses document-level annotation and is labeled with four classes: trusted , satire , hoax , and propaganda . TSHP-17 was developed using distant supervision, i.e., all articles from a given news outlet share the label of that outlet. The articles were collected from the English Gigaword corpus and from seven other unreliable news sources. Among them two were propagandistic. They trained a model using word n -gram representation with logistic regression and reported that the model performed well only on articles from sources that the system was trained on. Barron-Cedeno et al. (2019) developed a new corpus, QProp , with two labels: propaganda vs. non-propaganda. They also experimented on TSHP-17 and QProp corpora, where for the TSHP-17 corpus, they binarized the labels: propaganda vs. any of the other three categories. They performed massive experiments, investigated writing style and readability level, and trained models using logistic regression and SVMs. Their findings confirmed that using distant supervision, in conjunction with rich representations, might encourage the model to predict the source of the article, rather than to discriminate propaganda from non-propaganda. Similarly, Habernal et al. (2017, 2018) developed a corpus with 1.3k arguments annotated with five fallacies, including ad hominem , red herring , and irrelevant authority , which directly relate to propaganda techniques. A more fine-grained propaganda analysis was done by Da San Martino et al. (2019). They developed a corpus of news articles annotated with 18 propaganda techniques. The annotation was at the fragment level, and enabled two tasks: ( i ) binary classification given a sentence in an article, predict whether any of the 18 techniques has been used in it; ( ii ) multi-label multi-class classification and span detection task given a raw text, identify both the specific text fragments where a propaganda technique is being used as well as the type of the technique. On top of this work, they proposed a multi-granular deep neural network that captures signals from the sentence-level task and helps to improve the fragment-level classifier. Subsequently, a system was developed and made publicly available (Da San Martino et al., 2020). Multimodal Content Previous work has explored the use of multimodal content for detecting misleading information (Volkova et al., 2019), deception (Glenski et al., 2019), emotions and propaganda (Abd Kadir et al., 2016), hateful memes (Kiela et al., 2020; Lippe et al., 2020; Das et al., 2020), antisemitism (Chandra et al., 2021) and propaganda in images (Seo, 2014). Volkova et al. (2019) proposed models for detecting misleading information using images and text. They developed a corpus of 500,000 Twitter posts consisting of images labeled with six classes: disinformation, propaganda, hoaxes, conspiracies, clickbait, and satire. Then, they modeled textual, visual, and lexical characteristics of the text. Glenski et al. (2019) explored multilingual multimodal content for deception detection. They had two multi-class classification tasks: ( i ) classifying social media posts into four categories (propaganda, conspiracy, hoax, or clickbait), and ( ii ) classifying social media posts into five categories (disinformation, propaganda, conspiracy, hoax, or clickbait). Multimodal hateful memes have been the target of the popular Hateful Memes Challenge, which the participants addressed using fine-tuned state-of-art multi-modal transformer models such as ViLBERT (Lu et al., 2019), Multimodal Bitransformers (Kiela et al., 2019), and VisualBERT (Li et al., 2019) to classify hateful vs. not-hateful memes (Kiela et al., 2020). Lippe et al. (2020) explored different early-fusion multimodal approaches and proposed various methods that can improve the performance of the detection systems. Our work differs from the above research in terms of annotation, as we have a rich inventory of 22 fine-grained propaganda techniques, which we annotate separately in the text and then jointly in the text+image, thus enabling interesting analysis as well as systems for multi-modal propaganda detection with explainability capabilities. Propaganda comes in many forms and over time a number of techniques have emerged in the literature (Torok, 2015; Miller, 1939; Da San Martino et al., 2019; Shah, 2005; Abd Kadir and Sauf-fiyan, 2014; IPA, 1939; Hobbs, 2015). Different authors have proposed inventories of propaganda techniques of various sizes: seven techniques (Miller, 1939), 24 techniques Weston (2018), 18 techniques (Da San Martino et al., 2019), just smear as a technique (Shah, 2005), and seven techniques (Abd Kadir and Sauffiyan, 2014). We adapted the techniques discussed in (Da San Martino et al., 2019), (Shah, 2005) and (Abd Kadir and Sauffiyan, 2014), thus ending up with 22 propaganda techniques. Among our 22 techniques, the first 20 are used for both text and images, while the last two Appeal to (Strong) Emotions and Transfer are reserved for labeling images only. Below, we provide the definitions of these techniques, which are included in the guidelines the annotators followed (see appendix A.2) for more detal. 1. Loaded language: Using specific words and phrases with strong emotional implications (ei-ther positive or negative) to influence an audience. 2. Name calling or labeling: Labeling the object of the propaganda campaign as something that the target audience fears, hates, finds undesirable or loves, praises. 3. Doubt: Questioning the credibility of someone or something. 4. Exaggeration / Minimisation: Either representing something in an excessive manner: making things larger, better, worse (e.g., the best of the best , quality guaranteed ) or making something seem less important or smaller than it really is (e.g., saying that an insult was actually just a joke). 5. Appeal to fear / prejudices: Seeking to build support for an idea by instilling anxiety and/or panic in the population towards an alternative. In some cases, the support is built based on preconceived judgements. 6. Slogans: A brief and striking phrase that may include labeling and stereotyping. Slogans tend to act as emotional appeals. 7. Whataboutism: A technique that attempts to discredit an opponent's position by charging them with hypocrisy without directly disproving their argument. 8. Flag-waving: Playing on strong national feeling (or to any group; e.g., race, gender, political preference) to justify or promote an action or an idea. 9. Misrepresentation of someone's position (Straw man): Substituting an opponent's proposition with a similar one, which is then refuted in place of the original proposition. 10. Causal oversimplification: Assuming a single cause or reason when there are actually multiple causes for an issue. This includes transferring blame to one person or group of people without investigating the complexities of the issue. 11. Appeal to authority: Stating that a claim is true simply because a valid authority or expert on the issue said it was true, without any other supporting evidence offered. We also include here the special case where the reference is not an authority or an expert, which is referred to as Testimonial in the literature. 12. Thought-terminating cliche: Words or phrases that discourage critical thought and meaningful discussion about a given topic. They are typically short, generic sentences that offer seemingly simple answers to complex questions or that distract the attention away from other lines of thought. 13. Black-and-white fallacy or dictatorship: Presenting two alternative options as the only possibilities, when in fact more possibilities exist. As an the extreme case, tell the audience exactly what actions to take, eliminating any other possible choices (Dictatorship). 14. Reductio ad hitlerum: Persuading an audience to disapprove an action or an idea by suggesting that the idea is popular with groups hated in contempt by the target audience. It can refer to any person or concept with a negative connotation. 15. Repetition: Repeating the same message over and over again, so that the audience will eventually accept it. 16. Obfuscation, Intentional vagueness, Confusion: Using words that are deliberately not clear, so that the audience may have their own interpretations. For example, when an unclear phrase with multiple possible meanings is used within an argument and, therefore, it does not support the conclusion. 17. Presenting irrelevant data (Red Herring): Introducing irrelevant material to the issue being discussed, so that everyone's attention is diverted away from the points made. 18. Bandwagon Attempting to persuade the target audience to join in and take the course of action because everyone else is taking the same action. 19. Smears: A smear is an effort to damage or call into question someone's reputation, by propounding negative propaganda. It can be applied to individuals or groups. 20. Glittering generalities (Virtue): These are words or symbols in the value system of the target audience that produce a positive image when attached to a person or an issue. 21. Appeal to (strong) emotions: Using images with strong positive/negative emotional implications to influence an audience. 22. Transfer: Also known as association , this is a technique that evokes an emotional response by projecting positive or negative qualities (praise or blame) of a person, entity, object, or value onto another one in order to make the latter more acceptable or to discredit it. We collected memes from our own private Facebook accounts, and we followed various Facebook public groups on different topics such as vaccines, politics (from different parts of the political spec-trum), COVID-19, gender equality, and more. We wanted to make sure that we have a constant stream of memes in the newsfeed. We extracted memes at different time frames, i.e., once every few days for a period of three months. We also collected some old memes for each group in order to make sure we covered a larger variety of topics. We annotated the memes using the 22 propaganda techniques described in Section 3 in a multilabel setup. The motivation for multilabel annotation is that the content in the memes often expresses multiple techniques, even though such a setting adds complexity both in terms of annotation and of classification. We also chose to consider annotating spans because the propaganda techniques can appear in the different chunk(s), which is also in line with recent research (Da San Martino et al., 2019). We could not consider annotating the visual modality independently because all memes contain the text as part of the image. The annotation team included six members, both female and male, all fluent in English, with qualifi-cations ranging from undergrad, to MSc and PhD degrees, including experienced NLP researchers; this helped to ensure the quality of the annotation. No incentives were provided to the annotators. The annotation process required understanding the textual and the visual content, which poses a great challenge for the annotator. Thus, we divided it into five phases, as discussed below and as shown in Figure 2. Among these phases there were three stages, ( i ) pilot annotations to train the annotators to recognize the propaganda techniques, ( ii ) inde-pendent annotations by three annotators for each meme (phase 2 and 4), ( iii ) consolidation (phase 3 and 5), where the annotators met with the other three team members, who acted as consolidators, and all six discussed every single example in detail (even those for which there was no disagreement). We chose PyBossa 3 as our annotation platform as it provides the functionality to create a custom annotation interface that can fit our needs in each phase of the annotation. Phase 1 is about filtering some of the memes according to our guidelines, e.g., low-quality memes, and such containing no propaganda technique. We automatically extracted the textual content using OCR, and then post-edited it to correct for potential OCR errors. We filtered and edited the text manually, whereas for extracting the text, we used the Google Vision API. 4 We presented the original meme and the extracted text to an annotator, who had to filter and to edit the text in phase 1 as shown in Figure 2. For filtering and editing, we defined a set of rules, e.g., we removed hard to understand, or low-quality images, cartoons, memes with no picture, no text, or for which the textual content was strongly dominant and the visual content was minimal and uninformative, e.g., a single-color background. More details about filtering and editing are given in Appendix A.1.1 and A.1.2. In phase 2, we presented the edited textual content of the meme to the annotators as shown in Figure 2. We asked the annotators to identify the propaganda techniques in the text and to select the corresponding text spans for each of them. Phase 3 is the consolidation step of the annotations from phase 2 as shown in Figure 2. This phase was essential for ensuring the quality, and it further served as an additional training opportunity for the entire team, which we found very useful. Step 4 is multimodal meme annotation, i.e., considering both the textual and the visual content in the meme. In this phase, we show the meme, the post-edited text, and the consolidated propaganda labels from phase 3 (text only) to the annotators, as shown in phase 4 from Figure 2. We intentionally provided the consolidated text labels to the annotators in this phase because we wanted them to focus on the techniques that require the presence of the image rather than to reannotate those from the text. 5 4.1.5 Phase 5: Multimodal Consolidation This is the consolidation phase for Phase 4; the setup is like for the consolidation at Phase 3, as shown in Figure 2. Note that, in the majority of the cases, the main reason why two annotations of the same meme might differ was due to one of the annotators not spotting some of the techniques, rather than because there was a disagreement on what technique should be chosen for a given textual span or what the exact boundaries of the span for a given technique instance should be. In the rare cases in which there was an actual disagreement and no clear conclusion could be reached during the discussion phase, we resorted to discarding the meme (there were five such cases in total). 5 Ideally, we would have wanted to have also a phase to annotate propaganda techniques when showing the image only; however, this is hard to do in practice as the text is embedded as part of the pixels in the image. We assessed the quality of the annotations for the individual annotators from phases 2 and 4 (thus, combining the annotations for text and images) to the final consolidated labels at phase 5, following the setting in (Da San Martino et al., 2019). Since our annotation is multilabel, we computed Krippendorff's  , which supports multi-label agreement computation (Artstein and Poesio, 2008; Passon-neau, 2006). The results are shown in Table 1 and indicate moderate to perfect agreement (Landis and Koch, 1977). After the filtering in phase 1 and the final consolidation, our dataset consists of 950 memes. The maximum number of sentences per meme is 13, but most memes comprise only very few sentences, with an average of 1.68. The number of words ranges between 1 and 73 words, with an average of 17.79  11.60. In our analysis, we observed that some propaganda techniques were more textual, e.g., Loaded Language and Name Calling , while others, such as Transfer , tended to be more image-related. Table 2 shows the number of instances of each technique when using unimodal (text only, i.e., after phase 3) vs. multimodal (text + image, i.e., after phase 5) annotations. Note also that a total of 36 memes had no propaganda technique annotated. We can see that the most common techniques are Smears , Loaded Language , and Name calling/Labeling , covering 63%, 51%, and 36% of the examples, respectively. These three techniques also form the most common pairs and triples in the dataset as shown in Table 3. We further show the distribution of the number of propaganda techniques per meme in Figure 3. We can see that most memes contain more than one technique, with a maximum of 8 and an average of 2.61. Table 2 shows that the techniques can be found both in the textual and in the visual content of the meme, thus suggesting the use of multimodal learning approaches to effectively exploit all information available. Note also that different techniques have different span lengths. For example, Loaded Language is about two words long, e.g., violence , mass shooter , and coward . However, techniques such as Whataboutism need much longer spans with an average length of 22 words. Among the learning tasks that can be defined on our corpus, here we focus on the following one: given a meme, find all the propaganda techniques used in it, both in the text and in the image, i.e., predict the techniques as per phase 5. We used two nave baselines. First, a Random baseline, where we assign a technique uniformly at random. Second, a Majority class baseline, which always predicts the most frequent class: Smears . Unimodal: text only. For the text-based unimodal experiments, we used BERT (Devlin et al., 2019), which is a state-of-the-art pre-trained Transformer, and fastText (Joulin et al., 2017), which can tolerate potentially noisy text from social media as it is trained on word and character n -grams. Unimodal: image. For the image-based unimodal experiments, we used ResNet152 (He et al., 2016), which was successfully applied in a related setup (Kiela et al., 2019). Multimodal: unimodally pretrained For the multimodal experiments, we trained separate models on the text and on the image, BERT and ResNet152, respectively, and then we combined them using (a) early fusion Multimodal Bitransformers (MMBT) (Kiela et al., 2019), (b) middle fusion (feature concatenation), and (c) late fusion (com-bining the predictions of the models). For middle fusion, we took the output of the second-to-last layer of ResNet-152 for the visual part and the output of the [CLS] token from BERT, and we fed them into a multilayer network. Multimodal: joint models. We further experimented with models trained using a multimodal objective. In particular, we used ViLBERT (Lu et al., 2019), which is pretrained on Conceptual Captions (Sharma et al., 2018), and Visual BERT (Lin et al., 2014), which is pretrained on the MS-COCO dataset (Lin et al., 2014). We split the data into training, development, and testing with 687 (72%), 63 (7%), and 200 (21%) examples, respectively. Since we are dealing with a multi-class multi-label task, where the labels are imbalanced, we chose micro-average F 1 as our main evaluation measure, but we also report macro-average F 1 . We used the Multimodal Framework (MMF) (Singh et al., 2020). We trained all models on Tesla P100-PCIE-16GB GPU with the following manually tuned hyper-parameters (on dev): batch size of 32, early stopping on the validation set optimizing for F1-micro, sequence length of 128, AdamW as an optimizer with learning rate of 5e-5, epsilon of 1e-8, and weight decay of 0.01. All reported results are averaged over three runs with random seeds. The average execution time for BERT was 30 minutes, and for the other models it was 55 minutes. Table 4 shows the results for the models in Section 5.1. Rows 1 and 2 show a random and a majority class baseline, respectively. Rows 3-5 show the results for the unimodal models. While they all outperform the baselines, we can see that the model based on visual modality only, i.e., ResNet-152 (row 3), performs worse than models based on text only (rows 4-5). This might indicate that identifying the techniques in the visual content is a harder task than in texts. Moreover, BERT significantly outperforms fastText, which is to be expected as it can capture contextual representation better. Rows 6-8 present results for multimodal fusion models. The best one is BERT + ResNet-152 (+2 points over fastText + ResNet-152). We observe that early fusion models (rows 7-8) outperform late fusion ones (row 6). This makes sense as late fusion is a simple mean of the results of each modality, while early fusion has a more complex architecture and trains a separate multi-layer perceptron for the visual and for the textual features. We can also see that both mid-fusion models (rows 7-8) improve over the corresponding text-only ones (rows 3-5). Finally, looking at the results in rows 9-11, we can see that each multimodal model consistently outperforms each of the unimodal models (rows 1-8). The best results are achieved with ViLBERT CC (row 10) and VisualBERT COCO (row 11), which use complex representations that combine the textual and the visual modalities. Overall, we can conclude that multimodal approaches are necessary to detect the use of propaganda techniques in memes, and that pretrained transformer models seem to be the most promising approach. We have proposed a new multi-class multi-label multimodal task: detecting the type of propaganda techniques used in memes. We further created and released a corpus of 950 memes annotated with 22 propaganda techniques, which can appear in the text, in the image, or in both. Our analysis of the corpus has shown that understanding both modalities is essential for detecting these techniques, which was further confirmed in our experiments with several state-of-the-art multimodal models. In future work, we plan to extend the dataset in size, including with memes in other languages. We further plan to develop new multi-modal models, specifically tailored to fine-grained propaganda detection, aiming for deeper understanding of the semantics of the meme and of the relation between the text and the image. A number of promising ideas have been already tried by the participants in a shared task based on this data at SemEval-2021 (Dimitrov et al., 2021), which can serve as an inspiration when developing new models. User Privacy Our dataset only includes memes and it does not contain any user information. Biases Any biases found in the dataset are unintentional, and we do not intend to do harm to any group or individual. We note that annotating propaganda techniques can be subjective, and thus it is inevitable that there would be biases in our gold-labeled data or in the label distribution. We address these concerns by collecting examples from a variety of users and groups, and also by following a well-defined schema, which has clear definitions. Our high inter-annotator agreement makes us con-fident that the assignment of the schema to the data is correct most of the time. Misuse Potential We ask researchers to be aware that our dataset can be maliciously used to unfairly moderate memes based on biases that may or may not be related to demographics and other information within the text. Intervention with human moderation would be required in order to ensure this does not occur. Intended Use We present our dataset to encourage research in studying harmful memes on the web. We believe that it represents a useful resource when used in the appropriate manner. This research is part of the Tanbih mega-project, 6 which is developed at the Qatar Computing Research Institute, HBKU, and aims to limit the impact of fake news, propaganda, and media bias by making users aware of what they are reading.
abstain abstain abstain abstain objective objective result method abstain abstain abstain abstain abstain abstain objective abstain abstain abstain other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method result abstain abstain abstain abstain abstain abstain abstain objective objective other result method other other other other other other other other other other other other other other other other other other other objective other other other other other other other other other objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain method method method abstain method method abstain abstain abstain other method method method method other abstain abstain abstain abstain abstain other abstain abstain method method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain result abstain result result abstain abstain objective method result objective objective abstain abstain abstain abstain method abstain method abstain method method other