We present BERTGEN , a novel generative, decoder-only model which extends BERT by fusing multimodal and multilingual pre-trained models VL-BERT and M-BERT , respectively. BERTGEN is auto-regressively trained for language generation tasks, namely image captioning, machine translation and multimodal machine translation, under a multitask setting. With a comprehensive set of evaluations, we show that BERTGEN outperforms many strong baselines across the tasks explored. We also show BERTGEN 's ability for zero-shot language generation, where it exhibits competitive performance to supervised counterparts. Finally, we conduct ablation studies which demonstrate that BERTGEN substantially benefits from multi-tasking and effectively transfers relevant inductive biases from the pre-trained models. Recent work in unsupervised and self-supervised pre-training has revolutionised the field of natural language understanding (NLU), resulting in high performance ceilings across multiple tasks (Devlin et al., 2019; Yang et al., 2019; Dong et al., 2019). The recent success of language model pre-training with masked language modelling (MLM) such as BERT (Devlin et al., 2019) further paved the way for more complex approaches that combine language pre-training with images (Tan and Bansal, 2019; Su et al., 2020; Lu et al., 2020), video (Sun et al., 2019), and speech (Chuang et al., 2020). Most of these approaches follow a task-specific fine-tuning step after the model is pre-trained. However, there has been little work on exploiting pre-trained MLMs for natural language generation (NLG) tasks. Previous work argues that the MLM objective is ill-suited for generation tasks such as machine translation (Yang et al., 2019; Rothe et al., 2020). Recent work in this direction has predominantly investigated the use of pre-trained models to either initialise Transformer-based encoder-decoder models (Imamura and Sumita, 2019; Clinchant et al., 2019; Yang et al., 2020; Rothe et al., 2020) or to distill knowledge for sequence generation tasks (Chen et al., 2020). In this work, we present BERTGEN , which extends BERT in a generative setting (  2.1). This results in a single generator  without a separation between the encoder and the decoder  capable of consuming multiple input modalities and generating in multiple languages. The latter features are achieved by transferring knowledge from state-of-the-art pre-trained models, namely VL-BERT (Su et al., 2020) and multilingual BERT (M-BERT ) (Devlin et al., 2019). We train BERTGEN on various tasks, including image captioning, machine translation and multimodal machine translation, and datasets in four different languages (  2.2). Based on a number of experiments, our findings (  3) show that BERTGEN (i) is surprisingly versatile as it is capable of describing images and performing translation in unimodal and multimodal settings, across all languages, (ii) generalises well across zero-shot image captioning, multimodal machine translation, and out-of-domain news translation tasks, and finally (iii) is parameter efficient when compared to state-of-the-art models for each of the tasks combined together. In this section, we describe BERTGEN and the tasks we explore. We then detail the baselines and SoTA systems that we compare against. This section details the main aspects of BERTGEN that distinguish it from the existing work on vision & language pre-training. Initialisation. We take advantage of the previous successes in large-scale pre-training and propose a hybrid initialisation for BERTGEN (Figure 2). This involves using the VL-BERT (Su et al., 2020) checkpoint and initialising the word embeddings, the Transformer weights and the MLM head with M-BERT (Devlin et al., 2019). We conjecture that this primes BERTGEN to be aware of the visual modality and of multiple languages. This is simply due to VL-BERT being pre-trained on English monolingual and image captioning corpora, as well as M-BERT offering a 119K WordPiece vocabulary, trained on the entire Wikipedia in 104 languages 1 . Input configuration. While BERTGEN is potentially capable of modeling a variety of generative tasks, we focus on three particular tasks, namely machine translation (MT), multimodal MT (MMT) and image captioning (IC). Therefore, depending on the task, the input configuration of the model may change during both training and testing. To clarify further, let us first denote a sequence of embeddings representing a source sentence by x ( i ) = [ x ( i ) 1 ,    , x ( i ) m ] , its target translation by y ( i ) = [ y ( i ) 1 ,    , y ( i ) n ] , and a collection of k regional visual features extracted from an associated image by v ( i ) = [ v ( i ) 1 ,    , v ( i ) k ] . Figure 1 depicts BERTGEN when processing a sample from the MMT task. This task's input configuration is a triplet that involves all the three sequences i.e. { x ( i ) , y ( i ) , v ( i ) } . Using this notation, the MT and IC tasks' configurations would correspond to { x ( i ) , y ( i ) } and { v ( i ) , y ( i ) } , respectively. Visual embeddings. We follow VL-BERT and represent images as a collection of k features v ( i ) defined for regions of interest (RoI). After pre-extracting the 2048-dimensional RoI features using the bottom-up-top-down object detector (Anderson et al., 2018), we keep between 10 and 100 (i.e. k  [10 , 100] ) of them depending on the confi-dence score. The final visual embedding for an RoI is obtained by summing its feature vector and its geometric embedding (i.e. the projection of the STOP MASK MASK MASK MASK Figure 3: A look at BERTGEN 's self-attention: the connections denote that self-attentive representations are re-computed in every step. The generation ends when STOP is predicted. The smileys refer to RoI features. bounding box coordinates). When encoding the non-visual positions, the same RoI feature vector for the full image is repeated (see Figure 1). We note that we do not fine-tune the object detector during training. Sequence unrolling. An important aspect of BERTGEN is that it does not explicitly distinguish between the encoder and the decoder blocks usually seen in sequence-to-sequence models. This is accomplished by formalising both encoding and generation using the MLM framework. Formally, let us consider the MMT task and define the maximum log-likelihood objective for a given triplet { x ( i ) , v ( i ) , y ( i ) } where the target y ( i ) has n tokens: L ( i ) = n (cid:88) t =1 log (cid:16) P ( y ( i ) t | x ( i ) ; v ( i ) ; y ( i ) < t ) (cid:17) (1) In a typical sequence-to-sequence model, each log-probability term would be computed by a decoder within the forward-pass of the same training example. In contrast, BERTGEN explicitly unrolls the example n times, forming n new training examples. In other words, each conditional term in Equation 1 is observed independently within an epoch of training. Therefore, sequence unrolling has a data augmentation effect since a training corpus with D examples is approximately augmented by a factor of the average length of the target sequences. Moreover, the unified encoder-decoder formalism halves the number of parameters, making BERTGEN parameter efficient. Self attention. Given that a single Transformer (Vaswani et al., 2017) performs both encoding and decoding, sequence unrolling affects self-attention as well (Figure 3). First, all positions attend to each other for a given unrolled example i.e. the attention is bi-directional. Second, since each unrolled case is an independent example, the self-attentive representations of early positions are naturally re-computed , in contrast to typical Transformer decoders. Finally, due to how inputs/outputs are represented in a single stream and encoded through shared self-attention, BERTGEN enforces an inductive bias towards a truly multi-modal and multi-lingual representation space. Target language specifiers. Finally, to select the language during generation, input sequences begin with special target language specifiers (Ha et al., 2016; Johnson et al., 2017) (Figure 1). The specifier is task-agnostic, i.e. the same specifier [DE] is used both when captioning into German and when translating into German. Training & hyper-parameters. We extend 2 the base configuration of VL-BERT which is a Transformer with 12 self-attention layers and 12 heads. The model and feed-forward dimensions are 768 and 3072, respectively. On a single 32GB V100 GPU, one epoch (  3) takes approximately two days to complete as we could only fit one example per task (i.e. batch size equal to 13) into the memory 3 . We use AdamW optimiser (Loshchilov and Hutter, 2019) with base learning rate set to 1 . 3  10  5 . The learning rate is warmed up in the first 16K steps and then decays linearly. We set the weight decay to 10  4 . During training, we let the model update the positional embeddings as BERTGEN needs to learn new positions not covered by VL-BERT pretraining. The final model has  89.3M parameters excluding the word embeddings. Decoding. At test time, we incrementally add the most likely prediction (i.e. greedy search) into the previously masked position (Figure 1) and shift the [MASK] token right by one. The reason we chose greedy over beam search is because the latter would make decoding much slower due to self-attentive representations being re-computed. The decoding ends when [STOP] is predicted. To evaluate BERTGEN 's generative abilities, we explore a diverse set of tasks: image captioning, text-only MT and multimodal MT. Table 1 summarises the training statistics for the various datasets we use. 2 https://github.com/ImperialNLP/BertGen 3 With careful optimisation of the training code and mixed precision multi-GPU training, the training time can be substantially reduced. Image captioning (IC) involves describing images in a specified natural language. We train BERTGEN for English, German and Turkish captioning tasks. Specifically, we use the FLICKR 30 K dataset (Young et al., 2014) that provides 29K training images, each with five English captions collected through crowd-sourcing. The validation and test sets contain approximately 1K images each. We use the MULTI 30 K dataset (Elliott et al., 2016), which annotates FLICKR 30 K images with five German captions. Finally, we use the TASVIRET dataset (Unal et al., 2016) which provides two Turkish captions for each of the 8,092 images in the FLICKR 8 K dataset (Rashtchian et al., 2010). Since FLICKR 8 K is a subset of FLICKR 30 K , we create a new split of TASVIRET to avoid data leakage between training and test splits. The resulting training, validation and test splits contain 6914, 543, and 543 images, respectively. To evaluate BERTGEN 's performance on IC, we compare it against previous work with strong performance on COCO (Chen et al., 2015) and FLICKR 30 K . More precisely, ADAPTIVEATTENTION (SENTINEL ) (Lu et al., 2017), which uses a sentinel token to distinguish between visual and non-visual representations, and NEURALBABYTALK (NBT ), which follows a slot-filling approach through explicit object region information (Lu et al., 2018). Multimodal Machine Translation (MMT) attempts to improve MT quality by incorporating information from modalities other than language (Suluba-cak et al., 2020). In our case, we train BERTGEN for EN  DE and EN  FRMMT tasks and use the MULTI 30 K dataset, the main dataset for image-informed translation, which provides caption translations for FLICKR 30 K images in German and French. To evaluate BERTGEN on MMT tasks, we use the original 2016 test set which contains 1,000 examples. For a comprehensive comparison with previous work, we train a SoTA recurrent MMT (Caglayan et al., 2020) solely on the MULTI 30 K dataset, which applies a secondary (visual) attention in the decoder over the RoI features i.e. the same features that are also used by BERTGEN (  2.1). There are two GRU (Cho et al., 2014) layers in both the encoder and the decoder and the embedding & hidden dimensions in the model are set to 200 and 320 , respectively. Each model has  5.6M parameters excluding the word embeddings. Besides the state-of-the-art constrained recurrent MMT model described above, we further compare BERTGEN  which is trained on various other MT and IC corpora  to an unconstrained Transformer-based MMT trained on  9M additional EN  DE sentences (Libovicky, 2019) 4 in addition to MULTI 30 K . We incorporate six text-only MT tasks into our training protocol. We use EN  DE and EN  FRMT datasets from IWSLT'14 (Cettolo et al., 2012) which consists of TED Talks' subtitles and their translations. We take the prepare-iwslt14 recipe from FAIRSEQ (Ott et al., 2019) to prepare the dev and test sets. This yields an EN  DE test set of 6,750 sentences which consists of dev2010, dev2012.TEDX, tst2010, tst2011 and tst2012 . Similarly, the EN  FR test set consists of dev2010, tst2010, tst2011 and tst2012 , which amounts to 4,493 sentences. For EN  TR directions, we use the SETIMES2 (Tiedemann, 2012) news dataset for training. For development and test sets, we take the official WMT test sets (Bojar et al., 2018), namely, newstest2016 and newstest2017 as the development 4 We obtained test set outputs from the author and preprocessed with M-BERT tokeniser to ensure comparability. set (6,007 sentences), and newstest2018 (6,000 sentences) as the test set. Both IWSLT and SETIMES2 corpora are medium-scale resources often used in MT research community, and have much harder test sets than the MMT and IC tasks, due to a sig-nificant domain shift. Finally, for each translation direction, we train a Transformer NMT model (Vaswani et al., 2017) using the IWSLT-DE-EN recipe of the FAIRSEQ toolkit (Ott et al., 2019). This recipe has six encoders and six decoders, each equipped with 4 -head self-attention layers. The model and feed-forward dimensions are set to 512 and 1024 , respectively. Each model has  31.5M parameters excluding the word embeddings. Since BERTGEN is a general purpose multilingual and multimodal generator, we expect it to perform in the same ballpark as these strong NMT baselines, but not necessarily be SoTA compared to novel & sophisticated NMT models, which also make use of a lot more training data. We train BERTGEN on lowercased sentences for 45 epochs, after which the overall performance on the tasks reached a plateau. We define one BERTGEN epoch as a single pass over all of the training data for the MULTI 30 KEN  DEMMT task and denote this task as the reference task . We use greedy search for all systems that we trained and merge back the word pieces before evaluation. We compute tokenised 5 BLEU (Papineni et al., 2002), METEOR (Denkowski and Lavie, 2014) and CIDEr (Vedantam et al., 2015) using coco-caption 6 . In what follows, we provide detailed quantitative and qualitative findings. Table 2 provides an overview of BERTGEN 's image captioning performance on different test sets and languages. First of all, on English FLICKR 30 K , BERTGEN is clearly able to outperform strong captioning models (  2.2.1) SENTINEL (Lu et al., 2017) and NBT (Lu et al., 2018), even though they use beam search for decoding. On COCO (Chen et al., 2015), an image captioning corpus much larger and diverse than FLICKR 30 K , we evaluate BERTGEN on Karpathy's test split (Karpathy and Fei-Fei, 2015) and notice that the scores are reasonable 5 Since M-BERT is aggressive on splitting apostrophes and hyphens, our results may slightly differ from other work. given that BERTGEN is not trained on COCO: our model lags behind NBT (w/ beam search) by 6.7 METEOR. For zero-shot French captioning (F30K FR ), we resort to the reference MMT translations from the MULTI 30 KEN  FR task, as there are no human references for French. Although this is problematic as the metrics will penalise captions that are not translations of English captions, we provide the scores to show that the zero-shot outputs are valid descriptions. We note that the low range of scores reported here is also due to having one reference caption instead of five references 7 as in FLICKR 30 K . Finally we report results for our custom Turkish split (  2.2.1) (F30K TR ) and German (F30K DE ). Even though there are no comparable results in the literature for these three tasks, we demonstrate through some qualitative examples that BERTGEN produces sensible outputs. Qualitative examples. We now focus on a few examples to examine the multilingual image captioning ability of BERTGEN in action (Table 3). For the first image, all captions are almost the same as the image has few salient points. For the second image however, we observe much more variation across captions, in line with the complexity of the scene. We are particularly surprised by the zero-shot French captioning performance, a task that BERTGEN is not trained for at all. Upon manual inspection, we noticed that the captions are often short, objective gists of the images. These observations also hold for the captions generated for the 7 As a reference, evaluating English captions using one reference at a time, yields 7.9 BLEU on average, compared to 27.0 BLEU in Table 2. EN a man in a red shirt and helmet is riding a motorbike on a dirt road. DE ein mann fahrt mit einem motorrad auf einem weg an einem flu entlang. a man rides a motorcycle on a path along a river. TR camurlu bir yolda motoruyla ilerlemekte olan krmz ustlu bir adam ve arkasndaki dag manzaras. A man in a red top riding his bike down a muddy road with a mountain landscape behind him. FR un homme avec un casque fait du motocross. a man with a helmet rides motocross. COCO test set, as we can see in the third example. A set of additional examples in the Appendix shows that BERTGEN does not simply retrieve caption translations learned from the EN  FR task. Overall, both quantitative and qualitative results provide evidence of the utility of multimodal and multilingual initialisation as well as the efficacy of knowledge transfer across different tasks for image captioning. Table 4 summarises BERTGEN 's performance on MMT. First of all, BERTGEN consistently outperforms the Transformer-based FAIRSEQNMT models and the recurrent MMT (Caglayan et al., 2020) models on both the EN  DE and the EN  FR language pairs. Furthermore, BERTGEN is also substantially better than a state-of-the-art unconstrained MMT (Libovicky, 2019) model trained on a  6x larger parallel corpus. Adversarial evaluation. Following Elliott (2018), we probe BERTGEN 's ability for integrating multiple modalities effectively. Specifically, we decode translations by shuffling { image, source caption } mappings so that the images do not correspond to the sentences to be translated. The EN  DE results showed that the incongruence leads to 1 . 1 and 0 . 9 point drops in BLEU and METEOR, respectively. For EN  FR , the drops are much more prominent with 3 . 1 and 2 . 3 points again for BLEU and METEOR. This indicates that the features are not ignored at all, unlike in (Caglayan et al., 2019), where they showed that sequence-to-sequence MMT models can learn to ignore the images when the linguistic signal is sufficient to perform the task. in Table 4 show the surprising ability of BERTGEN to perform MMT on directions unseen during training. Moreover, the zero-shot performance surpasses strong MMT and NMT systems by up to 2 and 3.3 METEOR for DE  FR and FR  DE , respectively. Similar to the image captioning results, this demonstrates the potential of BERTGEN to generalise over a variety of language pairs and tasks. First, we compare BERTGEN 's performance to each task-specific FAIRSEQ system. According to Table 5, we observe that the translation quality of BERTGEN is generally superior compared to the strong FAIRSEQ systems, especially in METEOR, where BERTGEN leads in all pairs. Second, we look at the learning efficiency by comparing the training curves between BERTGEN and each task-specific FAIRSEQ system (Figure 4). Here, the x axis represents how many times the spe-D E  FRFR  DEBLMTBLMTBERTGEN 19.6 40.5 13.1 36.7 TARTU  39.5 59.0 26.3 47.3 MSRA  46.5 64.2 38.2 56.4 Table 6: Zero-shot BERTGEN performance on WMT'19 test set: TARTU and MSRA systems are not zero-shot as they are trained on DE  FR corpora. cific task's training set has been seen by the models. BERTGEN is trained for 45 reference epochs (  3), and this corresponds to only a few complete passes over the training sets of NMT tasks 8 . This is in contrast to the single-task systems that usually require a large number of epochs for convergence. We notice a general trend and observe that BERTGEN tends to outperform single-task systems usually after only a few passes over the corresponding training set. Many factors could be contributing to this observation such as sequence unrolling, multitasking, shared input space or relevant inductive biases transferred from M-BERT . We partly address these in the ablation studies (  3.4) and leave further investigation to future work. Zero-shot performance. We use the DE  FR test set from the WMT'19 shared task on news translation (Barrault et al., 2019) to assess the zero-shot translation capability of BERTGEN . This test set includes 1,701 sentences from news data regarding European Elections . We compare our results to two shared task systems, namely TARTU (base-line) and MSRA (state-of-the-art) (Barrault et al., 2019), after re-tokenising them accordingly with M-BERT 9 . Although BERTGEN is expected to obtain lower scores than the dedicated WMT systems due to the domain mismatch of the test set, we consider both the quantitative (Table 6) and the qualitative results (Table 7) extremely encouraging. We train single-task MMT systems on the MULTI 30 KEN  DE language pair. Specifically, we begin with a baseline system which is initialised with random weights. We then train a second baseline where only the visual processing layers are 8 For example, only  3 passes over SETIMESEN  TR . 9 TARTU is the baseline and MSRA is the best performing system for the shared task BERTGEN : la decision est tombee au 70`eme anniversaire de ma femme. transferred from VL-BERT . Finally, we train a third baseline that is initialised similar to BERTGEN , i.e. using the hybrid initialisation (  2.1). Figure 5 compares the validation BLEU scores of these three systems. We observe that the benefits of knowledge transfer from pre-trained models are incrementally positive, however, BERTGEN 's hybrid initialisation outperforms the other two ablations. We now remove the multi-tasking aspect from BERTGEN to investigate the extent to which the performance improvements are related to other tasks. Similar to  3.4.1, we focus on the MULTI 30 KEN  DEMMT task and train a single-task, hybrid-initialised BERTGEN . Figure 6 compares the validation BLEU scores obtained by the default BERTGEN and the single-task variant. We observe that BERTGEN benefits from multi-task training and, more importantly, does not seem to exhibit patterns 0 5 10 15 30 35 40 45 50 BLEU Passes over EN-DE MMT data Single Task Multi Task 20 Figure 6: Validation scores on MULTI 30 KEN  DEMMT for the multi-tasking ablation: The default multi-task BERTGEN outperforms the single-task one. of catastrophic forgetting (French, 1999). Based on these observations, we expect similar model behavior to hold for other tasks. Research in NLP and related fields has been increasingly focusing on transfer learning approaches where a model is first pre-trained on a data-rich task, and then transferred to downstream tasks (Mc-Cann et al., 2017; Peters et al., 2018; Devlin et al., 2019). This framework presumably allows the model to capture useful inductive biases that generalise to a variety of NLP tasks, often after performing a task-specific fine-tuning (Raffel et al., 2020). Of these, the most relevant studies to our work are BERT (Devlin et al., 2019) and its multilingual version M-BERT , which pre-train a Transformer (Vaswani et al., 2017) on large monolingual corpora using the masked language modelling (MLM) objective. Recent research has also attempted to combine linguistic inputs with other modalities such as vision and speech, to achieve a grounded understanding of meaning. Successful approaches including LXMERT (Tan and Bansal, 2019), VL-BERT (Su et al., 2020) and others (Lu et al., 2019; Li et al., 2020a,b) achieve this by combining BERT's MLM objective with auxiliary tasks such as masked region classification and image sentence matching, and pre-train their model on large-scale image captioning corpora (Chen et al., 2015; Sharma et al., 2018). Similarly, SpeechBERT extends BERT by jointly training on speech and text data (Chuang et al., 2020). Although SoTA results are reported by these approaches, they focus on unimodal and multimodal natural language understanding (NLU) tasks, with a strong emphasis in English. The backbone of BERTGEN combines VL-BERT (Su et al., 2020) with M-BERT (Devlin et al., 2019) to realise a multilingual and multimodal generator that can be used for a diverse set of generative tasks and languages rather than NLU tasks. Previous work has studied how to benefit from pre-trained BERT models in generative tasks such as NMT (Imamura and Sumita, 2019; Clinchant et al., 2019; Zhu et al., 2020). BERTGEN differs from these as it is not fine-tuned for a particular MT corpus and it exhibits multi-lingual and multimodal properties for general purpose generation. Another related branch of work explores pretraining strategies specific to sequence-to-sequence tasks. This includes MASS (Song et al., 2019), which exploits an encoder-decoder framework with the MLM objective for task-specific generative pre-training and UniLM (Dong et al., 2019), which introduces uni-directional, bi-directional and sequence-to-sequence LM objectives by carefully adjusting the self-attention masks during training. Zhou et al. (2020) extend UniLM to vision & language pre-training using Conceptual Captions (Sharma et al., 2018) as the pre-training dataset. However, these models require a further fine-tuning step for generative tasks, unlike BERTGEN that is trained only once . Several approaches exist for multi-task learning & generation (Dong et al., 2015; Luong et al., 2016) in NLP, especially in multilingual NMT, where tasks denote different language pairs (Zoph and Knight, 2016; Firat et al., 2016). The multi-task (and zero-shot) generation ability of BERTGEN is mostly inspired by Ha et al. (2016) and Johnson et al. (2017). Both of these introduced target language specifiers to select the output language when decoding translations from their model. Our multilingual & multimodal take on multitask generation is most similar to Kaiser et al. (2017), where a single Transformer model is trained on different tasks including image captioning, object classification, machine translation, speech recognition and parsing. However, their architecture depends on particular structures such as encoders, decoders, modality-specific networks and I/O mixers, unlike BERTGEN which does not require task-specific modules. In this paper, we presented BERTGEN , a novel generative, decoder-only model which extends BERT by combining multimodal and multilingual pre-trained models. Our findings show that BERTGEN obtains strong performance on a variety of generative tasks and further generalises over unseen tasks. Importantly, our model demonstrates the potential for general-purpose (instead of task-specific) generation that is above and beyond the traditional pre-training and fine-tuning practices. BERTGEN is also parameter efficient as it has 89.3M total parameters and is trained on thirteen tasks encompassing MT, multimodal MT and image captioning. On the other hand, each of the single-task FAIRSEQNMT baselines has 31.5M parameters. Our ablation studies show that BERTGEN is able to efficiently transfer relevant inductive biases from the pre-trained models and benefits from multi-task learning without suffering from catastrophic forgetting. We hope that these findings will motivate future research in exploiting more sophisticated pre-trained models in place of M-BERT and VL-BERT and others. This paper is a follow-up work to the MSc. Thesis of Faidon Mitzalis, co-supervised by Prof. Lucia Specia and Dr. Ozan Caglayan. Lucia Specia, Pranava Madhyastha and Ozan Caglayan received support from MultiMT project (H2020 ERC Starting Grant No. 678017). Lucia Specia also received support from the Air Force Office of Scientific Research (under award number FA8655-20-1-7006).
objective abstain objective result objective abstain other abstain abstain abstain abstain objective abstain abstain method result abstain abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method abstain other abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain abstain abstain abstain method method abstain abstain other abstain method method abstain method method abstain abstain method abstain abstain method method abstain abstain abstain method abstain method method abstain abstain method abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain result abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain result abstain abstain other other abstain other other other other other other other other other other other other other other other method other objective result objective abstain abstain result objective other other other other