We take the first step towards multilingual style transfer by creating and releasing XFORMAL , a benchmark of multiple formal reformulations of informal text in Brazilian Portuguese, French, and Italian. Results on XFORMAL suggest that state-of-the-art style transfer approaches perform close to simple baselines, indicating that style transfer is even more challenging when moving multilingual. 1 1 Introduction Style Transfer ( ST ) is the task of automatically transforming text in one style into another (for example, making an impolite request more polite). Most work in this growing field has focused primarily on style transfer within English, while covering different languages has received disproportional interest. Concretely, out of 35 ST papers we reviewed, all of them report results for ST within English text, while there is just a single work covering each of the following languages: Chinese, Russian, Latvian, Estonian, and French (Shang et al., 2019; Tikhonov et al., 2019; Korotkova et al., 2019; Niu et al., 2018). Notably, even though some efforts have been made towards multilingual ST , researchers are limited to providing system outputs as a means of evaluation, and progress is hampered by the scarcity of resources for most languages. At the same time, ST lies at the core of human communication: when humans produce language, they condition their choice of grammatical and lexical transformations to a target audience and a specific situation. Among the many possible stylistic variations, Heylighen et al. (1999) argue that a dimension similar to formality appears as the most important and universal feature distinguishing styles, registers or genres in different languages  . Consider the informal excerpts and their formal reformulations in French ( FR ) and Brazilian Portuguese  Work done as a Research Intern at Dataminr, Inc. 1 Code and data: https://github.com/Elbria/xformal-FoST BRAZILIAN-PORTUGUESE saiam disso, fora de vontade!! get out of it, willpower!! Abandonem essa situao, tenham fora de vontade. Abandon this situation, have willpower! FRENCH Il avait les yeux braqus ailleurs. He had his eyes fixed elsewhere. ITALIAN in bocca al lupo! good luck! Ti rivolgo un sincero augurio! I send you a sincere wish! Il ne prtait pas attention  la situation. He was not paying attention to the situation. ( BR-PT ) in Table 1. Both informal-formal pairs share the same content. However, the informal language conveys more information than is contained in the literal meaning of the words (Hovy, 1987). These examples relate to the notion of deep formality (Heylighen et al., 1999), where the ultimate goal is that of adding the context needed to disambiguate an expression. On the other hand, variations in formality might just reflect different situational and personal factors, as shown in the Italian ( IT ) example. This work takes the first step towards a more language-inclusive direction for the field of ST by building the first corpus of style transfer for non-English languages. In particular, we make the following contributions: 1. Building upon prior work on Formality Style Transfer ( F o ST ) (Rao and Tetreault, 2018), we contribute an evaluation dataset , XFORMAL that consists of multiple formal rewrites of informal sentences in three Romance languages: Brazilian Portuguese ( BR-PT ), French ( FR ), and Italian ( IT ); 2. Without assuming access to any gold-standard training data for the languages at hand, we benchmark a myriad of leading ST baselines through automatic and human evaluation methods. Our results show that F o ST in non-English languages is particularly challenging as complex neural models perform on par with a simple rule-based system consisting of handcrafted transformations. We make XFORMAL , our annotations protocols, and analysis code publicly available and hope that this study facilitates and encourages more research towards Multilingual ST . Controlling style aspects in generation tasks is studied in monolingual settings with an English-centric focus (intra-language) and cross-lingual settings together with Machine Translation ( MT ) (inter-language). Our work rests in intra-language ST with a multilingual focus, in contrast to prior work. ST datasets that consist of parallel pairs in different styles include: GYAFC for formality (Rao and Tetreault, 2018), Yelp (Shen et al., 2017) and Amazon Product Reviews for sentiment (He and McAuley, 2016), political slant and gender controlled datasets (Prabhumoye et al., 2018), Expert Style Transfer (Cao et al., 2020), PASTEL for imitating personal (Kang et al., 2019), SIMILE for simile generation (Chakrabarty et al., 2020), and others. Intra-language ST was first cast as generation task by Xu et al. (2012) and is addressed through methods that use either parallel data or unpaired corpora of different styles. Parallel corpora designed for the task at hand are used to train traditional encoder-decoder architectures (Rao and Tetreault, 2018), learn mappings between latent representation of different styles (Shang et al., 2019), or fine-tune pre-trained models (Wang et al., 2019). Other approaches use parallel data from similar tasks to facilitate transfer in the target style via domain adaptation (Li et al., 2019), multi-task learning (Niu et al., 2018; Niu and Carpuat, 2020), and zero-shot transfer (Korotkova et al., 2019) or create pseudo-parallel data via data augmentation techniques (Zhang et al., 2020; Krishna et al., 2020). Approaches that rely on non-parallel data include disentanglement methods based on the idea of learning style-agnostic latent representations (e.g., Shen et al. (2017); Hu et al. (2017)). However, they are recently criticized for resulting in poor content preservation (Xu et al., 2018; Jin et al., 2019; Luo et al., 2019; Subramanian et al., 2018) and alternatively, translation-based models are proposed that use reconstruction and back-translation losses (e.g., Logeswaran et al. (2018); Prabhumoye et al. (2018)). Another line of work, focuses on manipulation methods that remove the style-specific attribute of text (e.g., Li et al. (2018); Xu et al. (2018)), while recent approaches use reinforcement learning (e.g., Wu et al. (2019); Gong et al. (2019), probabilistic formulations (He et al., 2020), and masked language models (Malmi et al., 2020). Inter-language ST is introduced by Mirkin and Meunier (2015) who proposed personalized MT for EN -French and EN -German. Subsequent MT works control for politeness (Sennrich et al., 2016a), voice (Yamagishi et al., 2016), personality traits (Rabinovich et al., 2017), user-provided terminology (Hasler et al., 2018), gender (Vanmassen-hove et al., 2018), formality (Niu et al., 2017; Feely et al., 2019), morphological variations (Moryossef et al., 2019), complexity (Agrawal and Carpuat, 2019) and reading level (Marchisio et al., 2019). We describe the process of collecting formal rewrites using data statements protocols (Bender and Friedman, 2018; Gebru et al., 2018). Curation rational To collect XFORMAL , we firstly curate informal excerpts in multiple languages. To this end, we follow the procedures described in Rao and Tetreault (2018) (hence-forth RT 18 ) who create a corpus of informal-formal sentence-pairs in English ( EN ) entitled Grammarly's Yahoo Answers Formality Corpus ( GYAFC ). Concretely, we use the L 6 Yahoo! Answers corpus that consists of questions and answers posted to the Yahoo! Answers platform. 2 The corpus contains a large number of informal text and allows control for different languages and different domains. 3 Similar to the collection of GYAFC , we extract all answers from the Family & Relationships (F&R) topic that correspond to the three languages of interest: Famlia e Relacionamen-tos ( BR-PT ), Relazioni e famiglia ( IT ), and Amour et relations ( FR ) ( Step 1 ). We follow the same pre-processing steps as described in RT 18 for consistency ( Step 2 ). We filter out answers that: a) consist of questions; b) include URL s; c) have fewer 2 https://webscope.sandbox.yahoo.com/ than five or more than 25 tokens; or d) constitute duplicates. 4 We automatically extract informal candidate sentences, as described in 5.3 ( Step 3 ). Finally, we randomly sample 1 , 000 sentences from the pool of informal candidates for each language. Table 2 presents statistics of the curation steps. Procedures We use the Amazon Mechanical Turk ( MT urk) platform to collect formal rewrites for our informal sentences. For each language, we split the annotation into 20 batches of 50 Human Intelligence Tasks ( HIT s). In each HIT , Turkers are given an informal excerpt and asked to generate its formal rewrite in the same language without changing its meaning . We collect 4 rewrites per excerpt and release detailed instructions under A.G. Annotation Workflow & Quality Control Our annotation protocol consists of multiple Quality Control ( QC ) steps to ensure the recruitment of high-quality annotators. As a first step, we use location restrictions ( QC 1) to limit the pool of workers to countries where native speakers are most likely to be found. Next, we run several small pilot studies (of 10 HIT s) to recruit potential workers. To participate in the pilot study, Turkers have to pass a qualification test ( QC 2) consisting of multiple-choice questions that test workers' understanding of formality (see A.L). The pilot study results are reviewed by a native speaker ( QC 3) of each language to exclude workers who performed consistently poorly. We find that the two main reasons for poor quality are: a) rewrites of minimum-level edits, or b) rewrites that change the input's meaning. Table 3 presents the number of workers at each QC step. Only workers passing all quality control steps (last row of Table 3) contribute to the final task. Finally, we post-process the collected rewrites by a) removing instances consisting of normalization-based edits only and b) correcting minor spelling errors using an off-the-shelf tool. 5 4 We tokenize with nltk: https://www.nltk.org/ api/nltk.tokenize.html 5 https://languagetool.org/ Step Description BR-PT FR IT ( QC 1) Location restriction 151 78 59 ( QC 2) Qualification test 54 40 33 ( QC 3) Rewrites review 9 16 11 Table 3: Number of Turkers after each QC step. Turkers' demographics We recruit Turkers from Brazil, France/Canada, and Italy for BR-PT , FR , and IT , respectively. Beyond their country of residence, no further information is available. Compensation We compensate at a rate of $ 0 . 10 per HIT with additional one-time bonuses that bumps them up to a target rate of over $ 10 /hour. After this entire process, we have constructed a high-quality corpus of formality rewrites of 1 , 000 sentences for three languages. In the next section, we provide statistics and an analysis of XFORMAL . Types of formality edits Following Pavlick and Tetreault (2016), we analyze the most frequent edit operations Turkers perform when formalizing the informal sentences. We conduct both an automatic analysis (details in A.I) of the whole set of rewrites, and a human analysis (details in A.H) of a random sample of 200 rewrites per language (we recruited a native speaker for each language). Table 4 presents both analyses' results, where we also include the corresponding statistics for the English language ( GYAFC ). In general, we observe similar trends across languages: humans make edits covering both the "noisy-text" sense of formality (e.g., fixing punctuation, spelling errors, capitalization) and the more situational sense (paraphrase-based edits). Although cross-language trends are similar, we also observe differences: deleting fillers and word completion seems to be more prominent in the English rewrites than in other languages; normalizing abbreviations is a considerably frequent edit type for Brazilian Portuguese; paraphrasing is more frequent in the three non-English languages. Surface differences of informal-formal pairs We quantify surface-level differences between the informal sentences and formal rewrites via computing their character-level Levenshtein distance (Fig-ure 1) and their pairwise Lexical Difference (LeD) based on the percentages of tokens that are not found in both sentences (Table 5). Both analyses show that Italian rewrites have the most edits com-(a) BR-PT pared to their corresponding informal sentences. French and Brazilian Portuguese follow, with English rewrites being closer to the informal inputs. a large number of reformulations consist of paraphrase-based edits (more than 50 %), we want to quantify the extent to which the formal rewrites of each sentence are diverse , in terms of their lexical choices. To that end, we quantify diversity via measuring selfBLEU (Zhu et al., 2018): considering one set of formal sentences as the hypothesis set and the others as references, we compute BLEU for each formal set and define the average BLEU score as a measure of the dataset's diversity. Higher scores imply less diversity of the set. Results (last row of Table 4) show that XFORMAL consists of more diverse rewrites compared to GYAFC . Formality shift of rewrites We analyze the formality distribution of the original informal sentences with their formal rewrites in GYAFC and XFORMAL , as predicted by formality m BERT models (5.3). The distributions of formal rewrites are skewed towards positive values (Figure 2). We benchmark eight ST models on XFORMAL to serve as baseline scores for future research. We describe the models (5.1), the experimental setting (5.2), the human and automatic evaluation methods (5.3 and 5.4), and results (5.5). Simple baselines We define three baselines: 1. COPY Motivated by Pang and Gimpel (2019) who notice that untransferred sentences with no alterations have the highest BLEU score by a large margin for ST tasks, we use this simple baseline as a lower bound; 2. RULE-BASED Based on the quantitative analysis of 4 and similarly to RT 18 , we develop a rule-based approach that performs a set of predefined edits-based operations defined by handcrafted rules. Example transformations include fix casing, remove repeated punctuation, handcraft a list of contraction expansionsa detailed description is found at A.C; 3. ROUND-TRIP MT Inspired by Zhang et al. (2020) who identify useful training pairs from the paraphrases generated by round-trip translations of millions of sentences, we devise a simpler baseline that starts from a text in language x , pivots to EN and then backtranslates to x , using the AWS translation service. 6 NMT-based models with synthetic parallel data We follow the TRANSLATE TRAIN (Conneau et al., 2018; Artetxe et al., 2020) approach to collect data in multilingual settings: we obtain pseudo-parallel corpora in each language via machine translating an EN resource of informal-formal pairs (5.2). 7 Then, starting with TRANSLATE TRAIN we benchmark the following NMT -based models: 1. TRANSLATE TRAIN TAG extends a leading EN F o ST approach (Niu et al., 2018) and trains a unified model that handles either formality direction via attaching a source tag that denotes the desired target formality; 2. MULTI-TASK TAG-STYLE Niu et al. (2018) augments the previous approach with bilingual data that is automatically identified as formal (5.3). The models are then trained in a multi-task fashion; 3. BACKTRANSLATE augments the TRANSLATE TRAIN data with back-translated sentences of automatically detected informal text (Sennrich et al., 2016b), using 1 . as the base model. We exclude backtranslated pairs consisting of copies. The output of the RULE-BASED system is given as input to each model at inference time. For all three models, we run each system with 4 random seeds, and combine them in a linear ensemble for decoding. LATION ( UNMT ) (Subramanian et al., 2018) defines a pseudo-supervised setting and combines de-noising auto-encoding and back-translation losses; 2. DEEP LATENT SEQUENCE MODEL ( DLSM ) (He et al., 2020) defines a probabilistic generative story that treats two unpaired corpora of separate styles as a partially observed parallel corpus and learns a mapping between them, using variational inference. Training data For TRANSLATE TRAIN TAG we use GYAFC , a large set of 110 KEN informal-formal parallel sentence-pairs obtained through crowd-sourcing. Additionally, we augment the translated resource with OpenSubtitles (Lison and Tiede-mann, 2016) bilingual data used for training MT models. 8 Given that bilingual sentence-pairs can be noisy, we perform a filtering step to extract noisy bitexts using the Bicleaner toolkit (Snchez-Cartagena et al.). 9 Furthermore, we apply the same filtering steps as in 3 (Curation rational). Finally, each of the remaining sentences is assigned a formality score (5.3), resulting in two pools of informal and formal text. Training instances are then randomly sampled from those pools: formal parallel pairs are used for MULTI-TASK TAG-STYLE ; informal target side sentences are backtranslated for BACKTRANSLATE ; both informal and formal target-side texts are independently sampled from the two pools for training unsupervised models. Finally, for unsupervised F o ST in FR , we additionally experiment with in-domain data from the L 26 French Yahoo! Answer Corpus that consists of 1 . 7 MFR questions. 10, 11 Table 6 includes statistics on training sizes. 12 METHODSGYAFC OpenSubs. 8 Data are available at: http://opus.nlpl.eu/ . 9 We use the publicly available pretrained Bicleaner models: https://github.com/bitextor/bicleaner , and discard sentences with a score lower than 0 . 5 . 10 https://webscope.sandbox.yahoo.com/ catalog.php?datatype=l&did=74 11 Split into 6 . 2 M/ 6 . 6 M formal/informal sentences. 12 Bilingual data statistics are in A.K. Preprocessing We preprocess data consistently across languages using MOSES (Koehn et al., 2007). Our pipeline consists of three steps: a) normalization; b) tokenization; c) true-casing. For NMT based approaches, we also learn joint source-target BPE with 32 K operations (Sennrich et al., 2016b). Model Implementations For NMT-based and unsupervised models we use the open-sourced im-pementations of Niu et al. (2018) and He et al. (2020), respectively. 13,14 We include more details on model architectures in A.D. 5.3 Automatic Evaluation Recent work on ST evaluation highlights the lack of standard evaluation practices (Yamshchikov et al., 2020; Pang, 2019; Pang and Gimpel, 2019; Mir et al., 2019). We follow the most frequent evaluation metrics used in EN tasks and measure the quality of the system's outputs with respect to four dimensions, while we leave an extensive evaluation of automatic metrics for future work. Formality We average the style transfer score of transferred sentences computed by a formality regression model. We fine-tune m BERT (Devlin et al., 2019) pre-trained language models on the machine-translated answers genre from Pavlick and Tetreault (2016) that consists of about 4 K human-annotated sentences rated on a 7 -point formality scale. To acquire an annotated corpus in the languages of interest, we follow the TRANSLATE TRAIN transfer approach: we propagate the original EN training data's human ratings to their corresponding translations, assuming that translation preserves formality. 15 To evaluate the multilingual formality regression models' performance, we crowdsourced human judgments of 5 Turkers for 200 sentences per language. We report Spearman correlations of 64 ( BR-PT ), 70 ( IT ), 71 ( FR ), and 81 ( EN ). Fluency We compute the logarithm of each sen-tence's probabilitycomputed by a 5 -gram Kneser-Ney language model (Kneser and Ney, 1995)and normalize it by the sequence length. We train each 13 https://github.com/xingniu/ multitask-ft-fsmt 14 https://github.com/cindyxinyiwang/ deep-latent-sequence-model 15 See A.A for discussion on this assumption. Overall We compute multiBLEU (Post, 2018) via comparing with multiple formal rewrites on XFORMAL . Freitag et al. (2020) shows that correlation with human judgments improves when considering multiple references for MT evaluation. Given that automatic evaluation of ST lacks standard evaluation practiceseven in cases when EN is consideredwe turn to human evaluation to reliably assess our baselines following the protocols of RT 18 . We sample a subset of 100 sentences from XFORMAL per language, evaluate outputs of 5 systems, and collect 5 judgments per instance.We open the task to all workers passing QC 2 in Table 3. We include inter-annotator agreement results in A.E. Formality We collect formality ratings for the original informal reference, the formal human rewrite, and the formal system outputs on a 7 point discrete scale of  3 to 3 , following Lahiri (2015) ( Very informal  Informal  Somewhat Informal  Neutral  Somewhat Formal  Formal  Very Formal ). Fluency We collect fluency ratings for the original informal reference, the formal human rewrite, and the formal system outputs on a discrete scale of 1 to 5 , following Heilman et al. (2014) ( Other  Incomprehensible  Somewhat Comprehensible  Comprehensible  Perfect ). Meaning Preservation We adopt the annotation scheme of Semantic Textual Similarity (Agirre et al., 2016): given the informal reference and formal human rewrite or the formal system outputs, Turkers rate the two sentences' similarity on a 1 to 6 scale ( Completely dissimilar  Not equivalent but on same topic  Not equivalent but share some details  Roughly equivalent  Mostly equivalent  Completely equivalent ). Overall We collect overall judgments of the system outputs using relative ranking : given the informal reference and a formal human rewrite, workers are asked to rank system outputs in the order of their overall formality, taking into account both fluency and meaning preservation. An overall score is then computed for each model via averaging results across annotating instances. Table 7 shows automatic results for all models across the four dimensions as well as human ratings for selected top models. NMT -based model evalatuation Concretely, the RULE-BASED baselines are significantly ( p < 0 . 05 ) the best performing models in terms of meaning preservation across languages. This result is intuitive as the RULE-BASED models act at the surface level and are unlikely to change the informal sen-tence's meaning. The BACKTRANSLATE ensemble systems are the second-best performing models in terms of meaning preservation, while the ROUNDTRIP MT outputs diverge semantically from the informal sentences the most. Those results are consistent across languages and human/automatic evaluations. On the other hand, when we compare systems in terms of their formality , we observe the opposite pattern: the RULE-BASED and BACKTRANSLATE outputs are the most informal compared to the other ensemble NMT -based approaches across languages. Interestingly, the ROUND-TRIP MT outputs exhibit the largest formality shift for BR-PT and FR as measured by human evaluation. The trade-off between meaning preservation and formality among models was also observed in EN ( RT 18 ). Moreover, when we move to fluency , we notice similar results across systems. Specifically, human evaluation assigns almost all models an average score of > 4 , denoting that system outputs are comprehensible on average, with small differences between systems not being statistically significant. Notably, perplexity tells a different story: all system outputs are significantly better compared to the RULE-BASED systems across configurations and languages. This result denotes that perplexity might not be a reliable metric to measure fluency in this setting, as noticed in Mir et al. (2019) and Krishna et al. (2020). When it comes to the overall ranking of systems, we observe that the NMT -based ensembles are better than the RULEBASED baselines for BR-PT and FR , yet by a small margin as denoted by both multiBLEU and human evaluation. However, the corresponding results for IT denote that there is no clear win, and the NMT based ensembles still fail to surpass the naive RULEBASED models, yet by a small margin. Finally, all ensembles outperform the trivial COPY baseline. Table 8 presents examples of system outputs. As a side note, we followed the recommendation of Tikhonov and Yamshchikov (2018) to show the performance of ST models of individual runs and visualize trade-offs between metrics better. Unlike their work which found that reruns of the same model showed wide performance discrepancies, we found that most of our NMT -based models did not vary in performance on XFORMAL . The results can be visualized in A.M. Unsupervised model evaluation We also benchmark the unsupervised models but focus solely on automatic metrics since they lag behind their supervised counterparts. As shown in Table 7, when ( BR-PT ) INFORMAL n preciso pedir pois sei q ela vai vir atras!! using out-of-domain data (e.g., OpenSubtitles) for training, the models perform worse than their NMT counterparts across all three languages. The difference is most stark when considering selfBLEU and multiBLEU scores. However, given access to large in-domain corpora (e.g., L 26 Yahoo! French Answers) the gap between the two model classes closes with DLSM achieving a multiBLEU score of 42.1 compared to 48.3 for the best performing NMT model BACKTRANSLATE . This shows the promise of unsupervised methods, assuming a large amount of in-domain data, on multilingual ST tasks. Lexical differences of system outputs Finally, in Figure 3 we analyze the diversity of outputs by leveraging LeD scores resulting from pair-wise comparisons of different NMT systems. A larger LeD score denotes a larger difference between the lexical choices of the two systems under comparison. First, we observe that the ROUND-TRIP MT outputs have the smallest lexical overlap with the informal input sentences. However, when this observation is examined together with human evaluation results, we conclude that the large number of lexical edits happens at the cost of diverging semantically from the input sentences. Moreover, we observe that the average lexical differences within NMT -based systems are small. This indicates that different systems perform similar edit operations that do not deviate a lot from the input sentence in terms of their lexical choices. This is unfortunate given that multilingual F o ST requires systems to perform more phrase-based operations, as shown in the analysis in 4. Evaluation Metric While evaluating evaluation metrics is not a goal of this work (though the data can be used for that purpose), we observe that the top models identified by the automatic metrics generally align with the top models identified by humans. While promising, further work is required to confirm if the automatic measures really do correlate with human judgments. This work extends the task of formality style transfer to a multilingual setting. Specifically, we contribute XFORMAL , an evaluation testbed consisting of informal sentences and multiple formal rewrites spanning three languages: BR-PT , FR , and IT . As in Rao and Tetreault (2018) Turkers can be effective in creating high quality ST corpora. In contrast to the aforementioned EN corpus, we find that the rewrites in XFORMAL tend to be more diverse, making it a more challenging task. Additionally, inspired by work on cross-lingual transfer and EN F o ST , we benchmark several methods and perform automatic and human evaluations on their outputs. We found that NMT -based ensembles are the best performing models for FR and BR-PT a result consistent with EN however, they perform comparably to a naive RULE-BASED baseline for IT . To further facilitate reproducibility of our evaluations and corpus creation processes, as well as drive future work, we will release our scripts, rule-based baselines, source data, and annotation templates, on top of the release of XFORMAL . Our results open several avenues for future work in terms of benchmarking and evaluating F o ST in a more language inclusive direction. Notably, current supervised and unsupervised approaches for EN F o ST rely on parallel in-domain datawith the latter treating the parallel set as two unpaired corporathat are not available in most languages. We suggest that benchmarking F o ST models in multilingual settings will help understand their ability to generalize and lead to safer conclusions when comparing approaches. At the same time, multilingual F o ST calls for more language-inclusive consideration for automatic evaluation metrics. Model-based approaches have been recently proposed for evaluating different aspects of ST . However, most of them rely heavily on English resources or pretrained models. How those methods can be extended to multilingual settings and how we evaluate their performance remain open questions. We thank Sudha Rao for providing references and materials of the GYAFC dataset, Chris Callison-Burch and Courtney Napoles for discussions on MTurk annotations, Svebor Karaman for helping with data collection, our colleagues at Dataminr, and the NAACL reviewers for their helpful and constructive comments. Finally, we address ethical considerations for dataset papers given that our work proposes a new corpus XFORMAL . We reply to the relevant questions posed in the NAACL 2021 Ethics FAQ . 16 7.1 Dataset Rights The underlying data for our dataset as well as training our F o ST models and formality classifiers are from Yahoo! Answers L6 dataset. We were granted written permission by Yahoo (now Verizon) to make the resulting dataset public for academic use. Turkers are paid over 10 USD an hour. We targeted a rate higher than the US national minimum wage of 7 . 50 USD given discussions with other researchers who use crowdsourcing. We include more information on collection procedures in 3. This question is not applicable for our work. We follow Bender and Friedman (2018) and Gebru et al. (2018) and report characteristics in 3 and 4.
objective abstain abstain abstain result abstain abstain abstain other objective objective objective objective objective objective objective objective abstain abstain abstain abstain abstain abstain objective other result method other objective other other other other other other other other other method method method method abstain method method method abstain abstain other abstain abstain method method abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain other abstain abstain abstain abstain method method abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain method abstain abstain result other result result other abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain other other abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain result abstain abstain result abstain abstain abstain result abstain abstain abstain result result abstain abstain abstain abstain abstain abstain abstain result abstain abstain result result abstain abstain abstain abstain objective objective abstain objective method result abstain result abstain method abstain abstain abstain objective other abstain abstain method abstain abstain abstain abstain abstain method method