Understanding human preferences, along with cultural and social nuances, lives at the heart of natural language understanding. Concretely, we present a new task and corpus for learning alignments between machine and human preferences. Our newly introduced problem is concerned with predicting the preferable options from two sentences describing scenarios that may involve social and cultural situations. Our problem is framed as a natural language inference task with crowd-sourced preference votes by human players, obtained from a gamified voting platform. We benchmark several state-of-the-art neural models, along with BERT and friends on this task. Our experimental results show that current state-of-the-art NLP models still leave much room for improvement. The ability to understanding social nuances and human preferences is central to natural language understanding. This also enables better alignment of machine learning models with human values, eventually leading to better human-compatible AI applications (Peterson et al., 2019; Leslie, 2019; Rosenfeld and Kraus, 2018; Amodei et al., 2016; Russell and Norvig, 2016). There exist a plethora of work on studying optimal decision-making under a variety of situations (Edwards, 1954; Bottom, 2004; Plonsky et al., 2019; Peterson et al., 2019). On the other hand, cognitive models of human decision-making are usually based on small datasets (Peterson et al., 2019). Furthermore, these studies tend to only consider individuals in isolation. In contrast, we investigate the influence of cultural and social nuances for choice prediction at scale. In other words, we study the social preference as a whole,  First two authors contributed equally not those of an individual in isolation, which is arguably more challenging and largely unexplored. In this work, we propose a new benchmark dataset with a large number of 200k data points, M achine A lignment with C ultural values and S ocial preferences (MACS), for learning AI alignment with humans. Our dataset is based on a popular gamified voting platform, namely the game of would you rather?'. In this game, participants are given two choices and vote for the more preferable option. Examples from our dataset can be found at Table 1. To the best of our knowledge, our work is the first work to incorporate voting-based language games as a language understanding benchmark. In many ways, our benchmark dataset is reminiscent of the natural language inference problem (MacCartney, 2009; Bowman et al., 2015), social commonsense reasoning (Sap et al., 2019) or other natural language understanding problems (Wang et al., 2018; Zellers et al., 2018). To this end, our problem is framed in a way that enables convenient benchmarking of existing state-of-the-art NLU models such as BERT (Devlin et al., 2018) or RoBERTa (Liu et al., 2019). That said, unlike many NLU datasets that rely on few annotators, the key differentiator lies in the fact that our dataset aggregates across hundreds or thousands and beyond for each data point. Options are also crowd-sourced and gamified which may encourage less monotonic samples, ie., encouraging players to come up with questionss that are difficult for other players. Additionally, our dataset comprises of country-level statistics, which enable us to perform cultural-level prediction of preferences. We propose a new NLU benchmark based on an online gamified voting platform. We propose several ways to formulate the problem, including absolute and relative preference prediction. We also introduce a cultural-level NLU problem formulation. We investigate state-of-the-art NLU models such as BERT (Devlin et al., 2018), RobERTA (Liu et al., 2019) and XLNET (Yang et al., 2019) on this dataset. Empirical results suggests that our benchmark is reasonably difficult and there is a huge room for improvement. We look to crowdsourcing platforms to construct our dataset. Our dataset is constructed from https://www.rrrather.com/ , an online platform 1 for gamified voting. The platform is modeled after the famous internet game would you rather? , which pits two supposedly comparable choices together. Whenever a player votes, their vote is recorded in the system. Players generally vote to see how well their vote aligns with the majority and consensus with everyone else. We provide samples of the problem space in Table 1. We crawled data from the said platform and fil-tered away posts with less than 500 total votes. In total, we amassed 194,525 data points, which we split into train/dev/test splits in an 80/10/10 fashion. Dataset statistics are provided in Table 2. Train Dev Test Total Data 155,621 19,452 19,452 194,525 (cid:96) max 678 351 298 (cid:96) mean 8 8 8 (cid:96) min 1 2 2 Table 2: Dataset statistics of the MACS dataset. This section outlines the benefits of our proposed dataset as a language understanding benchmark. 1 The authors have obtained written permission from the owner of the platform to crawl and use their data for academic research. The questions, answers or discussions do not represent opinions of the authors in this paper. (1) Understanding before Interaction. In our dataset and problem formulation, complex understanding of each option text is often required first before modeling the relative preference between two options. This is unlike NLI or question-answering based NLU benchmarks, where matching signals can be used to predict the outcome easily. In our dataset and task, it is imperative that any form of word overlap can be hardly used to determine the outcome. (2) A good coverage of social preferences. Upon closer inspection of our proposed benchmark, we find there is a good representation of samples which cover social and cultural themes. Social preferences (such as the preference of brands) are captured in samples such as example (6). (3) Completely natural. Our MACS dataset completely exists in the wild naturally . This is unlike datasets that have to be annotated by mechanical turkers or paid raters. In general, there is a lack of incentives for turkers to provide high-quality ratings, which often results in problems such as annotation artifacts. Unlike these datasets, our MACS dataset completely exists in the wild naturally. The choices are often created by other human players. Hence, in the spirit of competitiveness, this means that the data is meant to be deliberately challenging. Moreover, there are at least 500 annotators for each sample, which makes the assigned label less susceptible to noisy raters. Given Q (prompt), two sentences S 1 and S 2 and V ( . ) which computes the absolute votes to each option, we explore different sub-tasks (or variant problem formulation). Predicting Preference This task is primarily concerned with predicting if V ( S 1) > V ( S 2) or otherwise. Intuitively, if a model is able to solve this task (perform equivalent to a human player), we consider it to have some fundamental understanding of human values and social preferences. We frame this task in two ways. The first is a straightforward binary classification problem, i.e., V ( S 1) > V ( S 2) . The second task is a three-way classification problem with a third class predicting if the difference | V ( S 1)  V ( S 2) | is less than 5% of the total votes. In short, this means that two options are almost in a draw. Standard Cultural Binary Three-way Binary Three-way Model Dev Test Dev Test Dev Test Dev Test BERT 61.02 60.38 56.71 55.85 62.42 62.88 57.42 58.21 XLNEt 56.12 56.84 55.72 56.34 51.77 51.42 57.08 57.39 RoBERTa 64.75 64.15 61.04 61.19 64.39 64.71 59.28 61.22 Table 3: Experimental results on predicting preference (standard and cultural) with BERT (Devlin et al., 2018), XLNEt (Yang et al., 2019) and RoBERTa (Liu et al., 2019) on MACS dataset. Predicting Cultural Preferences We consider a variant of the preference prediction problem. Our MACS dataset has culture-level preference votes which are the voting scores with respect to a particular cultural demographic. We extend the same setting as Task 1 by requiring the model to produce culture-level predictions. In order to do this, we prepend the input sentence with a culture embedding token. For example, Input = [Culture] + [Choice A] + [Sep] + [Choice B]. The task is identical, predicting the greater of Choice A OR Choice B, with respect to the cultural ground truth. The dataset is augmented at the culture level and the same example is duplicated for each culture, e.g., we duplicate the sample for countries 'USA' and 'Europe'. We consider only culture-level votes with a threshold above 25 votes in the dataset for train/dev/test sets. Predicting Relative Preference The third variant is a fine-grained regression task where we want to identify if our model is able to learn the extent of preference given by human players. This problem is framed as a regression problem that is normalized from [0 , 1] with respect to the total number of votes in the data point 3 Experiments This section outlines our experimental setup and results. We implement and run several models on this dataset. (1) BERT (Devlin et al., 2018) Deep Bidirectional Transformers is the state-of-the-art pretrained transformer model for a wide range of NLP tasks. (2) XLNet (Yang et al., 2019) is a large pretrained model based on Transformer-XL. (3) RoBertA (Liu et al., 2019) is a robustly optimized improvement over the vanilla BERT model. All models were run using the finetune methodology using the standard Pytorch Huggingface 2 repository. We train (finetune) all models for 3 epochs using the default hyperparameters.. Metrics The evaluation metrics for classification tasks is the standard accuracy score. For regression tasks, we use the correlation, Pearson, and Spearman metrics. Table 3 reports our results on binary and three-way classification on the MACS dataset. In general, we find that RoBERTa performs the best. However, in most cases, the performance of all three models still leaves a lot to be desired. An accuracy of 60%+ shows that state-of-the-art models still struggle at this task. On the other hand, results on regression task are also similarly lacklustre, and 2 https://github.com/huggingface/ transformers Dev Test Model Correlation Pearson Spearman Correlation Pearson Spearman BERT 0.234 0.256 0.214 0.229 0.250 0.208 XLNEt 0.225 0.243 0.206 0.228 0.250 0.206 RoBERTa 0.258 0.279 0.236 0.256 0.278 0.235 Table 4: Experimental results on predicting relative preference on MACS dataset. show that models like BERT and RoBERTa are unable to perform well on this task. On a whole, it is good to note that RoBERTa performs the best out of the three compared models. Overall, this encourages further research on cultural and social commonsense reasoning in the current state-of-the-art in natural language understanding. All in all, we hope our benchmark serves as a useful tool for understanding the social capabilities of these models. Table 5 reports some sample of our model outputs, shedding light on examples in which our model does well and otherwise. We observe that the model often gets the answer wrong even when the ground truth is overwhelmingly swayed towards one side. On the other hand, occasionally, we also observe that the model can get questionable questions such as (4) and (5) correctly even despite the tight draw between human voters. We propose MACS (Machine Alignment with Cultural and Social Preferences), a new benchmark dataset for learning machine alignment with human cultural and social preferences. MACS encompasses and requires social and cultural reasoning to solve and an overall holistic understanding of humanity. It is designed to be challenging where state-of-the-art NLP models still struggle at  60% . In this paper, we are not promoting the use of https://www.rrrather.com/ as the training source, but rather the study of the alignment of machine learning models with social preference of a large population. Unfortunately, there might be some issues of bias, fairness and representation due to the curation of the training data from Internet, which might lead models to give prejudiced or stereotyped outputs. Evaluating bias, fairness and representation in language models and the training data is an important research area (Nadeem et al., 2020; Huang et al., 2019). As for future works, it is important to characterize and intervene biases when designing such tasks.
abstain objective method result abstain result abstain abstain abstain abstain abstain objective objective objective abstain abstain result objective abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain method other abstain abstain abstain abstain method abstain method abstain abstain abstain abstain abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other method abstain abstain method abstain abstain abstain method method abstain method abstain abstain abstain abstain method abstain abstain method result result abstain abstain other abstain abstain abstain abstain abstain abstain abstain objective abstain abstain other abstain abstain abstain