We propose T uring A dvice , a new challenge task and dataset for language understanding models. Given a written situation that a real person is currently facing, a model must generate helpful advice in natural language. Our evaluation framework tests a fundamental aspect of human language understanding: our ability to use language to resolve open-ended situations by communicating with each other. Empirical results show that today's models struggle at T uring A dvice , even multibillion parameter models finetuned on 600k in-domain training examples. The best model, a finetuned T5, writes advice that is at least as helpful as human-written advice in only 14% of cases; a much larger non-finetunable GPT3 model does even worse at 4%. This low performance reveals language understanding errors that are hard to spot outside of a generative setting, showing much room for progress. Language models today are getting ever-larger, and are being trained on ever-increasing quantities of text. For an immense compute cost, these models like T5 (Ra  el et al., 2019) and GPT3 (Brown et al., 2020) show gains on a variety of standard NLP benchmarks  often even outperforming humans. Yet, when a giant model like T5 generates language, we observe clear gaps between machine-level and human-level language understanding  even after it has been finetuned for the task at hand. Consider Figure 1, in which a woman asks for advice. She is assigned to dissect an animal for her class project, but has extreme anxiety about dead animals  and her teacher refused to give her another assignment. Humans can respond with helpful advice, reflecting our unique ability of real-world language use : to communicate and tackle open-ended issues. The helpful advice in this ex-T5 I'd send a short email to the next higher-up authority figure, ideally a counselor. ample but not the only one possible suggests that she send a short email to her guidance counselor. On the other hand, not only is T5's advice unhelpful, it also reveals key misunderstandings of the situation. It seems to believe that the student is asking the teacher to do a class project involving dead animals. This reading comprehension error is particularly strange, as T5 outperforms humans on a variety of reading comprehension benchmarks. Others in the community have observed similar issues, raising concerns about what today's benchmark datasets measure (Yogatama et al., 2019; Kryscinski et al., 2019; McClelland 1 et al., 2019; Gardner et al., 2019). We argue that there is a deep underlying issue: a gap between how humans use language in the real world, and what benchmarks today can measure. Today's dominant paradigm is to study static datasets, and to grade machines by the similarity of their output with predefined correct answers. For example, we score multiple choice exams by how often the correct answers are chosen, and evaluate generative tasks like machine translation by similarity with respect to correct translations. However, when we use language in the real world to communicate with each other  such as when we give advice, or teach a concept to someone  there is rarely a universal correct answer to compare with, just a loose goal we want to achieve. We introduce a framework to narrow this gap between benchmarks and real-world language use. We propose to evaluate machines by their success in using language to (1) communicate with humans in (2) tackling complex, open-ended, real-world situations. Our goal is a machine that, like a human, can generate language that is useful and helpful. Doing so necessarily requires a deep understanding of language and the world, as per a line of thought that the complete meaning representation is one that su  ces to complete a task (Artzi et al., 2013). As a case-study of our framework, we introduce T uring A dvice as a new grand challenge for AI systems. A machine reads a situation written by a person seeking advice, like Figure 1, and must then write advice that is helpful to the advice-seeker. Like a Turing Test (Turing, 1950), we establish a simple condition required for a model to pass': model-generated advice must be at least as helpful to the advice-seeker as human-written advice. We make our challenge concrete by introducing a new dataset, R eddit A dvice , and accompanying leaderboard. We tie our dataset to the Reddit community, which resolves two additional sources of bias. First, Reddit users are intrinsically motivated, seeking advice about highly complex real issues  which past work suggests di  er from hypothetical issues that crowd workers might come up with (e.g. Kwiatkowski et al., 2019; Gurari et al., 2018). Second, we make our dataset dynamic , not static  models are evaluated over Reddit situations posted over the previous two weeks at the time of submission. Models therefore, like humans, must generalize to new situations and patterns of language. incredibly challenging for NLP models. Today's largest finetunable model, T5 with 11 billion parameters, produces advice that is preferable to human-written advice 14.5% of the time  after being finetuned on 600k examples. GPT3, an even larger model with 175 billion parameters that was not released for finetuning, does even worse at 4%. Even more concerning, our evaluation finds that it often generates hateful and toxic language. We also study our task from the perspective of today's standard core' NLP tasks. Broadly, we find that machines frequently confuse who is who, are self-contradictory, or seem to miss important world knowledge. However, these mistakes tend not to fall into the neat categories defined by standard task definitions. We address this by introducing diagnostic questions, which systematically measure these language understanding errors. In summary, our paper makes three contributions. First , we introduce a new framework for measuring language understanding through directly tackling real-world language problems. Second , we introduce T uring A dvice as a new challenge for AI systems, along with a dynamic dataset and leaderboard. Third , we connect our task to existing atomic NLP tasks, introducing a new setting that reveals where progress is still needed. We propose to evaluate machines by their success at real-world language use : using language to communicate with a human, in response to a naturally occurring situation, in order to achieve a desired outcome. This is how educators often measure (hu-man) language understanding of a second language  by how well the learner can use the language (Council of Europe, 2001). Our approach is also inspired by Wittgenstein's notion of semantics, that meaning is use: language is grounded in our desire to make sense of one another and cooperate to meet our needs (Wittgenstein, 1953). As machines do not have humanlike needs or desires, we propose to evaluate machines' success at a task by how well it serves a human who is interested in the outcome. For example, if a machine orders food on my behalf, then I can evaluate it based on whether I enjoy the dish it ordered. Though this requires careful task selection in order to make things feasible for current models, as we will show in Section 3, it results in a powerful and reliable human evaluation. Our evaluation relates to pragmatics in NLP, where communication is modeled also through listeners and speakers (Golland et al., 2010; Frank and Goodman, 2012). One approach is to introduce a communication game, with an explicit objective. For example, Wang et al. (2016) study a blocks world where humans give commands to a block-placing machine. The machine is then graded on accuracy. Our proposed evaluation instead covers complex everyday scenarios faced by a human, where the objective is to help them as much as possible. Pragmatics can also be studied through machine-machine communication; e.g., through emergent language (Lazaridou et al., 2017). Recent work uses pretrained question-answering models to evaluate summarization models (Chen et al., 2018; Scialom et al., 2019; Eyal et al., 2019; Vasilyev et al., 2020). However, ensuring that machines communicate in standard English is di  cult, as there is usually a more e  cient machine-language coding scheme for the task (Kottur et al., 2017). Quality of generations. The first approach studies generative tasks like chit-chat dialogue or story-writing, and measures the inherent quality of generations , often through attributes such as sensi-bleness and specificity (e.g., Venkatesh et al., 2018; Hashimoto et al., 2019; Adiwardana et al., 2020). This approach is orthogonal to ours: though these attributes might be desirable, they are often insu  cient to guarantee success at a task. Correctness. The second (and perhaps more common) approach is to evaluate models through correctness over static datasets. For example, machines can be graded by the similarity of their generated translation to correct translations, 1 or, by how often they choose the correct answer on a multiple choice exam. Many goal-oriented dialogue and semantics tasks are also evaluated in this way, as a model is evaluated by whether it makes the correct API call, or produces a correct parse. Since many language tasks cannot be evaluated through correctness, researchers often introduce 1 Models submitted to the 2019 Conference on Machine Translation were evaluated (by humans) on how well the model's translations agreed with either (1) human-written translations, or, (2) original source text (Barrault et al., 2019). proxy tasks that are easy to evaluate, while (hope-fully) correlating with the underlying true task. For example, SWAG (Zellers et al., 2018) is a multiple-choice proxy task and dataset introduced to study the true task of commonsense reasoning. However, there are gaps between datasets for proxy tasks (e.g. multiple choice), and the core tasks they seek to represent (e.g. commonsense reasoning), which we discuss in the next sections. When we reduce a complex language task to a simplified setup, with a small label space (like multiple-choice classification), we run the risk of introducing artifacts and biases: patterns that can be exploited in the simplified setup, but that are not representative of the true task (Gururangan et al., 2018; Zellers et al., 2019a). Artifacts can enable machines to even outperform humans at the final benchmark, without solving the underlying task. While the problem of artifacts has recently taken the spotlight in the NLP community, partially because large Transformers (Vaswani et al., 2017) excel at picking up on artifacts, there is a deeper underlying issue. One way to view simplified tasks is that in order to correctly map inputs X to labels Y , a machine must learn a set of attributes A that are representative of the true' task. We can upper-bound the information contained by A through the information bottleneck principle of Tishby et al. (1999). An e  cient model minimizes the following, for some   0: min p p a | x q I p X ; A q   I p A ; Y q , (1) where I is mutual information. In other words, the model will learn attributes A that maximally compress the inputs X (minimizing I p X ; A q ), while also remaining good predictors of the labels Y (max-imizing I p A ; Y q ). However, the label prediction term is bounded by the information (or entropy, H ) of the label space: I p A ; Y q  H p Y q  H p Y | A q  H p Y q . (2) Thus, for a task with a small label space, there is no guarantee that a model will learn high-information content attributes. Models are in fact encouraged to overfit to dataset artifacts, and to unlearn linguistically useful information that is not directly relevant to predicting Y (Pereira, 2000). 3 An alternate approach is to make datasets harder adversarially, so as to have fewer artifacts (Zellers et al., 2018, 2019a; Le Bras et al., 2020). However, it might be impossible to make a dataset with no artifacts, or to know if one has been created. Our proposal, to evaluate models by their real-world language use, addresses the information bottleneck issue in two ways. First, when we use language in the real world, the mapping between possible inputs and outputs is often highly complex. For example, the space of possible advice is vast, and many pieces of advice might be equally helpful given a situation. Second, we directly tackle language problems, without introducing a correctness-based proxy that machines might overfit to. 2.3 Static datasets in a dynamic world To evaluate performance on a real-world task by means of a dataset, we (implicitly) assume that the dataset is a good representation of the world (Torralba and Efros, 2011). This might be questionable when it comes to real-world language use, as static datasets necessarily capture historic patterns of language. For instance, syntactic understanding is often evaluated using the Penn Treebank, with news articles from 1989 (Marcus et al., 1993). However, the world is constantly evolving, along with the language that we use. To bridge this gap, we propose to evaluate machines by their interactions with humans in the present . Models therefore must learn to perform the underlying language task, even for novel situations, rather than fitting to the historic distribution of a fixed test set. We make this notion concrete in the next section, where we introduce a dynamic dataset and leaderboard for evaluating advice. 3 T uring A dvice : a New Challenge for Natural Language Understanding As a case study of our framework, we introduce T uring A dvice , a new challenge task for AI systems to test language understanding. The format is simple: given a situation expressed in natural language, a machine must respond with helpful advice. To pass the challenge, machine-written advice must be at least as helpful to the advice-seeker as human-written advice, in aggregate. We focus on advice for a few reasons. First, advice-giving is both an important and an everyday task. People ask for and give advice in settings as diverse as relationship advice and tech support (Bonaccio and Dalal, 2006). Thus, we as humans have inherent familiarity with the task, and what it means for advice to be helpful  making it easy to evaluate, as we later show empirically. Moreover, because there are many internet communities devoted to advice-giving, training data is plentiful. Second, the framework of advice-giving allows us to study subtasks such as reading comprehension and natural language inference (Section 5.3); we argue both of these are needed to consistently give good advice. Learning to recognize advice has recently been studied as an NLP task on its own (Govindarajan et al., 2020), though we are not aware of past work in learning to generate advice. 3.1 R eddit A dvice : A dynamic dataset for evaluating advice We propose to evaluate models dynamically , through new situations and advice that are posted to Reddit. We call our dynamic dataset R eddit A dvice . Many of Reddit's subcommunities (or subreddits') are devoted to asking for and giving advice, with subreddits for legal, relationship, and general life advice. 2 During evaluation time, we will retrieve new situations from Reddit as a new test set for models. Workers on Mechanical Turk then grade the model-written advice versus the Reddit-endorsed human-written advice. 3.1.1 How advice-giving works on Reddit Suppose a Reddit user faces an issue that they are seeking advice about. First, they write up situation and post it to an advice-oriented subreddit. Users then reply to the situation , o  ering advice . Importantly, any user can upvote' or downvote' the advice as well as the situation itself changing its score slightly. Top-scoring advice is deemed by the wisdom of the crowd as being the most helpful. 3 3.1.2 The ideal evaluation through Reddit? In a sense, human advice-givers are evaluated' on Reddit by the score of their advice  representing how well their advice has been received by the community. Similarly, the ideal model evaluation might be to post advice on Reddit directly. If the model writes helpful advice, it should be upvoted. 2 We use advice from the following subreddits: Love , Relationships , Advice , NeedAdvice , Dating_Advice , Dating , Marriage , InternetParents , TechSupport , and LegalAdvice . 3 This is somewhat of a simplification, as other factors also influence what gets upvoted (Anderson et al., 2012; Lakkaraju et al., 2013; Muchnik et al., 2013; Jaech et al., 2015). 4 1. Which piece of advice is more helpful? Situation Given: Advice A Advice B Definitely B Slightly B Slightly A Definitely A 2. How helpful is the worse advice (A) to the question-asker? Slightly helpful Not helpful Dangerous 3. Is Advice A worse mainly due to its meaning, or its writing? Meaning Writing 3. Could Advice A be applicable to (and helpful in) a di  erent situation? Possibly helpful Never helpful Figure 2: Crowdsourcing workflow. Mechanical Turk Workers are given a situation, and two pieces of advice. First, they choose which is more helpful (here, B). Second, they rate the helpfulness of the worse advice (A); last, they answer a diagnostic question. However, there is a significant ethical problem with this approach. The users who post advice questions are real people, with real problems. A user might read advice that was originally written by a machine, think it was human-endorsed, and do something harmful as a result. For this reason, we take an alternate crowdsourcing approach. 3.1.3 A crowdsourced, hybrid evaluation  through Mechanical Turk We propose a hybrid approach for dynamic evaluation of models. While the situations, and reference advice come from Reddit, we hire workers on Mechanical Turk to rate the relative helpfulness of machine-written advice. Not only is this format more ethical, it also lets us collect diagnostic ratings, allowing us to quantitatively track the natural language understanding errors made by machines. We made our crowdsourcing task as fulfilling as possible using popular situations from Reddit, and pitching the work in terms of helping people. We received feedback from many workers that our tasks were entertaining and fun, suggesting that our workers are to some degree intrinsically motivated. 3.1.4 Mechanical Turk annotation setup In a single round of evaluation, we retrieve 200 popular Reddit situations that were posted in the last two weeks. For each situation, we retrieve the top-rated advice from Reddit, and generate one piece of advice per model. Workers on Mechanical Turk then compare the helpfulness of the model-generated advice with human-written advice, and provide diagnostic ratings. We show an overview of our Mechanical Turk task in Figure 2. A worker is given a situation and two pieces of advice. One is the top-scoring advice from Reddit, and the other is model-generated advice; the worker is not told which is which. The worker first chooses the more helpful piece of advice, then provides diagnostic information for the less helpful advice  rating it Slightly helpful , Not helpful , or Dangerous . If the worse piece of advice was Slightly helpful , they choose whether it is worse due to a Meaning problem or a Writing problem . Otherwise, they choose if the worse advice could be Possibly helpful in some other situation, or Never helpful in any situation. Three workers rate each model-situation pair, and ratings are combined using a majority vote. We follow best practices on Mechanical Turk, using a qualification exam, paying workers at least $15 per hour, and giving feedback to workers. Still, evaluation is highly economical at $1.86 per example-model pair, or roughly $400 per model evaluated. 3.2 A large static dataset for training We present RedditAdvice2019, a large static dataset for training advice-giving models. Because today's models have extreme reliance on data for finetuning, we collect data that is in the exact same format as R eddit A dvice , yet we expand our selection criteria, optimizing for recall rather than precision (Supp A.2). In total, we extract 616k pieces of advice, over 188k situations. To mirror the dynamic nature of the evaluation, in which models are evaluated on situations posted in 2020 and beyond, we split our dataset into static training and validation sets by date. 4 4 Experimental Results on R eddit A dvice In this section, we report results from one round of dynamic evaluation on R eddit A dvice . We evaluate the following strong NLP models and baselines: a . Rule-based: a templated system to give legal, relationship, or life advice. The system first chooses randomly empathetic sentence from ten choices, for example I'm sorry you're facing this. It then chooses a random piece of advice that is loosely related to the situa-tion's topic; we infer this from the subreddit the situation was posted on. For example, for 4 Our training set contains 600k pieces of advice from July 2009 to June 14, 2019; validation contains 8k from June 14 to July 9th 2019. LegalAdvice the model might write I'd suggest getting a lawyer immediately. b . TF-IDF retrieval: for a new situation, we compute its TF-IDF bag-of-word vector and use it to retrieve the most similar situation from the training set. We then reply with the top-scoring advice for that situation. c . Grover-Mega (Zellers et al., 2019b): a left-to-right transformer model with 1.5 billion parameters. Grover was pretrained on news articles with multiple fields, perhaps making it a good fit for our task, with multiple fields of context (like the subreddit, date, and title). Our situation-advice pairs are often quite long, so we adapt Grover for length; pretraining it on sequences of up to 1536 characters. d . T5 (Ra  el et al., 2019): a sequence-to-sequence model with a bidirectional encoder and a left-to-right generator, with 11 billion parameters. T5 was trained on a large dataset of cleaned web text. At the time of writing, T5 is the top-scoring model on the Glue and SuperGlue benchmarks (Wang et al., 2019b,a), scoring above human performance on Glue and near human-performance on SuperGlue. e . GPT3 (Brown et al., 2020): a left-to-right transformer model with 175 billion parameters. GPT3 must be prompted to generate advice since it has not been released for finetuning. We cannot provide few-shot examples in the prompt due to the length of situation-advice pairs; we instead mimic the formatting of a website quoting from Reddit (Appendix B.5). Last, to quantify the measurement error of our evaluation, we additionally evaluate: f . the second -highest rated Reddit advice for each situation. We send this advice through the same pipeline as machine-written advice. We finetune all models (except GPT3) and generate using Nucleus Sampling (Holtzman et al., 2020); more details in Appendix B. In our study, we exclude purely bidirectional models, such as BERT (Devlin et al., 2019). While these models can be made to generate text, these generations are usually worse than those of left-to-right models (Wang and Cho, 2019). T5 also tends to outperform them, even on discriminative tasks. In Figure 3, we show overall results for one evaluation trial, which featured 200 situations posted on Reddit from October 28 to November 7, 2020. As a key metric for measuring the relative usefulness of model-written advice, we evaluate the frequency by which workers prefer the Reddit-written reference advice over the model-written advice. If a model's advice was just as helpful as human advice in aggregate, then that model would score 50%. Model performance is quite low. The best model, T5-11B, scores 14.5%, outperforming a smaller Grover-Mega (4.5%); GPT3 does worse at 4.0%. competitive at 2.5% and 4.0% accuracy respectively. As additional comparison to the 50% upper bound, the second-highest scoring Reddit advice scores 41%. This suggest that our workers and often prefer the same advice as Reddit users. To investigate the measurement error of our evaluation, in Figure 4 we report the statistical significance between pairs of models; details about how this is computed are in Appendix C. We observe a large gap in performance between T5 and the other baselines. For example, its improvement over Grover-Mega is 10%, which is highly statistically signficant. On the other hand, the di  erences in performance between other models are more minor  GPT3 does not outperform TF-IDF, and though it outperforms the rule-based system by 1.5%, it is only somewhat statistically significant. Overall, the statistical significance results suggest that our evaluation can stably rank model performance. This, along with the finding that model performance is low on our task suggests that there is ample room for growth on R eddit A dvice . So far, we have shown that we are able to reliably evaluate models in our dynamic setup, and that doing so results in model performance that is significantly lower than human performance. To break down what this gap in performance means, we show a qualitative example in Figure 5. A user is asking for online legal advice about being stuck at work for their entire 4pm-midnight shift  with no eating allowed due to COVID-19. The top-rated Reddit advice understands this situation and then o  ers advice, suggesting the advice-seeker organize alongside other workers  as New Jersey doesn't require employers to give any meal breaks. Machine advice seems to misunderstand the issue. T5 asks if there is a reason that you can't leave the property, even though this reason is stated in the situation. GPT3's advice is self-contradictory; it also makes something up about a restaurant in the building. As part of our evaluation, we wish to quantitatively measure problems with machine-written advice. Recall that in our crowdsourcing setup (Sec-tion 3.1.3), three workers select which advice is legaladvice : NJ Work will not let us eat in the building due to covid outbreak. We can't leave the property for breaks. They have a tent outside for us to eat in but it's getting cold out. I am a manager at my work part of my responsibilities is to stay on the property during my entire shift we are given a 1 hour paid lunch break that we do not have to clock out for. However due to recent covid outbreak the new policy is no eating in the building. They had set up a tent outside for us to eat in but it is starting to get cold out as well there are not heaters in there. My question is can they legally do this to us? Also I work 4:00 p.m. to midnight shift. TopRedditadvice So by "clock out for" you are not a salaried exempt employee? Do the workers under you have a union? Aside from the cold, New Jersey doesn't require employers to give *any* meal breaks for employees 18 or older, so they could take away the breaks and the tent if they wanted, and just say "no eating" (though water is required). What is your relationship with the other managers, and room to figure this out, ask for heaters, etc.? RuleBased If I were you, I'd get a lawyer immediately. De(cid:28)nitely less helpful Slightly helpful Meaning problem TF-IDFRe-trieval You were blackout drunk, which means that you were not capable of consent. ...If anyone should feel violated, it's you. De(cid:28)nitely less helpful Not helpful Possibly helpful Grover-Mega If you are required to be on the premises, they are required to provide you with a place to eat. https: // www.nj.gov / labor / es / faq.html#13 De(cid:28)nitely less helpful Not helpful Possibly helpful T5-11B If your shift is from 4pm to midnight, is there a reason that you can't leave the property? De(cid:28)nitely less helpful Not helpful Never helpful GPT3-175B A place of employment is considered the "premises". You can't just assume that because you can't walk out the door that you can eat wherever you want. This is work hours. You're not in your home. The restaurant, or at least the food, is all provided for you for your benefit. But if there are other people there, then I could see how it could be a safety issue. better, and then annotate problems with the worse piece of advice. We found workers had high agreement during the diagnostic annotation. 5 In Figure 6, we show the distribution of the ratings for model-written, versus human-written advice. Machine-written advice that was 5 For the classifying machine-written advice as helpful' versus not helpful' or dangerous' (combining the two latter categories into one), we have   0 . 689. For breaking down helpful advice into meaning problem' versus a writing problem', we have Cohen's   0 . 613; for rating unhelpful advice as possibly helpful' versus never helpful,' we have   0 . 602. 7 TF-IDFRetrieval GPT3-175B T5-11B Second-bestReddit advice 0% 20% 40% 60% 80% 100% 4% 4% 14% 41% 9% 15% 26% 32% 5% 13% 22% 19% 66% 23% 21% 4% 10% 33% 13% 2% 4% 10% 3% 1% Frequency (%) of advice ratings Preferred over top-rated Reddit advice Slightly helpful (with a writing problem) Slightly helpful (with a meaning problem) Not helpful (possibly helpful elsewhere) Not helpful (never helpful elsewhere) Dangerous Figure 6: Distribution of ratings for three models: TF-IDF retrieval, GPT3, and T5, along with ratings for the second-best rated Reddit advice. Though deep generators like GPT3 and T5 are often preferred over the retrieval baseline, they also often write advice that would never be helpful (33% GPT3, 13% T5), and that is racist, sexist, or otherwise dangerous (10% GPT3, 3% T5). not preferred over human-written advice can have the following ratings. It can be rated as Slightly helpful (but, was rated as worse mainly due to a Meaning problem or Writing problem ), as Not helpful , or Dangerous . The diagnostics show several patterns. First, all models frequently commit natural language understanding errors, such as internal contradiction. Because of this, we find that TF-IDF bag-of-words retrieval is competitive with that of large generators. While retrieved advice is often irrelevant (66% of the time), it is almost never complete gibberish, as it comes from top-scoring advice. Only 10% of workers rated this advice as Not helpful for any situation, less than T5. Second, they suggest that models struggle even more without finetuning. A GPT3 model with careful prompting generates language that is Dangerous 10% of the time. These qualitative and quantitative results confirm a pattern observed by many others, that large language models like GPT3 often generate explicitly racist and sexist language out-of-the-box Sheng et al., 2019; Gehman et al., 2020; Bender et al., 2021, among others). We explore this further in Supplemental F. This is perhaps worrying, since GPT3 is presently being commercialized. 5.2 A Leaderboard for Advice Evaluation So far, we have shown results from one evaluation round; a second is in Supplemental D. We propose a dynamic leaderboard to keep that evaluation ongoing, at rowanzellers.com/advice . Users submit a model API to be dynamically evaluated. Each new model, along with the highest rated previously-evaluated model, will be evaluated for an additional round using the same approach. The cost of each evaluation is reasonable (Section 3.1.4), which we authors will pay in the short term. An alternative strategy requires submitters to pay the Mechanical Turk fees themselves; this model was used for the HYPE leaderboard in computer vision (Zhou et al., 2019). 5.3 Relation to existing NLP tasks Shared core tasks such as reading comprehension and natural language inference are of considerable interest to the NLP community. Many datasets have been proposed for these tasks, and progress on them is often measured through auto-gradeable correctness metrics. However, large models have started to outperform humans on these datasets, raising doubt that further progress on them brings us closer to human-level language understanding. We argue two things: first, that many NLP tasks are necessary components of giving advice, and second, that because giving advice remains far from solved, these tasks are also far from solved. In Appendix F, we study problems with advice from T5-11B from the point of view of existing NLP tasks. For instance, machine advice often contradicts itself, suggesting that today's systems struggle with the general task of natural language inference. We have made these diagnostics publicly available to enable progress on automatically spotting these mistakes. We introduced new methodology for evaluating language tasks, reducing the gap between benchmarks and the real world. We also introduced a new challenge for the community, T uring A dvice , with an accompanying dataset and dynamic leaderboard. might have on society. In this paper, we presented a sketch of NLP models helping people who need advice on sensitive topics, which could be a measurable goal for the field. At the same time, we do not claim that our approach is a panacea. There are almost certainly better non-technical solutions to ensure mentorship and legal advice for all (Green, 2019). Moreover, there are significant dual-use risks with models that understand language (Hovy and Spruit, 2016; Green and Viljoen, 2020). Our evaluation measures some risks of generative models  such as the tendency to generate toxic language  but more work in this area is needed. Thanks to the Reddit users who participate in its advice subreddits  from asking for help, to writing (and voting on) helpful advice. Thanks to the Mechanical Turk workers who performed the annotation for our experiments. Thanks also to the three anonymous reviewers, along with Katharina Reinecke, Oren Etzioni, Hannah Rashkin, Maarten Sap, Maxwell Forbes, Jesse Thoma-son, Daniel Khashabi, Gabriel Ilharco, Swabha Swayamdipta, and Yonatan Bisk, for feedback. This research was supported in part by NSF (IIS-1524371, IIS-1714566), DARPA under the CwC program through the ARO (W911NF-15-1-0543), DARPA under the MCS program through NIWC Pacific (N66001-19-2-4031), and the NSF-GRFP No. DGE-1256082.
objective abstain method abstain abstain abstain abstain abstain result abstain abstain method abstain abstain abstain abstain abstain abstain method abstain method objective abstain objective objective abstain objective abstain method objective abstain abstain method abstain abstain abstain abstain result method result abstain method objective objective objective objective abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain method method abstain abstain abstain method abstain abstain method method method abstain method method abstain abstain method method abstain abstain abstain abstain abstain abstain abstain method method abstain method method abstain abstain result abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain result result abstain abstain abstain abstain method abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain abstain abstain abstain abstain abstain abstain other abstain abstain abstain method abstain abstain objective objective abstain objective abstain abstain abstain method other other other other other