title: Discourse-Based Evaluation of Language Understanding
abstract: New models for natural language understanding have made unusual progress recently, leading to claims of universal text representations. However, current benchmarks are predominantly targeting semantic phenomena;  we make the case that discourse and pragmatics need to take center stage in the evaluation of natural language understanding.
  We introduce DiscEval, a new benchmark for the evaluation of natural language understanding, that unites 11 discourse-focused evaluation datasets. 
  DiscEval can be used as supplementary training data in a multi-task learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets.
  Using our evaluation suite, we show that natural language inference, a widely used pretraining task, does not result in genuinely universal representations, which opens a new challenge for multi-task learning.
authors: Damien Sileo Tim Van-De-Cruys Camille Pradel Philippe Muller
hIndex: Philippe Muller 25