New models for natural language understanding have made unusual progress recently, leading to claims of universal text representations. However, current benchmarks are predominantly targeting semantic phenomena; we make the case that discourse and pragmatics need to take center stage in the evaluation of natural language understanding. We introduce DiscEval, a new benchmark for the evaluation of natural language understanding, that unites 11 discourse-focused evaluation datasets. DiscEval can be used as supplementary training data in a multitask learning setup, and is publicly available, alongside the code for gathering and preprocessing the datasets. Using our evaluation suite, we show that natural language inference, a widely used pretraining task, does not result in genuinely universal representations, which opens a new challenge for multi-task learning.
- Overview: This work presents a benchmark for language understanding centered on understanding pragmatics and discourse, in contrast to existing NLP benchmarks which focus on mostly semantic understanding. The stated contributions are
    - The DiscEval benchmark, with some estimates of human performance
    - Baseline performance with state of the art models
    - Comparisons between commonly used supervised training objectives (NLI) and discourse-oriented objectives
- Review: This work offers a complementary evaluation benchmark for NLU systems to what currently exists in the field, and also offers compelling evidence that the current methods used do not provide good signal for learning the types of phenomena assessed in this benchmark. I recommend (weak) accept.
    - The paper would benefit a lot from having more in-depth explanation of the tasks, what the classes are, and what the classes mean. It's not always clear from the examples or the labels in Table 1 what exactly is being tested.
        - Additionally, it'd be nice to get the source of the data for each task.
    - The rough grouping of tasks in nice.
    - I understand that estimates of human performance, especially for tasks with a high number of classes, can be tricky to obtain. I do feel that they are especially important to have for this dataset. As you said, some of these tasks rely on context that isn't explicitly provided in the utterance, so I wonder if for some of the tasks, there may be insufficient context in just the provided utterances for humans to perform well.
        - Additionally, some tasks seem to have quite subjective label definitions. For example, "eloquence" and "specificity" in Persuasion; "formality" and "informativeness" in Squinky; etc. Validating that humans are consistent on these tasks seems crucial.
        - For the estimates you do have, are these expert or lay annotators? For some of these annotations, it seems like you'd need trained annotators.
    - Have you given any thought to exploitable data artifacts that may occur in these datasets that might be driving the fairly high performance on some of these tasks (e.g. Verifiability)?
    - I think it's quite interesting that fine-tuning on MNLI doesn't lead to good performance on DiscEval, as MNLI, as the authors point out, is commonly taken to be a useful pretraining task. This discrepancy gives practical weight to the authors' claim that discourse and pragmatic phenomena are not being sufficiently studied or evaluated for in current NLP research, despite the fact that these handling these phenomena will be crucial for NLP systems.
- Things to Improve + Questions + Comments
    - I would hope that most people in the NLP community would not say that language understanding is a solved problem, but I agree that putting "universal" in model names and paper titles is a reach-y thing to do.
    - Tasks
        - What is the original citation for STAC?
    - S4.3, Table 2: 
        - Any reason not to try combining all four pretraining/intermediate training tasks in a single model, or at least more combinations of DiscEval with other things?
        - Could you comment on the standard deviation of the scores (per task) given that you're averaging 6 runs?
    - Table 3
        - "The best overall unsupervised result is achieved with Discovery STILT": what does unsupervised mean here? It also doesn't look like BERT+Discovery is the best in any column here.
    - Tables 4 and 5 feel like a bit of a dump of numbers. It'd be more useful to the readers to extract trends (probably using the groupings and theoretical frameworks introduced earlier). The noting of BERT+MNLI being good at predicting absence of relations is nice.
    - Typos: There are a noticeable number of typos. Here are some I noted:
        - The formatting of the task descriptions in Section 3 is a bit inconsistent and awkward.
        - Table 1: "exemple"
        - P4: "We use a custom train/dev validation split"
        - P5: "that help *realize* speech acts' intentions."
        - P6: "Prediction of discourse markers based *off* the context clauses..."