GLUE: A MULTI-TASK BENCHMARK AND ANALYSIS PLATFORM FOR NATURAL LANGUAGE UNDERSTAND- ING
For natural language understanding (NLU) technology to be maximally useful, it must be able to process language in a way that is not exclusive to a single task, genre, or dataset. In pursuit of this objective, we introduce the General Language Understanding Evaluation (GLUE) benchmark, a collection of tools for evaluating the performance of models across a diverse set of existing NLU tasks. By including tasks with limited training data, GLUE is designed to favor and encourage models that share general linguistic knowledge across tasks. GLUE also includes a hand-crafted diagnostic test suite that enables detailed linguistic analysis of models. We evaluate baselines based on current methods for transfer and representation learning and find that multi-task training on all tasks performs better than training a separate model per task. However, the low absolute performance of our best model indicates the need for improved general NLU systems.
The paper proposes a new benchmark for natural language understanding: GLUE. Models will be evaluated based on a diverse set of existing language understanding tasks which encourages the models to learn shared knowledge across different tasks. The authors empirically show that models trained with multiple tasks in the dataset perform better than models that focused on one specific task. They also point out existing methods are not able achieve good performance in this dataset and request for more general natural language understanding system. The work also collects an expert evaluated diagnostic evaluation dataset for further examination for the models.
Quality: borderline, clarity:good, originality: borderline, significance: good,
Pros:
- The benchmark is set up in a online platform with leaderboard which can be easily accessible to people.
- The benchmark comes with a diagnostic evaluation dataset with coarse-grained and fine-grained categories that examine different aspect of language understanding abilities.
- Baseline results for major existing models are provided
Cons:
- The author should provide more detailed analysis and interpretable explanations for the results as opposed to simply stating that the overall performance is better.
For example, why attention hurts performance in single task training? Why multi-tasks training actually leads to worse performance on some of the dataset? Do these phenomenons still exist if you train on a different subset of the dataset?
What are the samples that the models failed to perform well? It would be nice to get some more insights and conclusions based on the results obtained from this benchmark to shed some lights on how to improve these models. The results section should be seriously revised.
- The diagnostic evaluation dataset seems to be a way to better understand the model, however, it is hard to see the scope of the data (are the samples under each categories balanced?). Besides, the examples in the dataset seems very confusing even for humans (Table 2).  The evaluation with NLP expert is also far from perfect. I wonder how accurate is this dataset annotated (or even the sentences make sense or not), and how suitable it is for evaluating model’s language understanding abilities. It would be nice if the authors can include some statistics about the dataset.
The paper proposes a useful benchmark that measures different aspects of language understanding abilities which would be helpful to the community. However, I feel the novelty or take away messages from the experiment section is limited.