Google T5 scores 88.9 on SuperGLUE Benchmark, approaching human baseline
As someone working in the field, I congratulate the excellent accomplishment but agree with the authors that we shouldn&#x27;t get too excited yet (their quote below after the four reasons). Here are some reasons:<p>1) Most likely, the model is still susceptible to adversarial triggers as demonstrated on other systems here: <a href="http:&#x2F;&#x2F;www.ericswallace.com&#x2F;triggers" rel="nofollow">http:&#x2F;&#x2F;www.ericswallace.com&#x2F;triggers</a><p>2) T5 was trained with ~750GB of texts or ~150 billion words, which is &gt; 100 times the number of words native English speakers acquire by the age of 20.<p>3) Most or all of the tests are multiple-choice. Learning complex correlations from sufficient data should help solve most of them. This is useful but human-level understanding is more than correlations.<p>4) The performance on datasets that require commonsense knowledge, COPA and WSC, are the weakest relative to humans (who score 100.0 on both).<p>Exploring the Limits of Transfer Learning with a
Unified Text-to-Text Transformer, p.32
<a href="https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;1910.10683.pdf" rel="nofollow">https:&#x2F;&#x2F;arxiv.org&#x2F;pdf&#x2F;1910.10683.pdf</a><p>&quot;Interestingly, on the reading comprehension tasks (MultiRC and ReCoRD) we exceed human performance by a large margin, suggesting the evaluation metrics used for these tasks may be biased towards machine-made predictions. On the other hand, humans achieve 100% accuracy on both COPA and WSC, which is significantly better than our model’s performance. This suggests that there remain linguistic tasks that are hard for our model to perfect, particularly in the low-resource setting.&quot;<p>I’d like to emphasize that the work and the paper are excellent. Still, we are quite far from human-level language understanding.<p>---<p>We may need more advanced tests to probe the actual language <i>understanding</i> ability of AI systems. Here are some ideas:<p>* Test for conceptual understanding in a non-multiple-choice format. Example: Write a summary for a New Yorker article, rather than standard news pieces (which tend to follow repeated patterns).<p>* Commonsense test with longer chains of inference than those needed for solving Winograd Schema and set in non-standard situations (e.g. fantasy world). This should greatly reduce the chance that an approach can simply detect correlations from huge datasets.<p>* Understanding novel, creative metaphors like those used in some essays by professional writers or some of the Economist&#x27;s title articles.