Do NLP Beyond English
&gt;&gt; Recent models have repeatedly matched human-level performance on
increasingly difficult benchmarks—that is, in English using labelled datasets
with thousands and unlabelled data with millions of examples. In the process,
as a community we have overfit to the characteristics and conditions of
English-language data. In particular, by focusing on high-resource languages,
we have prioritised methods that work well only when large amounts of labelled
and unlabelled data are available.<p>This is an interesting observation. Training on large datasets tends to be
framed as a strength, e.g. we have recently seen articles praising OpenAI&#x27;s
GPT-3 for being, well, big. The truth is that model size (and the assorted
dataset size) is a bug, not a feature. It is the result of a dearth of
research on approaches with low sample complexity. Or in other words, it&#x27;s the
result of consistently picking the low-hanging fruit and calling sour grapes
on any domain for which there isn&#x27;t sufficiently large data (&quot;who cares about
Swahilli?&quot;).<p>&gt;&gt; In contrast, most current methods break down when applied to the
data-scarce conditions that are common for most of the world&#x27;s languages. Even
recent advances in pre-training language models that dramatically reduce the
sample complexity for downstream tasks (Peters et al., 2018; Howard and Ruder,
2018; Devlin et al., 2019; Clark et al., 2020) require massive amounts of
clean, unlabelled data, which is not available for most of the world&#x27;s
languages (Artetxe et al., 2020). Doing well with few data is thus an ideal
setting to test the limitations of current models—and evaluation on
low-resource languages constitutes arguably its most impactful real-world
application.<p>And this is interesting to read in the context of the recent manic hype about
GPT-3, itself described in a paper titled &quot;Language models are few-shot
learners&quot;, hilariously attempting to sweep under the carpet the amount of data
and compute required to get to the point where a language model can do &quot;few&quot;
shot learning.<p>In general, the excitement about the successes that have come from training
large, deep neural nework models with very large datasets has only served to
avoid posing the obvious question about such data-hungry approaches: what do
we do when there isn&#x27;t a lot of data? Human language with all its minutely
fragmented diversity, turns out to be just that kind of domain.