GPT-2: 6-Month Follow-Up
<rant>
Are there any real use case for GPT-2? Does it solve any problem?
I've read almost all state of the art leaderboards of all Nlp tasks of paperswithcode.com and truth is except text generation, openAI has not one state of the art, they are not even visible in leaderboards. 
OpenAI is maybe the AI research center with the biggest funding and comparatively to other well known (Microsoft, Facebook, Google or even zalando..) they are the ones with the least results.<p>From my observations most SOTAs come from chineses researchers by far, followed by deepmind.<p>BTW isn't that a sad truth that not even one of all major AI actors has a draft of an AGI architecture, something comparable to CYC or opencog.
<a href="https://wiki.opencog.org/w/CogPrime_Overview" rel="nofollow">https://wiki.opencog.org/w/CogPrime_Overview</a><p>Two other observations I would like to share:
Many important NLP tasks have almost nobody publicly working on them it seems, on paperswithcode.com or NLP-progress (from github) some tasks have only one or two papers...
And many others have not evolved since 2016. 
Most of the time it seems trivial to beat the old state of the art, just use BERT or XLnet on a task where nobody applied it before and hop, free state of the art for you! Yet researchers don't seems to chase those low hanging, high returns fruits.
Also researchers seems to work a lot in isolation, many new generic improvements like new optimizers (RAdam for example) and new activation functions (Swish) allow to beat most of older state of the art on almost all task just by using them. 
Yet researchers will take years before using them because of an absurd inertia.
Also unlike an open source program, BERT and XLnet have very low response and activity on github despite major open issues... 
</rant>