Announcing GPT-NeoX-20B
So excited for this release.  In the wake of AI Dungeon's downfall, having GPT-Neo to fallback on has been a saving grace.  While the 6B model is nowhere near as good as the original AI Dungeon, which used OpenAI's 175B model, it was at least serviceable unlike the "gentled" AI Dungeon.  And you could run it locally or through Colab, which was really cool.  I ended up using it through NovelAI, since they've spent a lot of time fine-tuning the model and adding a plethora of features that end up improving the overall output.  (NovelAI's interface is like AI Dungeon on steroids!)  But there is a vibrant community of Colab notebooks and other tools for DIYers surrounding the GPT-Neo model.<p>That said, besides being overall "dumber" than 175B GPT-3, the 6B model was missing a critical feature: prompting.  175B GPT-3 could be "prompted" to write things.  For example, you could give it "Write a story about cyberpunk gnomes:" and it would go on to do just that, all on its own.  GPT-Neo didn't really have that capability in my experience.  The only way to get it to reliably write such a story is to begin writing it yourself, at which point GPT-Neo could help to continue the story.<p>So I'm excited to see not just how much "smarter" Eleuther's new 20B model is, but also if it has attained that coveted prompting ability.  Given the non-linear relationship between parameters and loss, my hopes are high.<p>P.S. NovelAI recently added the Fairseq 13B model to their repertoire.  I haven't had a chance to try it personally, but I've seen positive things about it.  My bet is on GPT-NeoX-20B being better still.