GPT Neo: open-source GPT model, with pretrained 1.3B & 2.7B weight models
Please test their models before you take it at face value.<p>Eleuther has a history of claiming to replicate projects when they haven't. For example, they shipped a DALL-E repo a few days after OpenAI announced it (<a href="https://twitter.com/theshawwn/status/1348017515897659392" rel="nofollow">https://twitter.com/theshawwn/status/1348017515897659392</a>) which was broken, and they've walked back their GPT-3 replication claims to replicating 1.5B due to the fact that their architecture doesn't scale.<p>As far as I can tell, they're generating a large amount of hype with grandiose claims that they can't deliver on.<p>All I care about is whether you like their models and actually use them in practice. If you do, please let me know and I'll pipe down. But so far, I haven't heard of anyone who uses anything they've produced, and that worries me. Has anyone?<p>One specific claim they made: <a href="https://twitter.com/BlancheMinerva/status/1347276975547809809" rel="nofollow">https://twitter.com/BlancheMinerva/status/134727697554780980...</a><p>"DALL-E is quite straight forward and already coded. We just need data to train it."<p>No, DALL-E is neither straightforward nor was it successfully coded, especially back on January 7th.<p>Anyway, carry on. I really don't like speaking badly of AI projects, and I hope that they succeed. The model release today is a good step forward, assuming it works. But it might be better to have the expectation of "the models don't work" until proven otherwise.<p>I'd also like to point out that there are some capable people doing work at Eleuther. Sid in particular is one of the best TPU hackers in the scene. I just wish they would scale down their claims, release more models, and not claim that they've done X until actually doing X. For example, the readme says they have "the ability to scale up to full GPT3 sizes (and possibly more!), using the mesh-tensorflow library," which they don't.