GPT-Neo – Building a GPT-3-sized model, open source and free
In my experience, the output from GPT-3, DALL-E, et al is similar to what you get from googling the prompt and stitching together snippets from the top results. These transformers are trained on "what was visible to google", which provides the limitation on their utility.<p>I think of the value proposition of GPT-X as "what would you do with a team of hundreds of people who can solve arbitrary problems only by googling them?". And honestly, not a lot of productive applications come to mind.<p>This is basically the description of a modern content farm. You give N people a topic, ie "dating advice", and they'll use google to put together different ideas, sentences and paragraphs to produce dozens of articles per day. You could also write very basic code with this, similar to googling a code snippet and combining the results from the first several stackoverflow pages that come up (which, incidentally, is how I program now). After a few more versions, you could probably use GPT to produce fiction that matches the quality of the average self published ebook. And DALL-E can come up with novel images in the same way that a graphic designer can visually merge the google image results for a given query.<p>One limitation of this theoretical "team of automated googlers" is that what they search for content is cached on the date of the last GPT model update. Right now the big news story is the Jan 6th, 20201 insurrection at the US Capitol. GPT-3 can produce infinite bad articles about politics, but won't be able to say anything about current events in real time.<p>I generally think that GPT-3 is awesome, and it's a damn shame that "Open"AI couldn't find a way to actually be open. At this point, it seems like a very interesting technology that is still in desperate need of a Killer App