OpenAI is giving Microsoft exclusive access to its GPT-3 language model
Possibly a hot take:<p>I would say that OpenAI <i>is</i> still “open”, in about the same sense that Nginx or Redis is “open”. They seem to be pursuing an open-<i>core</i> model, where the architecture and skeleton of the project is open, but the extra stuff that lots of corporate money was dedicated to building on top is closed.<p>Remember, GPT-3 is just GPT-2 with (a lot) more training data—training data that cost a lot of money to acquire, and therefore training data that (unlike GPT-2’s training set) has corporate interests invested in its creation. GPT-2 is still the “core” of GPT-3’s architecture; GPT-3 can be seen as a commercial “extension” on top of the GPT-2 core.<p>If you want to understand “how GPT-3 works”, you can just study GPT-2. Nothing is different architecturally between them.<p>It <i>is</i> a shame that GPT-3 isn’t open for study, because it has emergent capabilities (meta-learning) that GPT-2 didn’t express, and which are worthy of study. But they couldn’t have known it would do that when they set off to create GPT-3; and they <i>did</i> know they needed a bunch of money for compute + training-data scraping, so they likely made agreements (handshake or contractual) to license this beefed-up “extension” to GPT-2 <i>in advance of</i> knowing what it could do; and in fact <i>as a prerequisite to</i> getting that funding to build it.<p>I would suspect that OpenAI’s goal with GPT-3 was mostly to <i>observe for themselves</i> what a much-better-trained GPT-2 would be capable of; to study those observed capabilities (like meta-learning), and to try to figure out if they can replicate them using only tweaks to GPT-2’s architecture, <i>without</i> needing the big load of commercially-funded training data. (Sort of like seeing what optimal solutions a brute-force algorithm can find, and then trying to come up with an efficient algorithm to emit those same solutions.)<p>If they <i>can</i> come up with something like that, then there’s likely a GPT-4 brewing right now, which <i>wouldn’t</i> need all that compute and training data to create; and therefore <i>wouldn’t</i> be beholden to corporate interests; and therefore <i>would</i> be open for study.