7/11

21/08/2023, 16:10

OpenAI's GPT-3 Language Model: A Technical Overview

If GPT-2 was "too dangerous to release," and GPT-3 almost passed the Turing test for
writing short articles. What can a trillion parameter model do? For years the research
community has been searching for chatbots that "just works," could GPT-3 be the
breakthrough? Is it really possible to have a massive pre-trained model, so any downstream
tasks become a matter of providing a few examples or descriptions in the prompt? At a
broader scale, can this "data compilation + reprogram" paradigm ultimately lead us to AGI?
AI safety needs to go a long way to prevent techniques like these from being misused, but it
seems the day of having truly intelligent conversations with robots is just at the horizon.

Footnotes
1. We use Lambda GPU instance pricing of $1.50 / hour for a Tesla V100 (8x Tesla V100s
are $12.00/hour). 355Y × 365D/Y × 24H/D × $1.5/H = $4.6M
OpenAI's GPT-3 Language Model_ A Technical Overview