Answers.Body: <p>The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while all progress made in NLP was definitely useful, it also is important to just scale up the model size.</p>
  <p>This may not seem like surprising point, but it actually kind of is. Normally, performance would saturate or at least the gain would slope off, but this is not the case! So the main innovation may not be that big and is kind off brute-force but the point still stands: Bigger models are better.</p>
  <p>Another point to mention is the way they did the training. Such a large model needs some tricks to be actually trained (and fast at that). You also want to make use of multiple GPUs for parallel training. This means they also had to develop new structures for training.</p>
  <p>Why exactly it is predicted as a huge innovation may only be contained to some twitter demonstration, there are no real sources on this as far as I know. Especially because the model is not openly available.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-14T12:23:26.180Z
Answers.LastActivityDate: 2020-09-14T12:23:26.180Z
Answers.UserDisplayName: N. Kiefer
Body: <p>I've been hearing a lot about <a href="https://arxiv.org/abs/2005.14165" rel="nofollow noreferrer">GPT-3 by OpenAI</a>, and that it's a simple to use API with text in text out and has a big neural network off 175B parameters.</p>
  <p>But how did they achieve this huge number of parameters, and why is it being predicted as one of the greatest innovations?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2020-07-20T12:05:44.263Z
Comments.Text: You're asking "why is it being predicted as one of the greatest innovations?", but who is predicting it as a great innovation? Edit your post to include links to sources that show that people are predicting this. Any model that uses that big number of parameters doesn't seem like a big innovation to me but just brute-force.
Comments.UserDisplayName: nbro
Tags: <reinforcement-learning><open-ai><gpt>
Title: Why is GPT-3 such a game changer?
UserDisplayName: jash
ThreadText: # Why is GPT-3 such a game changer?
  
  (Asked by: username_0 on 2020-07-19)
  
  I've been hearing a lot about [GPT-3 by OpenAI](https://arxiv.org/abs/2005.14165), and that it's a simple to use API with text in text out and has a big neural network off 175B parameters.
  
  But how did they achieve this huge number of parameters, and why is it being predicted as one of the greatest innovations?
  
  **Tags:** reinforcement-learning, open-ai, gpt
  > **Comments:**
  > **username_1**: You're asking "why is it being predicted as one of the greatest innovations?", but who is predicting it as a great innovation? Edit your post to include links to sources that show that people are predicting this. Any model that uses that big number of parameters doesn't seem like a big innovation to me but just brute-force.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2020-09-14. Score: 3
  
  The main point in GPT-3 and already in 2 was the observation that performance was steadily increasing with increasing model size (As seen in Figure 1.2 in your linked paper). So it seems that while all progress made in NLP was definitely useful, it also is important to just scale up the model size.
  
  This may not seem like surprising point, but it actually kind of is. Normally, performance would saturate or at least the gain would slope off, but this is not the case! So the main innovation may not be that big and is kind off brute-force but the point still stands: Bigger models are better.
  
  Another point to mention is the way they did the training. Such a large model needs some tricks to be actually trained (and fast at that). You also want to make use of multiple GPUs for parallel training. This means they also had to develop new structures for training.
  
  Why exactly it is predicted as a huge innovation may only be contained to some twitter demonstration, there are no real sources on this as far as I know. Especially because the model is not openly available.
  
  
  ---
  
  