Answers.Body: <p>GPT-3 is based on <strong>in-context learning</strong>. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or &quot;GPT-3&quot;.</p>
  <p>Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an autoregressive LM, instead of incorporating bidirectional information (similarly to Bert).</p>
  <p><em>While in-context learning is more straightforward with autoregressive LMs, bidirectional models are known to be better at downstream tasks after fine-tuning</em>.</p>
  <p>In the end, training a bidirectional model at the scale of GPT-3 or trying to make bidirectional models work with few-shot learning is a promising direction for future research.</p>
  <p>Check out <a href="https://www.kdnuggets.com/2020/06/gpt-3-deep-learning-nlp.html" rel="nofollow noreferrer">this</a>, <a href="https://towardsdatascience.com/gpt-3-primer-67bc2d821a00" rel="nofollow noreferrer">this</a> and <a href="https://arxiv.org/pdf/2001.08361.pdf" rel="nofollow noreferrer">the paper on Scaling Laws for Neural Language Models</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-21T22:05:22.277Z
Answers.Text: You should probably cite some research papers that support this claim: " It’s common wisdom one can hope that bigger models will yield better in-context capabilities". Moreover, I think you should explain more in detail what "in-context learning" really means.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2020-09-16T02:41:34.507Z
Answers.LastActivityDate: 2020-09-16T02:41:34.507Z
Answers.UserDisplayName: Arpit-Gole
Body: <p>In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be explicitly fine-tuned for the new task.</p>
  <p>In GPT-3 (From my understanding), this is no longer the case. It can perform a larger array of language tasks from translation, open domain conversation, summarization, etc., with only a few examples. No explicit fine-tuning is needed.</p>
  <p>The actual theory behind GPT-3 is <em>fairly</em> simple, which would not suggest any level of ability other than what would be found in common narrow intelligence systems.</p>
  <p>However, looking past the media hype and the news coverage, GPT-3 is not explicitly programmed to &quot;know&quot; how to do these wider arrays of tasks. In fact, with limited examples, it can perform many language tasks quite well and &quot;learn on the fly&quot; so to speak. To me, this does seem to align fairly well with what most people would consider strong AI, but in a narrow context, which is language tasks.</p>
  <p>Thoughts? Is GPT-3 an early example of strong AI but in a narrower context?</p>
  
Tags: <philosophy><agi><gpt>
Title: Is GPT-3 an early example of strong AI in a narrow setting?
UserDisplayName: Landon G
ThreadText: # Is GPT-3 an early example of strong AI in a narrow setting?
  
  (Asked by: username_0 on 2020-09-15)
  
  In GPT-2, the large achievement was being able to generate coherent text over a long-form while maintaining context. This was very impressive but for GPT-2 to do new language tasks, it had to be explicitly fine-tuned for the new task.
  
  In GPT-3 (From my understanding), this is no longer the case. It can perform a larger array of language tasks from translation, open domain conversation, summarization, etc., with only a few examples. No explicit fine-tuning is needed.
  
  The actual theory behind GPT-3 is *fairly* simple, which would not suggest any level of ability other than what would be found in common narrow intelligence systems.
  
  However, looking past the media hype and the news coverage, GPT-3 is not explicitly programmed to "know" how to do these wider arrays of tasks. In fact, with limited examples, it can perform many language tasks quite well and "learn on the fly" so to speak. To me, this does seem to align fairly well with what most people would consider strong AI, but in a narrow context, which is language tasks.
  
  Thoughts? Is GPT-3 an early example of strong AI but in a narrower context?
  
  **Tags:** philosophy, agi, gpt
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2020-09-16. Score: 4
  
  GPT-3 is based on **in-context learning**. It’s common wisdom one can hope that bigger models will yield better in-context capabilities. And indeed, this holds true, in the case of GPT-3 175B or "GPT-3".
  
  Neverthless GPT-3 is more powerful than it's predecessors. In some of the tasks, GPT-3 failed miserably. This might be due to the choice to use an autoregressive LM, instead of incorporating bidirectional information (similarly to Bert).
  
  *While in-context learning is more straightforward with autoregressive LMs, bidirectional models are known to be better at downstream tasks after fine-tuning*.
  
  In the end, training a bidirectional model at the scale of GPT-3 or trying to make bidirectional models work with few-shot learning is a promising direction for future research.
  
  Check out [this](https://www.kdnuggets.com/2020/06/gpt-3-deep-learning-nlp.html), [this](https://towardsdatascience.com/gpt-3-primer-67bc2d821a00) and [the paper on Scaling Laws for Neural Language Models](https://arxiv.org/pdf/2001.08361.pdf).
  
  > **Comments:**
  > **username_2**: You should probably cite some research papers that support this claim: " It’s common wisdom one can hope that bigger models will yield better in-context capabilities". Moreover, I think you should explain more in detail what "in-context learning" really means.  
  
  ---
  
  