Body: <p>As shown in the figure:</p>
  <p><a href="https://i.sstatic.net/P5U4w.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/P5U4w.jpg" alt="BERT for-all generalization" /></a></p>
  <p>Why does token prediction work when &quot;Socrates&quot; is replaced with &quot;Plato&quot;?</p>
  <p>From the point of view of symbolic logic, the above example effectively performs the logic rule:</p>
  <pre><code>∀x. human(x) ⇒ mortal(x)
  </code></pre>
  <p>How might we explain this ability?  Moreover, how is this learned in just a few shots of examples?</p>
  <p>I think this question is key to understanding the Transformer's logical reasoning ability.</p>
  <p>Below are excerpts from 2 papers:</p>
  <p><a href="https://i.sstatic.net/ifuUG.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/ifuUG.png" alt="excerpt 1" /></a></p>
  <p><a href="https://i.sstatic.net/AodI6.jpg" rel="nofollow noreferrer"><img src="https://i.sstatic.net/AodI6.jpg" alt="excerpt 2" /></a></p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-07T13:37:26.307Z
Comments.Text: I feel that I must point out that "logical reasoning ability" isn't a good way to view any of these types of networks.  Transformers are interesting in how they provide attention through a sequence, but it is still just a series of mathematics transformations, not logical reasoning.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-08T02:01:41.937Z
Comments.Text: @DavidHoelzer Whether or not this is logical reasoning is a matter of philosophy, especially on a scale this large. After all, the human brain is ultimately emergent, as is the behaviour of such models.
Comments.UserDisplayName: Recessive
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-08T12:01:50.077Z
Comments.Text: @DavidHoelzer There is ample evidence that BERT/GPT is capable of logic reasoning, to a certain degree.
Comments.UserDisplayName: Yan King Yin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-09T08:45:24.203Z
Comments.Text: Can you please provide the link to the code that allows us to reproduce what you're claiming? Also, where did you take this image from? You also say "There is ample evidence that BERT/GPT is capable of logic reasoning, to a certain degree." , can you provide more examples (with **references** to the implementation or paper) of this? One such example does not convince anyone that it's capable of logical reasoning.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-10T09:14:29.427Z
Comments.Text: Having said that, in the example that you give, it's very likely that these models just replaced Socrates with Plato in both cases because, statistically, that seems to be a very common thing to do (from the training data). We cannot say it's logical reasoning, without more info about your specific example. If these models were just trained with data, then this is probably just a byproduct result of the statistics in the training data.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-24T12:27:17.290Z
Comments.Text: @nbro:  Yes, I can add some excerpts from papers to support that.
Comments.UserDisplayName: Yan King Yin
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-24T12:55:04.907Z
Comments.Text: @YanKingYin Thanks. Can you also provide the links to or names of the papers?
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-12-28T18:14:57.473Z
Comments.Text: @nbro: OK, 1 sec...
Comments.UserDisplayName: Yan King Yin
Tags: <transformer><bert><logic><gpt><reasoning>
Title: Why is BERT/GPT capable of "for-all" generalization?
UserDisplayName: Yan King Yin
ThreadText: # Why is BERT/GPT capable of "for-all" generalization?
  
  (Asked by: username_0 on 2021-12-07)
  
  As shown in the figure:
  
  Why does token prediction work when "Socrates" is replaced with "Plato"?
  From the point of view of symbolic logic, the above example effectively performs the logic rule:
  ∀x. human(x) ⇒ mortal(x)
  
  How might we explain this ability?  Moreover, how is this learned in just a few shots of examples?
  I think this question is key to understanding the Transformer's logical reasoning ability.
  Below are excerpts from 2 papers:
  
  **Tags:** transformer, bert, logic, gpt, reasoning
  > **Comments:**
  > **username_1**: I feel that I must point out that "logical reasoning ability" isn't a good way to view any of these types of networks.  Transformers are interesting in how they provide attention through a sequence, but it is still just a series of mathematics transformations, not logical reasoning.  
  > **username_2**: @DavidHoelzer Whether or not this is logical reasoning is a matter of philosophy, especially on a scale this large. After all, the human brain is ultimately emergent, as is the behaviour of such models.  
  > **username_0**: @DavidHoelzer There is ample evidence that BERT/GPT is capable of logic reasoning, to a certain degree.  
  > **username_3**: Can you please provide the link to the code that allows us to reproduce what you're claiming? Also, where did you take this image from? You also say "There is ample evidence that BERT/GPT is capable of logic reasoning, to a certain degree." , can you provide more examples (with **references** to the implementation or paper) of this? One such example does not convince anyone that it's capable of logical reasoning.  
  > **username_3**: Having said that, in the example that you give, it's very likely that these models just replaced Socrates with Plato in both cases because, statistically, that seems to be a very common thing to do (from the training data). We cannot say it's logical reasoning, without more info about your specific example. If these models were just trained with data, then this is probably just a byproduct result of the statistics in the training data.  
  > **username_0**: @nbro:  Yes, I can add some excerpts from papers to support that.  
  > **username_3**: @YanKingYin Thanks. Can you also provide the links to or names of the papers?  
  > **username_0**: @nbro: OK, 1 sec...  
  
  ---
  
  ## 0 answers
  
  