Answers.Body: <p>Two years after the original question was made I feel it is time to provide an updated answer:</p>
  <blockquote>
  <p>How can it understand our world using algorithms and then recreate
  human-like content?</p>
  </blockquote>
  <p>GPT-3, GPT-4 (transformers, Chat-GPTs and the likes) do not understand our world. They do provide answers based on any info available on the &quot;training datasets&quot; and on any additional &quot;training dataset&quot; used to &quot;fine-tune&quot; the model. Those answers recreate human-like content even if the models/algorithms do not understand nor have a representation of the world.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-04-27T09:04:26.453Z
Answers.LastActivityDate: 2023-04-27T09:43:45.607Z
Answers.LastEditDate: 2023-04-27T09:43:45.607Z
Answers.UserDisplayName: Raul Alvarez
Answers.Body: <p>GPT-3 (and the likes) don't really have any understanding of the <a href="https://en.wikipedia.org/wiki/Lexical_semantics" rel="nofollow noreferrer">semantics</a> nor <a href="https://en.wikipedia.org/wiki/Pragmatics" rel="nofollow noreferrer">pragmatics</a> involved in the language. However, they are good at constructing text content similar to the contents created by a person (when the texts and the concepts are not too &quot;complicated&quot;).</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T19:13:44.207Z
Answers.Text: But how do the different layers and math formulas work to create this type of intelligence?
Answers.UserDisplayName: DragonflyRobotics
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T20:18:53.200Z
Answers.Text: There are many questions regarding this issue on this  platform, though I wouldn't call it "intelligence". Attention is all you need....
Answers.UserDisplayName: Raul Alvarez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T21:10:29.217Z
Answers.Text: So then what is "Attention". I simply want to understand the working principle of GPT-3
Answers.UserDisplayName: DragonflyRobotics
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-10T16:31:42.490Z
Answers.Text: @DragonflyRobotics We don't have a great idea, we know the general principle of neural networks is to make something that can in principle do the computation, and then spend hundreds of thousands of dollars (in the worst case) renting GPUs to tweak it quadrillions of times until it performs well.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-08-10T16:32:18.943Z
Answers.Text: @DragonflyRobotics The attention unit is, AFAIK, the major contribution of the transformers paper. If you want to know what attention is then I suggest you focus your question around that.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2021-05-07T16:04:00.643Z
Answers.LastActivityDate: 2021-05-08T01:08:32.393Z
Answers.LastEditDate: 2021-05-08T01:08:32.393Z
Answers.UserDisplayName: Raul Alvarez
Body: <p>I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly factual essays. How can it understand our world using algorithms and then recreate human-like content? How can it learn to understand a description and program in HTML?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2021-05-07T16:00:17.707Z
Comments.Text: It can't "understand" anything. It just matches patterns to something it encountered in the training data.
Comments.UserDisplayName: Oliver Mason
Tags: <neural-networks><transformer><human-like><gpt-3>
Title: How do transformers understand data and answer custom questions?
UserDisplayName: DragonflyRobotics
ThreadText: # How do transformers understand data and answer custom questions?
  
  (Asked by: username_0 on 2021-05-06)
  
  I recently heard of GPT-3 and I don't understand how the attention models and transformers encoders and decoders work. I heard that GPT-3 can make a website from a description and write perfectly factual essays. How can it understand our world using algorithms and then recreate human-like content? How can it learn to understand a description and program in HTML?
  
  **Tags:** neural-networks, transformer, human-like, gpt-3
  > **Comments:**
  > **username_1**: It can't "understand" anything. It just matches patterns to something it encountered in the training data.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-04-27. Score: 1 | Accepted answer
  
  Two years after the original question was made I feel it is time to provide an updated answer:
  
  > How can it understand our world using algorithms and then recreate
  > human-like content?
  
  GPT-3, GPT-4 (transformers, Chat-GPTs and the likes) do not understand our world. They do provide answers based on any info available on the "training datasets" and on any additional "training dataset" used to "fine-tune" the model. Those answers recreate human-like content even if the models/algorithms do not understand nor have a representation of the world.
  
  
  ---
  
  ### Answer by username_2 on 2021-05-07. Score: 1
  
  GPT-3 (and the likes) don't really have any understanding of the [semantics](https://en.wikipedia.org/wiki/Lexical_semantics) nor [pragmatics](https://en.wikipedia.org/wiki/Pragmatics) involved in the language. However, they are good at constructing text content similar to the contents created by a person (when the texts and the concepts are not too "complicated").
  
  > **Comments:**
  > **username_0**: But how do the different layers and math formulas work to create this type of intelligence?  
  > **username_2**: There are many questions regarding this issue on this  platform, though I wouldn't call it "intelligence". Attention is all you need....  
  > **username_0**: So then what is "Attention". I simply want to understand the working principle of GPT-3  
  > **username_3**: @DragonflyRobotics We don't have a great idea, we know the general principle of neural networks is to make something that can in principle do the computation, and then spend hundreds of thousands of dollars (in the worst case) renting GPUs to tweak it quadrillions of times until it performs well.  
  > **username_3**: @DragonflyRobotics The attention unit is, AFAIK, the major contribution of the transformers paper. If you want to know what attention is then I suggest you focus your question around that.  
  
  ---
  
  