Answers.Body: <p>GPT-4 is a transformer like GPT-3 and any other GPT. The training is certainly new, because it has a different size, you just can not transfer GPT-3 weights into an GPT-4 to continue training.</p>
  <p>The details of the implementation are currently not known, the published report about it is not a scientific paper about how GPT-4 works, it basically states &quot;we do not say how it works&quot;, see below.</p>
  <p>The size of various components certainly differ,
  at least the number of weights in one part, but it could also be larger everywhere. The prompt length (context size) and the maximal output size increased a lot, from 4000 tokens to 8000 or even 32000 tokens.
  I would expect there are some other minor differences.</p>
  <p>Other major differences can be the amount of training data,
  and the compute used for training.
  My personal speculation is that GPT-4 used a lot more compute for training, with possibly a similar amount of training data as GPT-3.</p>
  <hr />
  <p>From the &quot;GPT-4 Technical Report&quot;, section 2:</p>
  <blockquote>
  <p>Given both the competitive landscape and the safety implications of
  large-scale models like GPT-4, this report contains no further details
  about the architecture (including model size), hardware, training
  compute, dataset construction, training method, or similar.</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T08:38:44.027Z
Answers.Text: Saying that GPT-3 is just a bigger GPT and GPT-2 is correct, but saying GPT-4 *is* the same as GPT-3, conceptually, I believe is incorrect. You cannot feed an image like a text without processing steps, such as like ViT's way. That makes GPT-4 differ from GPT-3, albeit small.
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T14:07:24.780Z
Answers.Text: Yes, I added discussion of the differences.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T20:34:22.143Z
Answers.Text: "you just can not transfer GPT-3 weights into an GPT-4 to continue training" - actually, you could, as long as what you transfer is compatible in mathematical terms (i.e. you can still multiply the matrices/tensors). In fact, a form of transfer learning is based on the idea of freezing some weights then connecting a new compatible set of weights. It's very well possible that they are doing something like that or some form of model compression, as training from scratch may take a long time.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-04T23:14:55.780Z
Answers.Text: @nbro If GPT-4 would be constructed to be kind of an extension of GPT-3, that would be possible, and very interesting. But that would require very different CNNs, I think you can not just hammer a small transformer into a larger one. We do not know what they actually did, but trying to build GPT-4 based of a learned model of a smaller CNN sounds like big scientific progress we would know of. Hmm... But then, what would happen when you just take the weights you have from an old smaller model, and use the as initialization as the beginning, together with 20 times the amount of noise or zeros?
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T08:23:51.937Z
Answers.LastActivityDate: 2023-03-22T14:06:30.553Z
Answers.LastEditDate: 2023-03-22T14:06:30.553Z
Answers.UserDisplayName: Volker Siegel
Answers.Body: <p>GPT-4 is largely based on GPT-3. From the <a href="https://arxiv.org/pdf/2303.08774.pdf" rel="noreferrer">GPT-4 Technical Report</a>:</p>
  <blockquote>
  <p>GPT-4 is a Transformer-style model [39]</p>
  </blockquote>
  <p>The transformer-style model originates from the paper <a href="https://arxiv.org/abs/1706.03762" rel="noreferrer">Attention Is All You Need</a>, which lays the foundation of GPT, GPT-2, and GPT-3.</p>
  <p><em>However</em>, there is one significant change: GPT-4 accepts images as inputs. This has been observed in the paper <a href="https://arxiv.org/pdf/2103.00020.pdf" rel="noreferrer">Learning Transferable Visual Models From Natural Language Supervision</a> by OpenAI. We can safely predict that GPT-4 is based on GPT-3 + CLIP.</p>
  <p>I say <em>predict</em> because the authors decided not to publish the architecture or anything:</p>
  <blockquote>
  <p>Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.</p>
  </blockquote>
  <p>However, we have no details regarding whether it is trained from scratch, or use any technique.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T11:14:23.037Z
Answers.Text: But did they re-train from scratch it or not?
Answers.UserDisplayName: Anixx
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T11:35:59.087Z
Answers.Text: `this report contains no further details about [...] training method`
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T12:42:14.187Z
Answers.Text: @Anixx
  As I already said in the removed answer, GPT-4 is just a retrained GPT-3 with more fine tuning. It's a matter of what has changed in the training data set. For example, it seem that the previously mentioned *`glitch tokes`* have now been removed, which is likely an effect of removing dumb scraping data, such as from the *counting* sub-reddit, or from scraping random debug log files, etc. In other words, the training is done from scratch, but it's likeley to be done on mostly the same data, adjusted to remove problem points, as discovered by humans from previous version.
Answers.UserDisplayName: not2qubit
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-21T22:00:49.093Z
Answers.Text: @not2qubit *glitch tokens* are an effect of having tokens in the token set which were not, or rarely, in the training data, as they result from spam data which was removed before training, but after selecting the tokens. They can be removed from the token splitter which converts text into tokens without needing to re-train.
Answers.UserDisplayName: Criticize SE actions means ban
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T04:13:07.410Z
Answers.Text: @not2qubit `GPT-4 is just a retrained GPT-3 with more fine tuning` is this what you *think* or it is something the authors say? Why does removing the glitch token infers GPT-4 is just a fine-tune GPT-3?
Answers.UserDisplayName: Minh-Long Luu
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T14:14:26.490Z
Answers.Text: @not2qubit The glitch tokens are not actually a problem in the transformer, it is a problem with building the set of tokens that can be used for learning input. I think it is enough to literally remove the glitch tokens from the set of valid tokens. I expect the set of tokens may be the same, apart from that. GPT-3 uses the same tokens GPT-2 uses.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-22T14:17:16.843Z
Answers.Text: @not2qubit it is quite possible that the training data did not change at all. The glitch tokens can be fixed without cleaning the data and retraining.
Answers.UserDisplayName: Volker Siegel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-19T01:36:38.023Z
Answers.LastActivityDate: 2023-03-22T08:35:59.510Z
Answers.LastEditDate: 2023-03-22T08:35:59.510Z
Answers.UserDisplayName: Minh-Long Luu
Body: <p>To me it looks like GPT-4 is based on GPT-3.</p>
  <p>On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-16T22:39:23.283Z
Comments.Text: are you thinking of the "glitch tokens"? Those can be fixed without re-training.
Comments.UserDisplayName: Criticize SE actions means ban
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-16T23:43:58.120Z
Comments.Text: @user253751 what is this?
Comments.UserDisplayName: Anixx
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-17T00:14:16.603Z
Comments.Text: [Glitch tokens](https://www.youtube.com/watch?v=WO2X3oZEJOA) - interesting stuff! If this is the bug you were talking about, it doesn't require re-training.
Comments.UserDisplayName: Criticize SE actions means ban
Tags: <open-ai><gpt><gpt-3><gpt-4>
Title: Is GPT-4 based on GPT-3 or was it trained from the scratch?
UserDisplayName: Anixx
ThreadText: # Is GPT-4 based on GPT-3 or was it trained from the scratch?
  
  (Asked by: username_0 on 2023-03-16)
  
  To me it looks like GPT-4 is based on GPT-3.
  
  On the other hand, there were rumors that training of GPT-3 was done with errors, but re-train was impossible due to the costs.
  
  **Tags:** open-ai, gpt, gpt-3, gpt-4
  > **Comments:**
  > **username_1**: are you thinking of the "glitch tokens"? Those can be fixed without re-training.  
  > **username_0**: @user253751 what is this?  
  > **username_1**: [Glitch tokens](https://www.youtube.com/watch?v=WO2X3oZEJOA) - interesting stuff! If this is the bug you were talking about, it doesn't require re-training.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_2 on 2023-03-22. Score: 4 | Accepted answer
  
  GPT-4 is a transformer like GPT-3 and any other GPT. The training is certainly new, because it has a different size, you just can not transfer GPT-3 weights into an GPT-4 to continue training.
  
  The details of the implementation are currently not known, the published report about it is not a scientific paper about how GPT-4 works, it basically states "we do not say how it works", see below.
  
  The size of various components certainly differ,
  at least the number of weights in one part, but it could also be larger everywhere. The prompt length (context size) and the maximal output size increased a lot, from 4000 tokens to 8000 or even 32000 tokens.
  I would expect there are some other minor differences.
  
  Other major differences can be the amount of training data,
  and the compute used for training.
  My personal speculation is that GPT-4 used a lot more compute for training, with possibly a similar amount of training data as GPT-3.
  
  ---
  
  From the "GPT-4 Technical Report", section 2:
  
  > Given both the competitive landscape and the safety implications of
  > large-scale models like GPT-4, this report contains no further details
  > about the architecture (including model size), hardware, training
  > compute, dataset construction, training method, or similar.
  
  > **Comments:**
  > **username_3**: Saying that GPT-3 is just a bigger GPT and GPT-2 is correct, but saying GPT-4 *is* the same as GPT-3, conceptually, I believe is incorrect. You cannot feed an image like a text without processing steps, such as like ViT's way. That makes GPT-4 differ from GPT-3, albeit small.  
  > **username_2**: Yes, I added discussion of the differences.  
  > **username_4**: "you just can not transfer GPT-3 weights into an GPT-4 to continue training" - actually, you could, as long as what you transfer is compatible in mathematical terms (i.e. you can still multiply the matrices/tensors). In fact, a form of transfer learning is based on the idea of freezing some weights then connecting a new compatible set of weights. It's very well possible that they are doing something like that or some form of model compression, as training from scratch may take a long time.  
  > **username_2**: @nbro If GPT-4 would be constructed to be kind of an extension of GPT-3, that would be possible, and very interesting. But that would require very different CNNs, I think you can not just hammer a small transformer into a larger one. We do not know what they actually did, but trying to build GPT-4 based of a learned model of a smaller CNN sounds like big scientific progress we would know of. Hmm... But then, what would happen when you just take the weights you have from an old smaller model, and use the as initialization as the beginning, together with 20 times the amount of noise or zeros?  
  
  ---
  
  ### Answer by username_3 on 2023-03-19. Score: 6
  
  GPT-4 is largely based on GPT-3. From the [GPT-4 Technical Report](https://arxiv.org/pdf/2303.08774.pdf):
  
  > GPT-4 is a Transformer-style model [39]
  
  The transformer-style model originates from the paper [Attention Is All You Need](https://arxiv.org/abs/1706.03762), which lays the foundation of GPT, GPT-2, and GPT-3.
  
  *However*, there is one significant change: GPT-4 accepts images as inputs. This has been observed in the paper [Learning Transferable Visual Models From Natural Language Supervision](https://arxiv.org/pdf/2103.00020.pdf) by OpenAI. We can safely predict that GPT-4 is based on GPT-3 + CLIP.
  
  I say *predict* because the authors decided not to publish the architecture or anything:
  
  > Given both the competitive landscape and the safety implications of large-scale models like GPT-4, this report contains no further details about the architecture (including model size), hardware, training compute, dataset construction, training method, or similar.
  
  However, we have no details regarding whether it is trained from scratch, or use any technique.
  
  > **Comments:**
  > **username_0**: But did they re-train from scratch it or not?  
  > **username_3**: `this report contains no further details about [...] training method`  
  > **username_5**: @Anixx As I already said in the removed answer, GPT-4 is just a retrained GPT-3 with more fine tuning. It's a matter of what has changed in the training data set. For example, it seem that the previously mentioned *`glitch tokes`* have now been removed, which is likely an effect of removing dumb scraping data, such as from the *counting* sub-reddit, or from scraping random debug log files, etc. In other words, the training is done from scratch, but it's likeley to be done on mostly the same data, adjusted to remove problem points, as discovered by humans from previous version.  
  > **username_1**: @not2qubit *glitch tokens* are an effect of having tokens in the token set which were not, or rarely, in the training data, as they result from spam data which was removed before training, but after selecting the tokens. They can be removed from the token splitter which converts text into tokens without needing to re-train.  
  > **username_3**: @not2qubit `GPT-4 is just a retrained GPT-3 with more fine tuning` is this what you *think* or it is something the authors say? Why does removing the glitch token infers GPT-4 is just a fine-tune GPT-3?  
  > **username_2**: @not2qubit The glitch tokens are not actually a problem in the transformer, it is a problem with building the set of tokens that can be used for learning input. I think it is enough to literally remove the glitch tokens from the set of valid tokens. I expect the set of tokens may be the same, apart from that. GPT-3 uses the same tokens GPT-2 uses.  
  > **username_2**: @not2qubit it is quite possible that the training data did not change at all. The glitch tokens can be fixed without cleaning the data and retraining.  
  
  ---
  
  