Answers.Body: <p>Actually, Google created a bigger model than GPT-3 and models in the GPT-3.5 series, and consequently ChatGPT too (because ChatGPT is based on a GPT-3.5 model) - <a href="https://arxiv.org/abs/2101.03961" rel="nofollow noreferrer">Switch-C</a> has trillions of parameters, one order of magnitude bigger than the GPT models that I know of, and it was developed before ChatGPT was announced. I don't know how many parameters ChatGPT has exactly, but it shouldn't have more than several billions of parameters.</p>
  <p>So, what makes reproducing a model like ChatGPT difficult for companies like Google? Definitely, not the lack of computational resources or money, but <strong>the lack of transparency</strong>. My impression is that Google also tends to be open-source, as opposed to OpenAI, which wants to make money of everything.</p>
  <p>Moreover, I'd like to note that the GPT models have received a lot of hype, but there are other pre-trained models (e.g. Lambda or Switch-C), <em>for example</em>, developed by Google, that maybe should also have our attention. Google simply doesn't need to generate all this hype to get the money, as they still get most of their revenue from ads (the last time I checked)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T09:25:37.593Z
Answers.Text: I think you need to back up that middle paragraph. AFAIK, OpenAI publish papers on their models, and explain how ChatGPT was trained here - https://openai.com/blog/chatgpt/ (also see links to their publications on Arxiv). Google's LLMs are not open source. So I am not seeing a difference in transparency between the two companies regarding these kind of products, unless you can provide more information
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T11:11:51.443Z
Answers.Text: @NeilSlater I'm saying that my impression is that Google tends to be open-source, while OpenAI doesn't. My impression could not be aligned with the reality. I'm not saying that Google is transparent about everything or more transparent than OpenAI about anything. I'm saying that what makes reproducing a model like ChatGPT is the lack of transparency. I'm also not saying that there isn't info about ChatGPT. I'm saying there isn't enough info about everything in order to reproduce it exactly.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T11:14:27.667Z
Answers.Text: @NeilSlater OpenAI also has open-source code, but making money of a GPT3 model for every char that you pass to the model or the model produces... while you use Google Translate and Google Search engine for free for years... To me, that's a clear sign of the mentality and political views/approaches of a company. I understand that OpenAI doesn't have the luxury of ads that Google has, but still I don't agree with their financial/political policy
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T11:22:28.730Z
Answers.Text: @NeilSlater Also note, the original question was "Can (i.e.) Google easily train a ChatGPT like model?", so clearly the OP was interested in knowing about if a specific company like Google can do it, not if every company can do it, which is what the other answer addresses. But then the question was changed and it's fine.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T14:43:56.807Z
Answers.Text: From your comment, then I think the second paragraph is muddled. It reads - to me - as a comparison on transparency and business plan between OpenAI and Google due to mentioning lack of transparency as a cause then a comparison on code release practices and revenue sources in the next sentence. I also see no evidence of a "lack of transparency" preventing Google doing anything in this space - they have LaMDA for example.
Answers.UserDisplayName: Neil Slater
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T15:36:11.117Z
Answers.Text: The question was about "reproducing" not doing something similar. So, the main reason why one _cannot_ reproduce is usually the lack of transparency and details. Definitely, not the compute cost, as the other answer suggests, for Google. Whether Google wants to reproduce the details or not, it's another story. They could in theory do it, given enough details of how the model was built, because they have the money.
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T15:46:26.363Z
Answers.Text: Anyone that tried to reproduce a paper, like me, knows that the lack of details/transparency is usually the main reason not to be able to reproduce something. Some authors even try to specify all details, but often, for different reasons, some are still missing
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T20:24:17.520Z
Answers.Text: @nbro [Why do so many publishing venues limit the length of paper submissions?](https://academia.stackexchange.com/q/57071/452) doesn't help reproducibility.
Answers.UserDisplayName: Franck Dernoncourt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-22T22:22:32.007Z
Answers.Text: @FranckDernoncourt Yes, that's one reason. But I think there could always be a longer version that circulates around, unless the journal or publisher prohibits it. Some already do that. But if we look at the InstructGPT or GPT-3 papers, they are already very long...
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T13:13:12.613Z
Answers.Text: There is something I'm not sure of and Icant find an answer even in OpenAI's blog. Is GPT-3.5 (series) a complete separate model thas has been trained from scratch or is it a fine tuned version of GPT-3 for code-completion and conversations ?
Answers.UserDisplayName: iMad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T13:39:13.860Z
Answers.Text: @iMad The only reliable info that I found about this topic was here: https://platform.openai.com/docs/model-index-for-researchers. Maybe it's not accurate to say that OpenAI claims that _all_ models in this series are more capable. I should rewrite that part of my answer
Answers.UserDisplayName: nbro
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T16:24:22.907Z
Answers.Text: Actually I wasn't referring to your answer specifically, it's just that I can't find any reliable information about how GPT-3.5 has been trained, its differences with GPT-3 etc.
  In OpenAI's docs, it's mentioned that InstructGPT is somehow a fine-tuned version of GPT-3, and that ChatGPT was trained in the same way as InstructGPT, but no explicit claim about the relationship between the GPT-3 base model and the GPT-3.5 series.
Answers.UserDisplayName: iMad
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-01T10:25:02.777Z
Answers.LastActivityDate: 2023-04-18T21:52:15.503Z
Answers.LastEditDate: 2023-04-18T21:52:15.503Z
Answers.UserDisplayName: nbro
Answers.Body: <p>Challenges to reproduce ChatGPT:</p>
  <ul>
  <li>Compute cost</li>
  <li>Collect training data</li>
  <li>Find the proper choice for network architecture + RL (OpenAI hasn't published all the details)</li>
  </ul>
  <hr />
  <p>Two interesting papers on training cost vs. LLM quality:</p>
  <ul>
  <li><a href="https://arxiv.org/pdf/2210.15424.pdf" rel="nofollow noreferrer">What Language Model to Train if You Have One Million GPU Hours</a>?</li>
  <li><a href="https://arxiv.org/pdf/2203.15556.pdf" rel="nofollow noreferrer">Training Compute-Optimal Large Language Models</a></li>
  </ul>
  <p>For some tasks, &quot;smaller LLMs&quot; can perform well e.g. see <em>Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto. <a href="https://arxiv.org/pdf/2301.13848.pdf" rel="nofollow noreferrer">Benchmarking Large Language Models for News Summarization</a>. arXiv:2301.13848.</em>:</p>
  <blockquote>
  <p>We find instruction tuning, and not model size, is the key to the
  LLM’s zero-shot summarization capability</p>
  </blockquote>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T23:52:22.420Z
Answers.Text: Indeed, compute cost is not trivial! Some estimates place the cost of training the model once at several million dollars, let alone the personnel costs, costs for data-gathering, and cost of any kind of hyper-parameter tuning or architecture search.
Answers.UserDisplayName: Sycorax
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-29T09:07:35.717Z
Answers.Text: Also, the human cost of having someone manually label the data.
Answers.UserDisplayName: Anshuman Kumar
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-29T09:15:17.203Z
Answers.Text: @AnshumanKumar yes I was including it in collect training data. There are some open access data eg C4 + https://github.com/nomic-ai/gpt4all
Answers.UserDisplayName: Franck Dernoncourt
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-31T07:08:32.977Z
Answers.LastActivityDate: 2023-02-23T09:16:23.443Z
Answers.LastEditDate: 2023-02-23T09:16:23.443Z
Answers.UserDisplayName: Franck Dernoncourt
Body: <p>Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it be possible for a company like Meta or Google to have a model equal to ChatGPT/GPT3.5 in the next month or so? Why or why not?</p>
  <p>I understand that a big language model is expensive to train, so I'm expecting only large companies to be able to train such models to a sufficient extent.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T09:28:34.503Z
Comments.Text: Can you provide more details about "oh its challenging Google" and about the articles and videos that claim that? It's challenging but most likely not because they don't have the computation power, but for other reasons, which are probably not true.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T10:13:05.220Z
Comments.Text: Im not interested in whether or not the articles or videos are true but only whether it is difficult for others to just replicate their work and what the reasons are for that. I removed the whole 'videos and articles'.
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T10:47:20.307Z
Comments.Text: Ok. I think now the question in the body is more objective. I would also change the title then. Another thing is: if you focus on the ChatGPT, then people could simple answer by saying that it's difficult because there's still no research paper available. Maybe you're interested just in ChatGPT or maybe in GPT3. I don't know. Make it clear if you're just interested in Google or other companies and even "normal people"
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T10:55:35.490Z
Comments.Text: Generalized it further ;) Thanks for the input
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T12:46:02.403Z
Comments.Text: Are you aware that Google made Lamda? https://blog.google/technology/ai/lamda/
Comments.UserDisplayName: Dr. Snoopy
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-25T14:04:38.223Z
Comments.Text: Another thing to note is that the transformer model was developed by Google Research.
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-31T21:57:42.093Z
Comments.Text: @nbro and attention in RNNs was invented by academic researches and positional-encodings in RNNs by FB (AFAIK). Giving all the credit to one team is ridiculous.
Comments.UserDisplayName: Mariah
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-01T08:35:55.340Z
Comments.Text: @Mariah You didn't understand the point of that comment. The point of that comment was: if Google researchers came up with the transformer, then they are familiar with the transformer and its potential. GPTs are nothing special. They just received more hype than other pre-trained models so that OpenAI gets more money. Google doesn't have to generate this hype to get the money, but they also developed other pre-trained models.
Comments.UserDisplayName: nbro
Tags: <training><language-model><large-language-models>
Title: What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?
UserDisplayName: Robin van Hoorn
ThreadText: # What makes reproducing a model like GPT3/GPT3.5/ChatGPT difficult?
  
  (Asked by: username_0 on 2023-01-25)
  
  Is it difficult for other companies to train a model similar to ChatGPT, and what makes it difficult? What is challenging about reproducing the results obtained by OpenAI with ChatGPT/GPT3.5? Would it be possible for a company like Meta or Google to have a model equal to ChatGPT/GPT3.5 in the next month or so? Why or why not?
  
  I understand that a big language model is expensive to train, so I'm expecting only large companies to be able to train such models to a sufficient extent.
  
  **Tags:** training, language-model, large-language-models
  
  > **Comments:**
  > **username_1**: Can you provide more details about "oh its challenging Google" and about the articles and videos that claim that? It's challenging but most likely not because they don't have the computation power, but for other reasons, which are probably not true.  
  > **username_0**: Im not interested in whether or not the articles or videos are true but only whether it is difficult for others to just replicate their work and what the reasons are for that. I removed the whole 'videos and articles'.  
  > **username_1**: Ok. I think now the question in the body is more objective. I would also change the title then. Another thing is: if you focus on the ChatGPT, then people could simple answer by saying that it's difficult because there's still no research paper available. Maybe you're interested just in ChatGPT or maybe in GPT3. I don't know. Make it clear if you're just interested in Google or other companies and even "normal people"  
  > **username_0**: Generalized it further ;) Thanks for the input  
  > **username_2**: Are you aware that Google made Lamda? https://blog.google/technology/ai/lamda/  
  > **username_1**: Another thing to note is that the transformer model was developed by Google Research.  
  > **username_3**: @username_1 and attention in RNNs was invented by academic researches and positional-encodings in RNNs by FB (AFAIK). Giving all the credit to one team is ridiculous.  
  > **username_1**: @username_3 You didn't understand the point of that comment. The point of that comment was: if Google researchers came up with the transformer, then they are familiar with the transformer and its potential. GPTs are nothing special. They just received more hype than other pre-trained models so that OpenAI gets more money. Google doesn't have to generate this hype to get the money, but they also developed other pre-trained models.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_5 on 2023-01-31. Score: 8 | Accepted answer
  
  Challenges to reproduce ChatGPT:
  
  * Compute cost
  * Collect training data
  * Find the proper choice for network architecture + RL (OpenAI hasn't published all the details)
  
  ---
  
  Two interesting papers on training cost vs. LLM quality:
  
  * [What Language Model to Train if You Have One Million GPU Hours](https://arxiv.org/pdf/2210.15424.pdf)?
  * [Training Compute-Optimal Large Language Models](https://arxiv.org/pdf/2203.15556.pdf)
  
  For some tasks, "smaller LLMs" can perform well e.g. see *Tianyi Zhang, Faisal Ladhak, Esin Durmus, Percy Liang, Kathleen McKeown, Tatsunori B. Hashimoto. [Benchmarking Large Language Models for News Summarization](https://arxiv.org/pdf/2301.13848.pdf). arXiv:2301.13848.*:
  
  > We find instruction tuning, and not model size, is the key to the
  > LLM’s zero-shot summarization capability
  
  > **Comments:**
  > **username_7**: Indeed, compute cost is not trivial! Some estimates place the cost of training the model once at several million dollars, let alone the personnel costs, costs for data-gathering, and cost of any kind of hyper-parameter tuning or architecture search.  
  > **username_8**: Also, the human cost of having someone manually label the data.  
  > **username_5**: @username_8 yes I was including it in collect training data. There are some open access data eg C4 + https://github.com/nomic-ai/gpt4all  
  
  ---
  
  ### Answer by username_1 on 2023-02-01. Score: 2
  
  Actually, Google created a bigger model than GPT-3 and models in the GPT-3.5 series, and consequently ChatGPT too (because ChatGPT is based on a GPT-3.5 model) - [Switch-C](https://arxiv.org/abs/2101.03961) has trillions of parameters, one order of magnitude bigger than the GPT models that I know of, and it was developed before ChatGPT was announced. I don't know how many parameters ChatGPT has exactly, but it shouldn't have more than several billions of parameters.
  
  So, what makes reproducing a model like ChatGPT difficult for companies like Google? Definitely, not the lack of computational resources or money, but **the lack of transparency**. My impression is that Google also tends to be open-source, as opposed to OpenAI, which wants to make money of everything.
  
  Moreover, I'd like to note that the GPT models have received a lot of hype, but there are other pre-trained models (e.g. Lambda or Switch-C), *for example*, developed by Google, that maybe should also have our attention. Google simply doesn't need to generate all this hype to get the money, as they still get most of their revenue from ads (the last time I checked)
  
  > **Comments:**
  > **username_6**: There is something I'm not sure of and Icant find an answer even in OpenAI's blog. Is GPT-3.5 (series) a complete separate model thas has been trained from scratch or is it a fine tuned version of GPT-3 for code-completion and conversations ?  
  > **username_1**: @username_6 The only reliable info that I found about this topic was here: https://platform.openai.com/docs/model-index-for-researchers. Maybe it's not accurate to say that OpenAI claims that _all_ models in this series are more capable. I should rewrite that part of my answer  
  > **username_6**: Actually I wasn't referring to your answer specifically, it's just that I can't find any reliable information about how GPT-3.5 has been trained, its differences with GPT-3 etc. In OpenAI's docs, it's mentioned that InstructGPT is somehow a fine-tuned version of GPT-3, and that ChatGPT was trained in the same way as InstructGPT, but no explicit claim about the relationship between the GPT-3 base model and the GPT-3.5 series.  
  > **username_4**: I think you need to back up that middle paragraph. AFAIK, OpenAI publish papers on their models, and explain how ChatGPT was trained here - https://openai.com/blog/chatgpt/ (also see links to their publications on Arxiv). Google's LLMs are not open source. So I am not seeing a difference in transparency between the two companies regarding these kind of products, unless you can provide more information  
  > **username_1**: @username_4 I'm saying that my impression is that Google tends to be open-source, while OpenAI doesn't. My impression could not be aligned with the reality. I'm not saying that Google is transparent about everything or more transparent than OpenAI about anything. I'm saying that what makes reproducing a model like ChatGPT is the lack of transparency. I'm also not saying that there isn't info about ChatGPT. I'm saying there isn't enough info about everything in order to reproduce it exactly.  
  > **username_1**: @username_4 OpenAI also has open-source code, but making money of a GPT3 model for every char that you pass to the model or the model produces... while you use Google Translate and Google Search engine for free for years... To me, that's a clear sign of the mentality and political views/approaches of a company. I understand that OpenAI doesn't have the luxury of ads that Google has, but still I don't agree with their financial/political policy  
  > **username_1**: @username_4 Also note, the original question was "Can (i.e.) Google easily train a ChatGPT like model?", so clearly the OP was interested in knowing about if a specific company like Google can do it, not if every company can do it, which is what the other answer addresses. But then the question was changed and it's fine.  
  > **username_4**: From your comment, then I think the second paragraph is muddled. It reads - to me - as a comparison on transparency and business plan between OpenAI and Google due to mentioning lack of transparency as a cause then a comparison on code release practices and revenue sources in the next sentence. I also see no evidence of a "lack of transparency" preventing Google doing anything in this space - they have LaMDA for example.  
  > **username_1**: The question was about "reproducing" not doing something similar. So, the main reason why one _cannot_ reproduce is usually the lack of transparency and details. Definitely, not the compute cost, as the other answer suggests, for Google. Whether Google wants to reproduce the details or not, it's another story. They could in theory do it, given enough details of how the model was built, because they have the money.  
  > **username_1**: Anyone that tried to reproduce a paper, like me, knows that the lack of details/transparency is usually the main reason not to be able to reproduce something. Some authors even try to specify all details, but often, for different reasons, some are still missing  
  > **username_5**: @username_1 [Why do so many publishing venues limit the length of paper submissions?](https://academia.stackexchange.com/q/57071/452) doesn't help reproducibility.  
  > **username_1**: @username_5 Yes, that's one reason. But I think there could always be a longer version that circulates around, unless the journal or publisher prohibits it. Some already do that. But if we look at the InstructGPT or GPT-3 papers, they are already very long...  
  
  ---
  
  