Answers.Body: <p>Yes, with caveats.</p>
  <p>Yes: If the data covers a niche and is very rare, you can indeed <a href="https://stats.stackexchange.com/questions/331369/what-is-meant-by-fine-tuning-of-neural-network">fine-tune</a> a large model to your needs.</p>
  <p>Caveats: Fine tuning a model still require significant compute. Moreover the largest models to start the fine-tuning with (e.g. GPT-3) are not open-source, even though this may change, so you would settle on a older open-source model. And without the expensive <a href="https://ai.stackexchange.com/questions/38521/how-was-chatgpt-trained/38561#38561">RLHF</a> process to train the agent (paying the human labellers), your agent will be very knowledgeable in your niche area, but overall poorly aligned and potentially toxic.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T19:06:28.237Z
Answers.Text: Thank you very much. I think it would be interesting to have total control over the transformer - oversight or deliberate selection of all data it "knows" - not pre-training on say, the web in general, and then fine-tuning with specialized knowledge. I guess I thought the toxicity of a model was (often) just a reflection of toxic material / debris that fell into its training material, trawled in from the internet at large. I still don't have a great intuition about how much data a transformer needs to perform well... they say quality over quantity. What if trained exclusively on like, OED?
Answers.UserDisplayName: Julius Hamilton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T19:07:57.850Z
Answers.Text: If a transformer knew nothing except the entirety of the Oxford English Dictionary, I would hope it must find at least some patterns, and it could be interesting to see at minimum what it is capable of, if it has any interesting use at all.
Answers.UserDisplayName: Julius Hamilton
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T20:25:55.753Z
Answers.Text: If you are proposing training from scratch, that would give you the highest quality, but it is going to be expensive (millions). Indeed the network must be large to be good (at least as of today architectures), which means a lot of parameters to train, therefore a lot of compute required.
Answers.UserDisplayName: Rexcirus
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-30T00:05:05.487Z
Answers.LastActivityDate: 2023-01-30T00:05:05.487Z
Answers.UserDisplayName: Rexcirus
Body: <p>Soft question here.</p>
  <p>I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million parameters and the training data could be from 1GB - 1TB, and that the training could take from about a day to a week. Also, there is an open source GPT <a href="https://github.com/karpathy/nanoGPT" rel="nofollow noreferrer">here</a>.</p>
  <p>My question is, if you consider that ChatGPT is trained on a very large and diverse amount of data, you may think a solo project could never compete with it. However, what if you chose a specialized set of training data that was smaller but a much richer, more reliable knowledge base, like only academic science textbooks, or only English literature, or only Python libraries documentation, and so on?</p>
  <p>Could it actually be much more useful because it's open-source, you have freedom of use (unlike ChatGPT's heavy behavioral conditioning from OpenAI), and you can choose what kind of knowledge the transformer contains? If the data is smaller but way, way higher quality, could you just make a library of niche GPTs for any topic you are studying?</p>
  
Tags: <transformer><gpt><language-model><fine-tuning><chatgpt>
Title: Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
UserDisplayName: Julius Hamilton
ThreadText: # Would a transformer trained on highly specific material be as usable as a commercial product like ChatGPT?
  
  (Asked by: username_0 on 2023-01-28)
  
  Soft question here.
  
  I was recently learning a bit about how it is feasible to train a transformer on a personal computer like an M1 Mac. I have been told that the model could have 1-3 million parameters and the training data could be from 1GB - 1TB, and that the training could take from about a day to a week. Also, there is an open source GPT [here](https://github.com/karpathy/nanoGPT).
  
  My question is, if you consider that ChatGPT is trained on a very large and diverse amount of data, you may think a solo project could never compete with it. However, what if you chose a specialized set of training data that was smaller but a much richer, more reliable knowledge base, like only academic science textbooks, or only English literature, or only Python libraries documentation, and so on?
  
  Could it actually be much more useful because it's open-source, you have freedom of use (unlike ChatGPT's heavy behavioral conditioning from OpenAI), and you can choose what kind of knowledge the transformer contains? If the data is smaller but way, way higher quality, could you just make a library of niche GPTs for any topic you are studying?
  
  **Tags:** transformer, gpt, language-model, fine-tuning, chatgpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-01-30. Score: 2 | Accepted answer
  
  Yes, with caveats.
  
  Yes: If the data covers a niche and is very rare, you can indeed [fine-tune](https://stats.stackexchange.com/questions/331369/what-is-meant-by-fine-tuning-of-neural-network) a large model to your needs.
  
  Caveats: Fine tuning a model still require significant compute. Moreover the largest models to start the fine-tuning with (e.g. GPT-3) are not open-source, even though this may change, so you would settle on a older open-source model. And without the expensive [RLHF](https://ai.stackexchange.com/questions/38521/how-was-chatgpt-trained/38561#38561) process to train the agent (paying the human labellers), your agent will be very knowledgeable in your niche area, but overall poorly aligned and potentially toxic.
  
  > **Comments:**
  > **username_0**: Thank you very much. I think it would be interesting to have total control over the transformer - oversight or deliberate selection of all data it "knows" - not pre-training on say, the web in general, and then fine-tuning with specialized knowledge. I guess I thought the toxicity of a model was (often) just a reflection of toxic material / debris that fell into its training material, trawled in from the internet at large. I still don't have a great intuition about how much data a transformer needs to perform well... they say quality over quantity. What if trained exclusively on like, OED?  
  > **username_0**: If a transformer knew nothing except the entirety of the Oxford English Dictionary, I would hope it must find at least some patterns, and it could be interesting to see at minimum what it is capable of, if it has any interesting use at all.  
  > **username_1**: If you are proposing training from scratch, that would give you the highest quality, but it is going to be expensive (millions). Indeed the network must be large to be good (at least as of today architectures), which means a lot of parameters to train, therefore a lot of compute required.  
  
  ---
  
  