Answers.Body: <blockquote>
  <p>Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?</p>
  </blockquote>
  <p>No, assuming it's regular, consumer hardware. However, some open-source alternatives  aim to be runnable on consumer hardware, e.g. <a href="https://github.com/LAION-AI/Open-Assistant" rel="nofollow noreferrer">https://github.com/LAION-AI/Open-Assistant</a> (not released yet):</p>
  <blockquote>
  <p>We want to do this in a way that is open and accessible, which means we must not only build a great assistant, but also make it small and efficient enough to run on consumer hardware.</p>
  </blockquote>
  <p>There exist  however  smaller language models, e.g. see this <a href="https://arxiv.org/pdf/2410.20011" rel="nofollow noreferrer">survey of small language models</a>.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-17T03:15:14.537Z
Answers.Text: the moment this becomes feasible, every thing changes again, for good and for bad.
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-11T03:09:12.397Z
Answers.LastActivityDate: 2024-10-29T05:33:37.787Z
Answers.LastEditDate: 2024-10-29T05:33:37.787Z
Answers.UserDisplayName: Franck Dernoncourt
Answers.Body: <p>Here, I assume the ChatGPT architecture is similar to the published GPT-3 model [1].</p>
  <p>The <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)" rel="nofollow noreferrer">Transformer</a> architecture used for GPT3 is much more memory-bound than compute-bound at a high parameter count and a low context size. A large batch size (3.2M across ~few thousands GPUs for GPT-3) is used during training, but relatively low computational power is needed at the inference time for non-batched applications such as real-time chat. OpenAI is probably batching multiple users' requests to improve the throughput.</p>
  <p>GPT-3 has 175B parameters and would require 326GiB memory to store the weight in float16. <em>Quantization</em> methods are proposed to reduce the memory required to store model weights, where the model weights are stored in lower precision. In the GPTQ paper [2], the authors have shown that large GPT models perform almost as good as the original model with 4-bit per parameter, reducing the 175B model to less than 90GiB. 3-bit quantization is also considered, reducing the size to less than 70GiB.
  Note that additional memory required for internal states is small compared to the weights, especially if a <a href="https://github.com/facebookresearch/xformers" rel="nofollow noreferrer">memory-efficient attention</a> is used.</p>
  <p>Recently, such quantization methods have gained attention due to the release of the pre-trained <a href="https://github.com/facebookresearch/llama/" rel="nofollow noreferrer">LLaMA</a> models.
  Some people are running 65B LLaMA model even CPU-only [3] [4] with a reasonable throughput.</p>
  <p>Therefore, for the question of whether it would be possible to run a 175B model in a home computer, I would say it is currently stretch but certainly possible if somewhat slow responses are acceptable.</p>
  <ul>
  <li>[1]: Brown, Tom, et al. &quot;Language models are few-shot learners.&quot; Advances in neural information processing systems 33 (2020): 1877-1901. <a href="https://arxiv.org/abs/2005.14165" rel="nofollow noreferrer">https://arxiv.org/abs/2005.14165</a></li>
  <li>[2]: Frantar, Elias, et al. &quot;GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.&quot; (2022). <a href="https://arxiv.org/abs/2210.17323" rel="nofollow noreferrer">https://arxiv.org/abs/2210.17323</a></li>
  <li>[3]: <a href="https://twitter.com/thesudoer_/status/1634883960525422597" rel="nofollow noreferrer">Satyajit Ghana on Twitter</a></li>
  <li>[4]: <a href="https://twitter.com/tg3517/status/1635095157480759297" rel="nofollow noreferrer">Taiga Takano on Twitter</a></li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-13T09:38:14.137Z
Answers.LastActivityDate: 2023-03-15T10:26:12.727Z
Answers.LastEditDate: 2023-03-15T10:26:12.727Z
Answers.UserDisplayName: pcpthm
Body: <p>Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?</p>
  <h2>Assumptions</h2>
  <ol>
  <li>I assume getting output based on some input is, at least, hundreds of times faster than training such a model.</li>
  <li>I assume no additional output parsing/input limitations are used. In particular I can imagine all the boiler plate to keep the ChatGPT model(s) acting politically correct etc. may be a significant overhead. This is to be ignored for this question.</li>
  </ol>
  <h2>Data</h2>
  <p>So far I've found the ChatGPT 3.5 model to have <a href="https://lifearchitect.ai/chatgpt/" rel="noreferrer">175 billion parameters</a>:</p>
  <p><a href="https://i.sstatic.net/DeIZD.png" rel="noreferrer"><img src="https://i.sstatic.net/DeIZD.png" alt="enter image description here" /></a></p>
  <p>Though I do not yet know how large that is in <code>Mb</code> nor do I have an idea on how long generating an output would typically take.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-06T14:06:13.653Z
Comments.Text: 175 billion parameters means at least 175 billion numbers, means it's probably in the range of 175GB (if all numbers are 1 byte - unlikely) - 1.4 TB (if all numbers are 64bit = 8byte, more likely).
Comments.UserDisplayName: kutschkem
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-09T02:53:41.343Z
Comments.Text: ... and apart from storage/RAM, also need the computation power to do the math and return an answer within reasonable time. So yes, it is feasible, if you have a few hundred thousand bucks to spend I guess.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-09T02:56:14.690Z
Comments.Text: Ah, don't forget the cooling and electricity bill.
Comments.UserDisplayName: lpounng
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-22T03:26:17.640Z
Comments.Text: @kutschkem Thdd we numbers are half-precision, so 175B parameters is 350 GB.
Comments.UserDisplayName: Charles
Tags: <open-ai><chatgpt><open-source>
Title: Would self-hosting ChatGPT be feasible, w.r.t. computation costs?
UserDisplayName: a.t.
ThreadText: # Would self-hosting ChatGPT be feasible, w.r.t. computation costs?
  
  (Asked by: username_0 on 2023-02-04)
  
  Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?
  
  ## Assumptions
  
  1. I assume getting output based on some input is, at least, hundreds of times faster than training such a model.
  2. I assume no additional output parsing/input limitations are used. In particular I can imagine all the boiler plate to keep the ChatGPT model(s) acting politically correct etc. may be a significant overhead. This is to be ignored for this question.
  
  ## Data
  
  So far I've found the ChatGPT 3.5 model to have [175 billion parameters](https://lifearchitect.ai/chatgpt/):
  
  [![enter image description here](https://i.sstatic.net/DeIZD.png)](https://i.sstatic.net/DeIZD.png)
  
  Though I do not yet know how large that is in `Mb` nor do I have an idea on how long generating an output would typically take.
  
  **Tags:** open-ai, chatgpt, open-source
  
  > **Comments:**
  > **username_1**: 175 billion parameters means at least 175 billion numbers, means it's probably in the range of 175GB (if all numbers are 1 byte - unlikely) - 1.4 TB (if all numbers are 64bit = 8byte, more likely).  
  > **username_2**: ... and apart from storage/RAM, also need the computation power to do the math and return an answer within reasonable time. So yes, it is feasible, if you have a few hundred thousand bucks to spend I guess.  
  > **username_2**: Ah, don't forget the cooling and electricity bill.  
  > **username_3**: @username_1 Thdd we numbers are half-precision, so 175B parameters is 350 GB.  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_6 on 2023-03-13. Score: 6 | Accepted answer
  
  Here, I assume the ChatGPT architecture is similar to the published GPT-3 model [1].
  
  The [Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) architecture used for GPT3 is much more memory-bound than compute-bound at a high parameter count and a low context size. A large batch size (3.2M across ~few thousands GPUs for GPT-3) is used during training, but relatively low computational power is needed at the inference time for non-batched applications such as real-time chat. OpenAI is probably batching multiple users' requests to improve the throughput.
  
  GPT-3 has 175B parameters and would require 326GiB memory to store the weight in float16. *Quantization* methods are proposed to reduce the memory required to store model weights, where the model weights are stored in lower precision. In the GPTQ paper [2], the authors have shown that large GPT models perform almost as good as the original model with 4-bit per parameter, reducing the 175B model to less than 90GiB. 3-bit quantization is also considered, reducing the size to less than 70GiB.
  Note that additional memory required for internal states is small compared to the weights, especially if a [memory-efficient attention](https://github.com/facebookresearch/xformers) is used.
  
  Recently, such quantization methods have gained attention due to the release of the pre-trained [LLaMA](https://github.com/facebookresearch/llama/) models.
  Some people are running 65B LLaMA model even CPU-only [3] [4] with a reasonable throughput.
  
  Therefore, for the question of whether it would be possible to run a 175B model in a home computer, I would say it is currently stretch but certainly possible if somewhat slow responses are acceptable.
  
  * [1]: Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901. <https://arxiv.org/abs/2005.14165>
  * [2]: Frantar, Elias, et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." (2022). <https://arxiv.org/abs/2210.17323>
  * [3]: [Satyajit Ghana on Twitter](https://twitter.com/thesudoer_/status/1634883960525422597)
  * [4]: [Taiga Takano on Twitter](https://twitter.com/tg3517/status/1635095157480759297)
  
  
  ---
  
  ### Answer by username_4 on 2023-02-11. Score: 4
  
  > Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?
  
  No, assuming it's regular, consumer hardware. However, some open-source alternatives aim to be runnable on consumer hardware, e.g. <https://github.com/LAION-AI/Open-Assistant> (not released yet):
  
  > We want to do this in a way that is open and accessible, which means we must not only build a great assistant, but also make it small and efficient enough to run on consumer hardware.
  
  There exist however smaller language models, e.g. see this [survey of small language models](https://arxiv.org/pdf/2410.20011).
  
  > **Comments:**
  > **username_5**: the moment this becomes feasible, every thing changes again, for good and for bad.  
  
  ---
  
  