<blockquote>
<p>Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?</p>
</blockquote>
<p>No, assuming it's regular, consumer hardware. However, some open-source alternatives  aim to be runnable on consumer hardware, e.g. <a href="https://github.com/LAION-AI/Open-Assistant" rel="nofollow noreferrer">https://github.com/LAION-AI/Open-Assistant</a> (not released yet):</p>
<blockquote>
<p>We want to do this in a way that is open and accessible, which means we must not only build a great assistant, but also make it small and efficient enough to run on consumer hardware.</p>
</blockquote>
<p>There exist  however  smaller language models, e.g. see this <a href="https://arxiv.org/pdf/2410.20011" rel="nofollow noreferrer">survey of small language models</a>.</p>

CC BY-SA 4.0
2023-03-17T03:15:14.537Z
the moment this becomes feasible, every thing changes again, for good and for bad.
wide_eyed_pupil
CC BY-SA 4.0
2023-02-11T03:09:12.397Z
2024-10-29T05:33:37.787Z
2024-10-29T05:33:37.787Z
Franck Dernoncourt
<p>Here, I assume the ChatGPT architecture is similar to the published GPT-3 model [1].</p>
<p>The <a href="https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)" rel="nofollow noreferrer">Transformer</a> architecture used for GPT3 is much more memory-bound than compute-bound at a high parameter count and a low context size. A large batch size (3.2M across ~few thousands GPUs for GPT-3) is used during training, but relatively low computational power is needed at the inference time for non-batched applications such as real-time chat. OpenAI is probably batching multiple users' requests to improve the throughput.</p>
<p>GPT-3 has 175B parameters and would require 326GiB memory to store the weight in float16. <em>Quantization</em> methods are proposed to reduce the memory required to store model weights, where the model weights are stored in lower precision. In the GPTQ paper [2], the authors have shown that large GPT models perform almost as good as the original model with 4-bit per parameter, reducing the 175B model to less than 90GiB. 3-bit quantization is also considered, reducing the size to less than 70GiB.
Note that additional memory required for internal states is small compared to the weights, especially if a <a href="https://github.com/facebookresearch/xformers" rel="nofollow noreferrer">memory-efficient attention</a> is used.</p>
<p>Recently, such quantization methods have gained attention due to the release of the pre-trained <a href="https://github.com/facebookresearch/llama/" rel="nofollow noreferrer">LLaMA</a> models.
Some people are running 65B LLaMA model even CPU-only [3] [4] with a reasonable throughput.</p>
<p>Therefore, for the question of whether it would be possible to run a 175B model in a home computer, I would say it is currently stretch but certainly possible if somewhat slow responses are acceptable.</p>
<ul>
<li>[1]: Brown, Tom, et al. &quot;Language models are few-shot learners.&quot; Advances in neural information processing systems 33 (2020): 1877-1901. <a href="https://arxiv.org/abs/2005.14165" rel="nofollow noreferrer">https://arxiv.org/abs/2005.14165</a></li>
<li>[2]: Frantar, Elias, et al. &quot;GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers.&quot; (2022). <a href="https://arxiv.org/abs/2210.17323" rel="nofollow noreferrer">https://arxiv.org/abs/2210.17323</a></li>
<li>[3]: <a href="https://twitter.com/thesudoer_/status/1634883960525422597" rel="nofollow noreferrer">Satyajit Ghana on Twitter</a></li>
<li>[4]: <a href="https://twitter.com/tg3517/status/1635095157480759297" rel="nofollow noreferrer">Taiga Takano on Twitter</a></li>
</ul>

CC BY-SA 4.0
2023-03-13T09:38:14.137Z
2023-03-15T10:26:12.727Z
2023-03-15T10:26:12.727Z
pcpthm
<p>Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?</p>
<h2>Assumptions</h2>
<ol>
<li>I assume getting output based on some input is, at least, hundreds of times faster than training such a model.</li>
<li>I assume no additional output parsing/input limitations are used. In particular I can imagine all the boiler plate to keep the ChatGPT model(s) acting politically correct etc. may be a significant overhead. This is to be ignored for this question.</li>
</ol>
<h2>Data</h2>
<p>So far I've found the ChatGPT 3.5 model to have <a href="https://lifearchitect.ai/chatgpt/" rel="noreferrer">175 billion parameters</a>:</p>
<p><a href="https://i.sstatic.net/DeIZD.png" rel="noreferrer"><img src="https://i.sstatic.net/DeIZD.png" alt="enter image description here" /></a></p>
<p>Though I do not yet know how large that is in <code>Mb</code> nor do I have an idea on how long generating an output would typically take.</p>

CC BY-SA 4.0
2023-02-06T14:06:13.653Z
175 billion parameters means at least 175 billion numbers, means it's probably in the range of 175GB (if all numbers are 1 byte - unlikely) - 1.4 TB (if all numbers are 64bit = 8byte, more likely).
kutschkem
CC BY-SA 4.0
2023-02-09T02:53:41.343Z
... and apart from storage/RAM, also need the computation power to do the math and return an answer within reasonable time. So yes, it is feasible, if you have a few hundred thousand bucks to spend I guess.
lpounng
CC BY-SA 4.0
2023-02-09T02:56:14.690Z
Ah, don't forget the cooling and electricity bill.
lpounng
CC BY-SA 4.0
2023-03-22T03:26:17.640Z
@kutschkem Thdd we numbers are half-precision, so 175B parameters is 350 GB.
Charles
<open-ai><chatgpt><open-source>
Would self-hosting ChatGPT be feasible, w.r.t. computation costs?
a.t.
# Would self-hosting ChatGPT be feasible, w.r.t. computation costs?

(Asked by: username_0 on 2023-02-04)

Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?

## Assumptions

1. I assume getting output based on some input is, at least, hundreds of times faster than training such a model.
2. I assume no additional output parsing/input limitations are used. In particular I can imagine all the boiler plate to keep the ChatGPT model(s) acting politically correct etc. may be a significant overhead. This is to be ignored for this question.

## Data

So far I've found the ChatGPT 3.5 model to have [175 billion parameters](https://lifearchitect.ai/chatgpt/):

[![enter image description here](https://i.sstatic.net/DeIZD.png)](https://i.sstatic.net/DeIZD.png)

Though I do not yet know how large that is in `Mb` nor do I have an idea on how long generating an output would typically take.

**Tags:** open-ai, chatgpt, open-source

> **Comments:**
> **username_1**: 175 billion parameters means at least 175 billion numbers, means it's probably in the range of 175GB (if all numbers are 1 byte - unlikely) - 1.4 TB (if all numbers are 64bit = 8byte, more likely).  
> **username_2**: ... and apart from storage/RAM, also need the computation power to do the math and return an answer within reasonable time. So yes, it is feasible, if you have a few hundred thousand bucks to spend I guess.  
> **username_2**: Ah, don't forget the cooling and electricity bill.  
> **username_3**: @username_1 Thdd we numbers are half-precision, so 175B parameters is 350 GB.  

---

## 2 answers

### Answer by username_6 on 2023-03-13. Score: 6 | Accepted answer

Here, I assume the ChatGPT architecture is similar to the published GPT-3 model [1].

The [Transformer](https://en.wikipedia.org/wiki/Transformer_(machine_learning_model)) architecture used for GPT3 is much more memory-bound than compute-bound at a high parameter count and a low context size. A large batch size (3.2M across ~few thousands GPUs for GPT-3) is used during training, but relatively low computational power is needed at the inference time for non-batched applications such as real-time chat. OpenAI is probably batching multiple users' requests to improve the throughput.

GPT-3 has 175B parameters and would require 326GiB memory to store the weight in float16. *Quantization* methods are proposed to reduce the memory required to store model weights, where the model weights are stored in lower precision. In the GPTQ paper [2], the authors have shown that large GPT models perform almost as good as the original model with 4-bit per parameter, reducing the 175B model to less than 90GiB. 3-bit quantization is also considered, reducing the size to less than 70GiB.
Note that additional memory required for internal states is small compared to the weights, especially if a [memory-efficient attention](https://github.com/facebookresearch/xformers) is used.

Recently, such quantization methods have gained attention due to the release of the pre-trained [LLaMA](https://github.com/facebookresearch/llama/) models.
Some people are running 65B LLaMA model even CPU-only [3] [4] with a reasonable throughput.

Therefore, for the question of whether it would be possible to run a 175B model in a home computer, I would say it is currently stretch but certainly possible if somewhat slow responses are acceptable.

* [1]: Brown, Tom, et al. "Language models are few-shot learners." Advances in neural information processing systems 33 (2020): 1877-1901. <https://arxiv.org/abs/2005.14165>
* [2]: Frantar, Elias, et al. "GPTQ: Accurate Post-Training Quantization for Generative Pre-trained Transformers." (2022). <https://arxiv.org/abs/2210.17323>
* [3]: [Satyajit Ghana on Twitter](https://twitter.com/thesudoer_/status/1634883960525422597)
* [4]: [Taiga Takano on Twitter](https://twitter.com/tg3517/status/1635095157480759297)


---

### Answer by username_4 on 2023-02-11. Score: 4

> Suppose the pre-trained, current date (2023-02-04) ChatGPT model was released open source, would it be feasible for regular users to interact with the model on a self-hosted computer?

No, assuming it's regular, consumer hardware. However, some open-source alternatives aim to be runnable on consumer hardware, e.g. <https://github.com/LAION-AI/Open-Assistant> (not released yet):

> We want to do this in a way that is open and accessible, which means we must not only build a great assistant, but also make it small and efficient enough to run on consumer hardware.

There exist however smaller language models, e.g. see this [survey of small language models](https://arxiv.org/pdf/2410.20011).

> **Comments:**
> **username_5**: the moment this becomes feasible, every thing changes again, for good and for bad.  

---

