Answers.Body: <p><strong>Thanks for the tip @Yoric!</strong>
  (Sorry, my reputation points are too low to comment, so I have to do this answer-variant)</p>
  <p><strong>Alright, here's the lowdown on the energy use of AI from the paper <a href="https://www.sciencedirect.com/science/article/pii/S2542435123003653#fig1" rel="nofollow noreferrer">&quot;The growing energy footprint of artificial intelligence&quot;</a></strong></p>
  <ul>
  <li><p>AI and Energy Use: AI's getting big, and so is its energy appetite.<br />
  People are starting to notice how much electricity AI and data<br />
  centers are gobbling up, and it's got some environmental impacts.</p>
  </li>
  <li><p>AI Training Takes a Lot of Power &amp; time = energy!: Training AI models is a real energy
  hog. For instance, the BLOOM model used up 433 MWh, GPT-3 needed a whopping 1287 MWh, Gopher 1066 MWh, and OPT 324 MWh. That's a lot of <strike>power</strike> energy!</p>
  </li>
  <li><p>AI Working Overtime: When AI models like ChatGPT get down to
  answering our questions, they're using a good chunk of energy.<br />
  OpenAI's ChatGPT needs about 564 MWh every day just for this.</p>
  </li>
  <li><p>More AI, More Power?: If we start using AI in things like Google
  Search, the power use could jump up. We're talking about 6.9–8.9 Wh
  for every AI-powered search.</p>
  </li>
  <li><p>Getting Smarter and Greener: Good news is, better hardware and
  smarter AI could help cut down on the power use. But, there's a catch - as AI gets more efficient, we might just end up using it more.</p>
  </li>
  <li><p>Balancing Act: The paper really highlights that we need to balance
  the cool stuff AI does with how much energy it uses. We've       got
  to think about both sides of the coin.</p>
  </li>
  </ul>
  <p><strong>I have done some research</strong> from multiple sources regarding the energy consumption and operational efficiency of OpenAI's ChatGPT-4 Turbo, but specific details that can help to calcualate its average energy expenditure per query are not to be found. Cntacting OpenAI directly for specific data on ChatGPT's energy consumption is needed.</p>
  <ul>
  <li><p>ChatGPT-4 Turbo's Notable Features: This model stands out for its impressive speed and efficiency. It's got a huge context window of 128k, meaning it can handle a conversation as lengthy as 300 pages. Plus, it's up-to-date with events until April 2023. When it comes to cost, it's more efficient, cutting down input token costs by 3x and output token costs by 2x compared to GPT-4.</p>
  </li>
  <li><p>A Nod to Better Performance: OpenAI has enhanced ChatGPT-4 Turbo's performance, which likely translates to better energy efficiency. However, we're short on exact energy figures for this.</p>
  </li>
  <li><p>Comparing with the Past: Previous models like GPT-3 had a reputation for high energy use, especially during training. While ChatGPT-4 Turbo is thought to be more efficient in query responses, the specific energy figures aren't disclosed.</p>
  </li>
  <li><p>A Friendly Comparison: If ChatGPT-4 Turbo is indeed more energy-efficient, imagine its energy use being similar to a short smartphone charge rather than brewing a coffee, the analogy once used for Google searches. But remember, without solid data, this is more of an educated guess. To get the full picture, detailed info from OpenAI or expert analysis would be key.</p>
  </li>
  </ul>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-01-08T13:08:39.893Z
Answers.Text: Minor nitpick: "That's a lot of power!" talking about MWh, it should be "That's a lot of energy!"
Answers.UserDisplayName: Eric Duminil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-17T05:39:24.553Z
Answers.Text: thanks for the link to that paper and your answer, @EarthHunger.
  
  searching for open access to the paper I came across another paper that was discursive in nature (and over-generalising I thought, reaching hysterical level of assumption in suggesting crypto will help solve climate change LOL)  rather than quantitative. The author concluded with the following sentence:
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-17T05:40:03.433Z
Answers.Text: "Dodge concurs, noting that in his research, a lot of the CO2 emissions were calculated from the electric- ity consumed in training the model. “Choosing the region that you’re train- ing your model in or putting your mod- el into production can have a pretty big impact,” Dodge says. “We found that training in the least carbon-in- tense region, compared with the most carbon-intense region, could reduce your emissions by about two-thirds, to one-third of what the full emissions would’ve been.”
  
  DOI: 10.1145/3603746
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-17T05:40:09.597Z
Answers.Text: [The growing energy footprint of artificial intelligence](https://web.p.ebscohost.com/ehost/detail/detail?vid=1&sid=3666b8d2-6228-49cb-b28e-125c64a6a334%40redis&bdata=JkF1dGhUeXBlPXNoaWImc2l0ZT1laG9zdC1saXZl#AN=166919231&db=bsu)
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-03-18T07:33:42.207Z
Answers.Text: Thanks for the input @wide_eyed_pupil. Can you link to the paper outside of a log-in-wall? Much appriciated!
Answers.UserDisplayName: EarthHunger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-05-08T00:52:15.783Z
Answers.Text: hi @EarthHunger here's a DOI, not sure I can find you an opensource url yet, https://dx.doi.org/10.1016/j.joule.2023.09.004 — seemingly my own university library account no longer has access to the full text!
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-17T10:59:35.700Z
Answers.LastActivityDate: 2024-03-18T07:36:00.380Z
Answers.LastEditDate: 2024-03-18T07:36:00.380Z
Answers.UserDisplayName: EarthHunger
Answers.Body: <p>I found a reply in <a href="https://www.sciencedirect.com/science/article/pii/S2542435123003653#fig1" rel="nofollow noreferrer">this research paper</a>. According to them, it's ~3Wh per query.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-01T16:31:56.770Z
Answers.LastActivityDate: 2023-11-01T16:31:56.770Z
Answers.UserDisplayName: Yoric
Answers.Body: <p>I've taken a stab at estimating the carbon footprint of ChatGPT <a href="https://towardsdatascience.com/the-carbon-footprint-of-chatgpt-66932314627d" rel="nofollow noreferrer">here</a>. I estimated the daily carbon footprint of the ChatGPT service to be around 23 kgCO2e and the primary assumption was that the service was running on 16 A100 GPUs. I made the estimate at a time with little information about the user base was available. I now believe that the estimate is way too low because ChatGPT reportedly had 590M visits in January which I don't think 16 gpus can handle.</p>
  <p>Recently, I also estimated <a href="https://towardsdatascience.com/chatgpts-electricity-consumption-7873483feac4" rel="nofollow noreferrer">ChatGPT's electricity consumption</a> in January 2023 to be between 1.1M and 23M KWh.</p>
  <p>To convert that into a carbon footprint, we'd need to know the carbon intensity of the electricity grid in every location where a ChatGPT instance is running. We don't have this info, but if we instead convert the electricity consumption into a carbon footprint using a very low carbon intensity like Sweden's 9g / KWh (which is the lowest in EU and lower than the US), the carbon footprint of ChatGPT in January 2023 would be estimated to be between 10 and 207 tons CO2e.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-04T15:47:03.593Z
Answers.Text: I asked CHatGTP what cloud computing services openAI use… it listed four of the big ones…
  > 1. AWS, 2. Microsoft Azure, 3. Google Cloud Platform (GVP) 4. IBM Cloud… just ask ChatGTP for details, but it doesn't get into specifics or proportions of workloads used across these services used by ChatGTP so it's all pretty vague.
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-05T09:46:05.147Z
Answers.Text: Since ChatGPT is known to hallucinate, I don't really trust ChatGPT's answers in this regard. Since OpenAI has what appears to be a pretty close partnership with Microsoft, I'd be very surprised if ChatGPT also run on other cloud vendors :) btw I've asked ChatGPT about its carbon footprint many times, but never gotten anything useful. But perhaps I'm not good enough at prompt engineering
Answers.UserDisplayName: KasperGL
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T05:29:31.960Z
Answers.Text: true, I often correct it and sometimes for fun I correct it with a lie and then it repeats the lie back to me like I schooled it in something. :-)
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T05:31:51.020Z
Answers.Text: it's pretty amazing that it's not a question that one hundred blogger, podcasters and YouTubers haven't asked. if the C footprint is in the scale of 10^3 or higher and literally all information based website start using it or similar architectures, (which is a lot if you think about it a little) then that's a huge added load on the worlds mostly dirty power systems.
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-07T07:21:14.390Z
Answers.Text: off-topic, but do you know if humans ever intervene to redirect ChatGTP-3? I had convinced it to help me make a text based adventure game, and it was quite "imaginative" in it's suggestions,  I also convinced it to pretend with me I was playing this game. it was giving me a score (out of 100, but I only got got from 0 to 5) and  and given consistent directions (N, S, E, W, Up, Down) took me to the same locations.
  But a few days later when I typed score I got a default message saying it was a [chatbot] by OpenAI and couldn't simulate playing the game etc
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-11-30T13:35:30.057Z
Answers.Text: 16 GPUs? It’s more like 30,000.
Answers.UserDisplayName: Benji XVI
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-04T09:47:03.727Z
Answers.LastActivityDate: 2023-03-04T09:47:03.727Z
Answers.UserDisplayName: KasperGL
Answers.Body: <p>Sam Altman states &quot;probably single-digits cents&quot; thus worst case 0,09€/request.</p>
  <p>I guess a least half the cost are energy at a cost of   0,15€/1kWh, a request would cost 0,09€/request*50%/0,15€/1kW=0,3kWh/request = 300Wh per request. 60 Smartphone charges of 5Wh per Charge ;)
  Source:<a href="https://www.forbes.com/sites/ariannajohnson/2022/12/07/heres-what-to-know-about-openais-chatgpt-what-its-disrupting-and-how-to-use-it/" rel="noreferrer">https://www.forbes.com/sites/ariannajohnson/2022/12/07/heres-what-to-know-about-openais-chatgpt-what-its-disrupting-and-how-to-use-it/</a></p>
  <p>Google Search request 0.0003 kWh = 0,3Wh, thus a search request by Google uses 1000x less, but as Google has started to use AI to, probably a search consumes more by now as well.
  Source: <a href="https://store.chipkin.com/articles/did-you-know-it-takes-00003-kwh-per-google-search-and-more" rel="noreferrer">https://store.chipkin.com/articles/did-you-know-it-takes-00003-kwh-per-google-search-and-more</a></p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-03T17:14:39.817Z
Answers.Text: Correct the difference between € and $ are not so important, compared to the other factors, like energy cost. As I live in the EU, it is harder for me to estimate the energy price OpenAI might pay, may you provide that, so I can update my estimate? 
  I don't have the knowledge neither inside to calculate from GPU usage of the model. I am looking forward to that calculation to see how off I was.
Answers.UserDisplayName: KFilter
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-17T02:57:15.330Z
Answers.Text: yes, like CAPEX and OPEX, there would be training phase and embodied energy of the GPU servers (embodied energy as a one off energy and emissions 'cost' equivalent of CAPEX) and the energy consumption of the servers since ChatGPT was opened up for public access. I'm guessing operational energy is the majority of energy use.
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-17T03:00:02.570Z
Answers.Text: @Kfilter depends if they are paying for energy on a Power Purchase Agreement (probably if their consumption is high) and if it's 100% RE or dirty power supply. depends what state the servers are operating it too (they might be in Greenland or Alaska running on geothermal with cheap cooling with the freezing air exchange outside). I'll try and find a wholesale power price. Do you kwon where their servers are located geographically?
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T16:54:06.543Z
Answers.Text: It’s worth noting that Sam Altman and OpenAI are American, so USD and American energy costs are more relevant than Euros.  Also, we know [ChatGPT is trained from a GPT3.5 foundation model](https://openai.com/blog/chatgpt/), so the architecture and energy expenditure can be estimated from that direction as well.  (I assume that running the language model itself absolutely dominates the energy intensity, compared to e.g web servers, moderation models, etc.)
Answers.UserDisplayName: kdbanman
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T12:48:44.677Z
Answers.LastActivityDate: 2023-02-02T12:48:44.677Z
Answers.UserDisplayName: KFilter
Answers.Body: <p>So I asked ChatGPT-4 about its energy use per query. The dialog is appended at the bottom of this message. (I realize the OP asked about ChatGPT-3)</p>
  <h2>TL;DR</h2>
  <p>If ChatGPT-4's claim that a typical answer requires 30 TFLOPs on modern AI hardware is true then the computational energy per answer is typically 300 watt-seconds (less than 0.1 kWh), ignoring communication and other overhead.</p>
  <h2>Discussion</h2>
  <p>ChatGPT-4 stated that most queries are answered with a single forward pass through the model requiring about 30 trillion FLOPs. It produced a calculation this requires around 0.000833 kWh but the calculation contains at least two errors:</p>
  <ol>
  <li>It uses FLOPs/w to compute watt-seconds.</li>
  <li>It screws up the conversion of watt-second to
  kWh by a factor of 1000.</li>
  </ol>
  <p>I found a <a href="https://arxiv.org/pdf/2210.17331.pdf" rel="nofollow noreferrer">recent paper</a> from researchers at SLAC and MIT with some energy per operation data.  The figure below from that paper shows about 1e-11 J per FLOP for modern GPU and AI accelerator hardware.
  <a href="https://i.sstatic.net/RxYgh.png" rel="nofollow noreferrer"><img src="https://i.sstatic.net/RxYgh.png" alt="enter image description here" /></a></p>
  <p><strong>If we take ChatGPT-4's 30 TFLOP/answer at face value, then a typical answer would consume 30e12 FLOP * 1e-11 J/FLOP = 300 J = 8.33e-5 kWH</strong></p>
  <p>This is a factor of 10 smaller than ChatGPT4's error-laden calculation but in the opposite direction.  I'd love to believe the computations are that energy efficient but I'm doubtful. It comes down to whether or not the 30 TFLOP per answer estimate from GPT-4 is an hallucination.</p>
  <hr />
  <h2>Dialog with GPT4</h2>
  <p><em><strong>How much energy did you expend to answer this query?</strong></em>
  As an AI language model, I don't consume energy directly in the traditional sense. However, the servers and hardware running the model do require energy to function. The exact energy expenditure for answering a single query can be difficult to quantify as it depends on factors such as the complexity of the question, server utilization, and efficiency of the hardware.</p>
  <p>In general, the energy usage for running an AI model like me is measured in floating point operations per second (FLOPs) and the energy efficiency of the hardware in joules per FLOP. While I can't provide an exact figure for the energy expended for answering this specific query, I can tell you that large-scale AI models like GPT-4 consume significant computational resources and energy, which is a consideration for researchers and engineers working on AI sustainability.</p>
  <p><em><strong>Please provide an approximate energy expenditure in kWH for answering typical queries.</strong></em>
  It is challenging to provide a precise energy expenditure for answering a typical query, as the energy usage depends on various factors such as the model's complexity, infrastructure efficiency, and hardware. However, I can give you a rough estimate based on some assumptions.</p>
  <p>Let's assume that a single forward pass of GPT-4 requires about 30 trillion floating-point operations (FLOPs) – this is a rough estimate, as the actual number of FLOPs can vary depending on the model size and input length. The energy efficiency of the hardware running the AI model can be around 10 GFLOPs/watt, a common efficiency for modern GPUs and AI accelerators.</p>
  <p>To calculate the energy usage for a single forward pass:</p>
  <p>Energy usage (in watt-seconds) = (30 trillion FLOPs) / (10 GFLOPs/watt) = 3,000,000 watt-seconds</p>
  <p>Now we convert watt-seconds to kilowatt-hours (kWh):</p>
  <p>Energy usage (in kWh) = (3,000,000 watt-seconds) * (1 kW / 1,000 W) * (1 hour / 3,600 seconds) ≈ 0.000833 kWh</p>
  <p>So, a very rough approximation of the energy usage for a single forward pass to answer a typical query is around 0.000833 kWh. Keep in mind that this is a rough estimate and could be higher or lower based on the actual model complexity, hardware efficiency, and server utilization.</p>
  <p><em><strong>How many forward passes does GPT-4 typically require to answer a typical query?</strong></em>
  GPT-4, like other generative language models, is based on the Transformer architecture, which uses a decoder-only design for generating text. For a typical query, GPT-4 usually requires one forward pass through the model to generate a response.</p>
  <p>During the forward pass, the model processes the input tokens and generates a probability distribution over the vocabulary for the next token at each position in the sequence. The process continues sequentially until an end-of-sequence token is generated or a predetermined maximum length is reached.</p>
  <p>In summary, GPT-4 generally needs one forward pass to answer a typical query. However, it's important to note that, in practice, additional computational overhead may be involved in tokenization, decoding, and other preprocessing and postprocessing steps.</p>
  <p>Share</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-31T13:06:49.927Z
Answers.Text: Nice! The 8.33e-5 seems reasonable. BLOOM once used 0.00396 KWh per query (https://arxiv.org/abs/2211.02001) and  using the ChatGPT user and GPU numbers from an article on SemiAnalysis, we arrive at 0.000712 KWh per query. Daily requests: 15 x 13m = 195,000,000, which takes 28,936 A100 GPUs (https://www.semianalysis.com/p/the-inference-cost-of-search-disruption?utm_medium=email)
  
  Energy consumption per query assuming power draw is 50% of max TDP: 28,936 GPUs * (0.4KW/2) * 24 = 0,000712 KWh
Answers.UserDisplayName: KasperGL
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-20T15:38:58.117Z
Answers.LastActivityDate: 2023-03-20T15:38:58.117Z
Answers.UserDisplayName: Mike Ellis
Answers.Body: <p>Great topic! I'd love to see this knowledge added as a subsection to <a href="https://en.wikipedia.org/wiki/ChatGPT" rel="nofollow noreferrer">Wikipedia's ChatGPT page</a>. I'll look into getting it started.</p>
  <p>Out of curiosity, I posed the same questions to GPT-4, approximately 7 months later. Here's its answer:</p>
  <blockquote>
  <p>How much energy did you expend to answer this query? ChatGPT</p>
  <p>I don't consume energy in the same way traditional electronics do.
  Instead, the servers that run my underlying computations use energy.
  The precise energy consumption for a single interaction with me,
  ChatGPT, depends on multiple factors such as:</p>
  <ol>
  <li>The specific hardware used for the computation.</li>
  <li>The complexity and length of the request and the produced answer.</li>
  <li>Backend optimizations and efficiency of the software.</li>
  </ol>
  <p>Typically, running a query on a model like GPT-4 consumes a small
  amount of energy, often measured in milliwatt-hours (mWh) or
  watt-hours (Wh), but certainly not kilowatt-hours (kWh). For a brief
  interaction like this one, the energy consumed would be closer to the
  mWh range.</p>
  <p>For a more precise measurement, you would need detailed specifications
  from the provider hosting the model – in this case, OpenAI – about
  their infrastructure's energy consumption during inference. User
  Please provide an approximate energy expenditure in kWH for answering
  typical queries.  ChatGPT</p>
  <p>It's challenging to provide a precise energy consumption value for a
  single query without specific details from OpenAI's infrastructure.
  However, I can provide a very rough estimation based on general
  knowledge of server operation.</p>
  <p>Let's make some assumptions:</p>
  <ol>
  <li>Assume a server running the GPT-4 model uses around 200 watts (0.2 kW) when active.</li>
  <li>Assume the time taken to process a query and produce a response is 0.1 seconds (100 milliseconds), which is a generous estimate and likely an overestimation.</li>
  </ol>
  <p>Given these assumptions:</p>
  <p>Energy = Power × Time</p>
  <p>Energy = 0.2 kW × 0.1/3600 hours (because 0.1 seconds is 0.1/3600 of
  an hour)</p>
  <p>Energy ≈ 5.56 × 10^-5 kWh, or 0.0000556 kWh</p>
  <p>This is a very rough estimation and is likely on the higher side for a
  single query. The actual energy expenditure could be lower due to
  various factors, including optimizations in server hardware, the
  efficiency of the model, and software improvements. Also, remember
  that this is a per-query estimate; when dealing with millions of
  queries, these numbers add up. Still, for an individual interaction,
  it's quite low. User How many forward passes does GPT-4 typically
  require to answer a typical query?  ChatGPT</p>
  <p>GPT-4, like its predecessors, requires a single forward pass through
  the model to generate a response for a typical query. When you send a
  query or prompt, the model processes the input through its layers and
  produces an output sequence, which is the response you receive. This
  whole process is a single forward pass.</p>
  <p>However, there are some nuances:</p>
  <ol>
  <li><p>Length: If the generated response exceeds the model's maximum token (word piece) limit, the completion might be cut off, and additional
  logic or processing might be needed to handle this. But this doesn't
  necessarily mean multiple forward passes for the model itself, just
  possibly additional computation.</p>
  </li>
  <li><p>Iterative Approaches: Some implementations or applications might use iterative or feedback-driven approaches where the output of one
  forward pass is fed back into the model for refinement. This isn't the
  standard mode of operation for a single response but is a technique
  that can be employed in specific scenarios.</p>
  </li>
  <li><p>Sampling and Temperature: The model can generate responses using techniques like nucleus sampling or by adjusting the &quot;temperature&quot; of
  the output, which affects the randomness of the generated text. These
  techniques might influence the nature of the output but still operate
  within a single forward pass.</p>
  </li>
  </ol>
  <p>For a typical interaction, like most questions and responses in this
  chat format, only one forward pass is needed.</p>
  </blockquote>
  <p>Thoughts?</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-16T06:35:16.157Z
Answers.Text: I got quite a few evasive answers before I got a more specific answer from ChatGTP 3.5! The numbers youve cited from ChatGTP 4 are significant;y lower than human estimates so far on this thread. Self-delusional-hallucinations?
Answers.UserDisplayName: wide_eyed_pupil
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-17T10:39:04.463Z
Answers.Text: Yes, good point. That seems likely 
Answers.UserDisplayName: EarthHunger
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2024-02-17T20:06:10.933Z
Answers.Text: That "calculation" just gives what amount of energy a midsized server takes in 0.1s. Well, I'd be extremely surprised if that's enough to produce a ChatGPT-4 response. It probably takes a whole cluster of such machines given the memory size of the model alone.
Answers.UserDisplayName: Tilman Vogel
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-10-11T07:26:57.227Z
Answers.LastActivityDate: 2023-10-11T07:26:57.227Z
Answers.UserDisplayName: EarthHunger
Body: <p>I note <a href="https://ai.stackexchange.com/questions/22877/how-much-computing-power-does-it-cost-to-run-gpt-3">this question</a> was deemed off-topic, so I'm trying to clearly frame <em>this</em> <em>question</em> in terms of scope of response I'm interested in, namely <strong>ethics</strong> and <strong>sustainability</strong> issues associated with the soon-to-be proliferation of OpenAI Chat GPT types of tools for all manner of online information seeking behavior (from humans and other bots). <em>This is not a programming or specific hardware question.</em></p>
  <p><strong>On average, how much energy is consumed for each response that Open AI's public chatgpt-3 provides?</strong>
  i.e. what is the energy to run the entire system for 24 hours divided by the number of responses generated in 24 hours (ignoring energy consumed to train the system or build the hardware components).</p>
  <p>How does this compare to a Google/Duck Duck Go/Bing search inquiry?</p>
  <p>I read somewhere an OpenAI employee on the ChatGPT team that the computer power used to provide responses to queries is &quot;ridiculous&quot;, and there's documentation of the size of the memory requirements of hosting servers and parameters but without knowing its throughput for example it's hard to quantify the energy consumption.</p>
  <p>I often get more interesting results from Chat GPT than Duck Duck Go on certain types of queries where I used to know the answer but cannot remember the answer. IN these cases I can fact check for myself, I'm looking for a memory prompts with names and jargon that will remind me.</p>
  <p>Also when seeking out counter-views to my own (say critiques of degrowth or heterodoxy economics concepts) Chat GPT is good at providing names and papers/reports/books that critiques the view I provide it.</p>
  <p>In many cases more usefully than conventional search engines. Therefore, I can see the popularity of these tools ballooning rapidly, especially when the operational costs CAPEX + OPEX of the servers and maintainers is borne by large amounts of seed funding (eg OpenAI) or any other loss-leader startup wishing to ride the next wave of AI.</p>
  <p>The heart of my question is &quot;at what <code>externalized</code> costs do we gain these tools in terms of greenhouse gases, use of limited mineral resources, GPUs scarcity etc.&quot;</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-11-06T15:01:03.377Z
Comments.Text: GHG emissions depend on the energy source (renewable, nuclear, fossil) and also whether the waste heat is used.
Comments.UserDisplayName: darsie
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-01-08T13:05:38.053Z
Comments.Text: "ignoring energy consumed to train the system"? Why??? Without knowing the orders of magnitude involved, you really shouldn't ignore the training. According to https://www.reddit.com/r/GPT3/comments/p1xf10/comment/h8h3sl4/?utm_source=share&utm_medium=web2x&context=3 ,  10_000 V100 GPUs were used, and the training cost $5M in compute time, for GPT-3 alone.
Comments.UserDisplayName: Eric Duminil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-17T05:46:02.520Z
Comments.Text: @EricDuminil you are right. I just found a paper/article that was discursive rather than quantitative but it said that the training of large models would likely use the more energy than that used to query it, but I guess that all depends on the costs/price to access the model for users and the time it stays in the public domain. The article also made the point that if AI is used for enhanced oil recovery (EOR) then the carbon footprint for Oil and Gas companies using the model rather than the training of the model is going to be massive by comparison. given methane leaks & combustion GHGs.
Comments.UserDisplayName: wide_eyed_pupil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-03-17T05:48:05.913Z
Comments.Text: @EricDuminil one reason I said “ignoring training” was because I thought it would be much harder to quantify using reliable source(s) than the servers in operation delivering the returns on prompts.
Comments.UserDisplayName: wide_eyed_pupil
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-31T06:51:07.393Z
Comments.Text: Note the environment tag is for reinforcement learning, not the physical environment of earth. It has nothing to do with your question.
Comments.UserDisplayName: chessprogrammer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-31T11:46:29.567Z
Comments.Text: Unless there is someone from the OpenAI team reading your question, no one here can answer this since they have not published any concrete information about the final design or supporting hardware architecture.
Comments.UserDisplayName: David Hoelzer
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-01T20:54:40.517Z
Comments.Text: Other answers on this forum have have very general descriptions of the size of  GPU architecture, from which power consumption could be estimated assuming a constant workload. What i haven't seen in public domain is a estimate of the throughput of response numbers per second/hour/day/week.
Comments.UserDisplayName: wide_eyed_pupil
Tags: <social><ethics><chatgpt><green-ai>
Title: How much energy consumption is involved in Chat GPT responses being generated?
UserDisplayName: wide_eyed_pupil
ThreadText: # How much energy consumption is involved in Chat GPT responses being generated?
  
  (Asked by: username_0 on 2023-01-31)
  
  I note [this question](https://ai.stackexchange.com/questions/22877/how-much-computing-power-does-it-cost-to-run-gpt-3) was deemed off-topic, so I'm trying to clearly frame *this* *question* in terms of scope of response I'm interested in, namely **ethics** and **sustainability** issues associated with the soon-to-be proliferation of OpenAI Chat GPT types of tools for all manner of online information seeking behavior (from humans and other bots). *This is not a programming or specific hardware question.*
  
  **On average, how much energy is consumed for each response that Open AI's public chatgpt-3 provides?**
  i.e. what is the energy to run the entire system for 24 hours divided by the number of responses generated in 24 hours (ignoring energy consumed to train the system or build the hardware components).
  
  How does this compare to a Google/Duck Duck Go/Bing search inquiry?
  
  I read somewhere an OpenAI employee on the ChatGPT team that the computer power used to provide responses to queries is "ridiculous", and there's documentation of the size of the memory requirements of hosting servers and parameters but without knowing its throughput for example it's hard to quantify the energy consumption.
  
  I often get more interesting results from Chat GPT than Duck Duck Go on certain types of queries where I used to know the answer but cannot remember the answer. IN these cases I can fact check for myself, I'm looking for a memory prompts with names and jargon that will remind me.
  
  Also when seeking out counter-views to my own (say critiques of degrowth or heterodoxy economics concepts) Chat GPT is good at providing names and papers/reports/books that critiques the view I provide it.
  
  In many cases more usefully than conventional search engines. Therefore, I can see the popularity of these tools ballooning rapidly, especially when the operational costs CAPEX + OPEX of the servers and maintainers is borne by large amounts of seed funding (eg OpenAI) or any other loss-leader startup wishing to ride the next wave of AI.
  
  The heart of my question is "at what `externalized` costs do we gain these tools in terms of greenhouse gases, use of limited mineral resources, GPUs scarcity etc."
  
  **Tags:** social, ethics, chatgpt, green-ai
  > **Comments:**
  > **username_3**: Note the environment tag is for reinforcement learning, not the physical environment of earth. It has nothing to do with your question.  
  > **username_4**: Unless there is someone from the OpenAI team reading your question, no one here can answer this since they have not published any concrete information about the final design or supporting hardware architecture.  
  > **username_0**: Other answers on this forum have have very general descriptions of the size of  GPU architecture, from which power consumption could be estimated assuming a constant workload. What i haven't seen in public domain is a estimate of the throughput of response numbers per second/hour/day/week.  
  > **username_1**: GHG emissions depend on the energy source (renewable, nuclear, fossil) and also whether the waste heat is used.  
  > **username_2**: "ignoring energy consumed to train the system"? Why??? Without knowing the orders of magnitude involved, you really shouldn't ignore the training. According to https://www.reddit.com/r/GPT3/comments/p1xf10/comment/h8h3sl4/?utm_source=share&utm_medium=web2x&context=3 ,  10_000 V100 GPUs were used, and the training cost $5M in compute time, for GPT-3 alone.  
  > **username_0**: @EricDuminil you are right. I just found a paper/article that was discursive rather than quantitative but it said that the training of large models would likely use the more energy than that used to query it, but I guess that all depends on the costs/price to access the model for users and the time it stays in the public domain. The article also made the point that if AI is used for enhanced oil recovery (EOR) then the carbon footprint for Oil and Gas companies using the model rather than the training of the model is going to be massive by comparison. given methane leaks & combustion GHGs.  
  > **username_0**: @EricDuminil one reason I said “ignoring training” was because I thought it would be much harder to quantify using reliable source(s) than the servers in operation delivering the returns on prompts.  
  
  ---
  
  ## 6 answers
  
  ### Answer by username_9 on 2023-02-02. Score: 8
  
  Sam Altman states "probably single-digits cents" thus worst case 0,09€/request.
  
  I guess a least half the cost are energy at a cost of 0,15€/1kWh, a request would cost 0,09€/request\*50%/0,15€/1kW=0,3kWh/request = 300Wh per request. 60 Smartphone charges of 5Wh per Charge ;)
  Source:<https://www.forbes.com/sites/ariannajohnson/2022/12/07/heres-what-to-know-about-openais-chatgpt-what-its-disrupting-and-how-to-use-it/>
  
  Google Search request 0.0003 kWh = 0,3Wh, thus a search request by Google uses 1000x less, but as Google has started to use AI to, probably a search consumes more by now as well.
  Source: <https://store.chipkin.com/articles/did-you-know-it-takes-00003-kwh-per-google-search-and-more>
  
  > **Comments:**
  > **username_10**: It’s worth noting that Sam Altman and OpenAI are American, so USD and American energy costs are more relevant than Euros.  Also, we know [ChatGPT is trained from a GPT3.5 foundation model](https://openai.com/blog/chatgpt/), so the architecture and energy expenditure can be estimated from that direction as well.  (I assume that running the language model itself absolutely dominates the energy intensity, compared to e.g web servers, moderation models, etc.)  
  > **username_9**: Correct the difference between € and $ are not so important, compared to the other factors, like energy cost. As I live in the EU, it is harder for me to estimate the energy price OpenAI might pay, may you provide that, so I can update my estimate?  I don't have the knowledge neither inside to calculate from GPU usage of the model. I am looking forward to that calculation to see how off I was.  
  > **username_0**: yes, like CAPEX and OPEX, there would be training phase and embodied energy of the GPU servers (embodied energy as a one off energy and emissions 'cost' equivalent of CAPEX) and the energy consumption of the servers since ChatGPT was opened up for public access. I'm guessing operational energy is the majority of energy use.  
  > **username_0**: @Kfilter depends if they are paying for energy on a Power Purchase Agreement (probably if their consumption is high) and if it's 100% RE or dirty power supply. depends what state the servers are operating it too (they might be in Greenland or Alaska running on geothermal with cheap cooling with the freezing air exchange outside). I'll try and find a wholesale power price. Do you kwon where their servers are located geographically?  
  
  ---
  
  ### Answer by username_5 on 2023-11-17. Score: 5
  
  **Thanks for the tip @Yoric!**
  (Sorry, my reputation points are too low to comment, so I have to do this answer-variant)
  
  **Alright, here's the lowdown on the energy use of AI from the paper ["The growing energy footprint of artificial intelligence"](https://www.sciencedirect.com/science/article/pii/S2542435123003653#fig1)**
  
  * AI and Energy Use: AI's getting big, and so is its energy appetite.  
    People are starting to notice how much electricity AI and data  
    centers are gobbling up, and it's got some environmental impacts.
  * AI Training Takes a Lot of Power & time = energy!: Training AI models is a real energy
    hog. For instance, the BLOOM model used up 433 MWh, GPT-3 needed a whopping 1287 MWh, Gopher 1066 MWh, and OPT 324 MWh. That's a lot of power energy!
  * AI Working Overtime: When AI models like ChatGPT get down to
    answering our questions, they're using a good chunk of energy.  
    OpenAI's ChatGPT needs about 564 MWh every day just for this.
  * More AI, More Power?: If we start using AI in things like Google
    Search, the power use could jump up. We're talking about 6.9–8.9 Wh
    for every AI-powered search.
  * Getting Smarter and Greener: Good news is, better hardware and
    smarter AI could help cut down on the power use. But, there's a catch - as AI gets more efficient, we might just end up using it more.
  * Balancing Act: The paper really highlights that we need to balance
    the cool stuff AI does with how much energy it uses. We've got
    to think about both sides of the coin.
  
  **I have done some research** from multiple sources regarding the energy consumption and operational efficiency of OpenAI's ChatGPT-4 Turbo, but specific details that can help to calcualate its average energy expenditure per query are not to be found. Cntacting OpenAI directly for specific data on ChatGPT's energy consumption is needed.
  
  * ChatGPT-4 Turbo's Notable Features: This model stands out for its impressive speed and efficiency. It's got a huge context window of 128k, meaning it can handle a conversation as lengthy as 300 pages. Plus, it's up-to-date with events until April 2023. When it comes to cost, it's more efficient, cutting down input token costs by 3x and output token costs by 2x compared to GPT-4.
  * A Nod to Better Performance: OpenAI has enhanced ChatGPT-4 Turbo's performance, which likely translates to better energy efficiency. However, we're short on exact energy figures for this.
  * Comparing with the Past: Previous models like GPT-3 had a reputation for high energy use, especially during training. While ChatGPT-4 Turbo is thought to be more efficient in query responses, the specific energy figures aren't disclosed.
  * A Friendly Comparison: If ChatGPT-4 Turbo is indeed more energy-efficient, imagine its energy use being similar to a short smartphone charge rather than brewing a coffee, the analogy once used for Google searches. But remember, without solid data, this is more of an educated guess. To get the full picture, detailed info from OpenAI or expert analysis would be key.
  
  > **Comments:**
  > **username_2**: Minor nitpick: "That's a lot of power!" talking about MWh, it should be "That's a lot of energy!"  
  > **username_0**: thanks for the link to that paper and your answer, @EarthHunger.  searching for open access to the paper I came across another paper that was discursive in nature (and over-generalising I thought, reaching hysterical level of assumption in suggesting crypto will help solve climate change LOL)  rather than quantitative. The author concluded with the following sentence:  
  > **username_0**: "Dodge concurs, noting that in his research, a lot of the CO2 emissions were calculated from the electric- ity consumed in training the model. “Choosing the region that you’re train- ing your model in or putting your mod- el into production can have a pretty big impact,” Dodge says. “We found that training in the least carbon-in- tense region, compared with the most carbon-intense region, could reduce your emissions by about two-thirds, to one-third of what the full emissions would’ve been.”  DOI: 10.1145/3603746  
  > **username_0**: [The growing energy footprint of artificial intelligence](https://web.p.ebscohost.com/ehost/detail/detail?vid=1&sid=3666b8d2-6228-49cb-b28e-125c64a6a334%40redis&bdata=JkF1dGhUeXBlPXNoaWImc2l0ZT1laG9zdC1saXZl#AN=166919231&db=bsu)  
  > **username_5**: Thanks for the input @wide_eyed_pupil. Can you link to the paper outside of a log-in-wall? Much appriciated!  
  > **username_0**: hi @EarthHunger here's a DOI, not sure I can find you an opensource url yet, https://dx.doi.org/10.1016/j.joule.2023.09.004 — seemingly my own university library account no longer has access to the full text!  
  
  ---
  
  ### Answer by username_6 on 2023-11-01. Score: 3
  
  I found a reply in [this research paper](https://www.sciencedirect.com/science/article/pii/S2542435123003653#fig1). According to them, it's ~3Wh per query.
  
  
  ---
  
  ### Answer by username_7 on 2023-03-04. Score: 3
  
  I've taken a stab at estimating the carbon footprint of ChatGPT [here](https://towardsdatascience.com/the-carbon-footprint-of-chatgpt-66932314627d). I estimated the daily carbon footprint of the ChatGPT service to be around 23 kgCO2e and the primary assumption was that the service was running on 16 A100 GPUs. I made the estimate at a time with little information about the user base was available. I now believe that the estimate is way too low because ChatGPT reportedly had 590M visits in January which I don't think 16 gpus can handle.
  
  Recently, I also estimated [ChatGPT's electricity consumption](https://towardsdatascience.com/chatgpts-electricity-consumption-7873483feac4) in January 2023 to be between 1.1M and 23M KWh.
  
  To convert that into a carbon footprint, we'd need to know the carbon intensity of the electricity grid in every location where a ChatGPT instance is running. We don't have this info, but if we instead convert the electricity consumption into a carbon footprint using a very low carbon intensity like Sweden's 9g / KWh (which is the lowest in EU and lower than the US), the carbon footprint of ChatGPT in January 2023 would be estimated to be between 10 and 207 tons CO2e.
  
  > **Comments:**
  > **username_0**: I asked CHatGTP what cloud computing services openAI use… it listed four of the big ones… > 1. AWS, 2. Microsoft Azure, 3. Google Cloud Platform (GVP) 4. IBM Cloud… just ask ChatGTP for details, but it doesn't get into specifics or proportions of workloads used across these services used by ChatGTP so it's all pretty vague.  
  > **username_7**: Since ChatGPT is known to hallucinate, I don't really trust ChatGPT's answers in this regard. Since OpenAI has what appears to be a pretty close partnership with Microsoft, I'd be very surprised if ChatGPT also run on other cloud vendors :) btw I've asked ChatGPT about its carbon footprint many times, but never gotten anything useful. But perhaps I'm not good enough at prompt engineering  
  > **username_0**: true, I often correct it and sometimes for fun I correct it with a lie and then it repeats the lie back to me like I schooled it in something. :-)  
  > **username_0**: it's pretty amazing that it's not a question that one hundred blogger, podcasters and YouTubers haven't asked. if the C footprint is in the scale of 10^3 or higher and literally all information based website start using it or similar architectures, (which is a lot if you think about it a little) then that's a huge added load on the worlds mostly dirty power systems.  
  > **username_0**: off-topic, but do you know if humans ever intervene to redirect ChatGTP-3? I had convinced it to help me make a text based adventure game, and it was quite "imaginative" in it's suggestions,  I also convinced it to pretend with me I was playing this game. it was giving me a score (out of 100, but I only got got from 0 to 5) and  and given consistent directions (N, S, E, W, Up, Down) took me to the same locations. But a few days later when I typed score I got a default message saying it was a [chatbot] by OpenAI and couldn't simulate playing the game etc  
  > **username_8**: 16 GPUs? It’s more like 30,000.  
  
  ---
  
  ### Answer by username_11 on 2023-03-20. Score: 1
  
  So I asked ChatGPT-4 about its energy use per query. The dialog is appended at the bottom of this message. (I realize the OP asked about ChatGPT-3)
  
  ## TL;DR
  
  If ChatGPT-4's claim that a typical answer requires 30 TFLOPs on modern AI hardware is true then the computational energy per answer is typically 300 watt-seconds (less than 0.1 kWh), ignoring communication and other overhead.
  
  ## Discussion
  
  ChatGPT-4 stated that most queries are answered with a single forward pass through the model requiring about 30 trillion FLOPs. It produced a calculation this requires around 0.000833 kWh but the calculation contains at least two errors:
  
  1. It uses FLOPs/w to compute watt-seconds.
  2. It screws up the conversion of watt-second to
     kWh by a factor of 1000.
  
  I found a [recent paper](https://arxiv.org/pdf/2210.17331.pdf) from researchers at SLAC and MIT with some energy per operation data. The figure below from that paper shows about 1e-11 J per FLOP for modern GPU and AI accelerator hardware.
  [![enter image description here](https://i.sstatic.net/RxYgh.png)](https://i.sstatic.net/RxYgh.png)
  
  **If we take ChatGPT-4's 30 TFLOP/answer at face value, then a typical answer would consume 30e12 FLOP \* 1e-11 J/FLOP = 300 J = 8.33e-5 kWH**
  
  This is a factor of 10 smaller than ChatGPT4's error-laden calculation but in the opposite direction. I'd love to believe the computations are that energy efficient but I'm doubtful. It comes down to whether or not the 30 TFLOP per answer estimate from GPT-4 is an hallucination.
  
  ---
  
  ## Dialog with GPT4
  
  ***How much energy did you expend to answer this query?***
  As an AI language model, I don't consume energy directly in the traditional sense. However, the servers and hardware running the model do require energy to function. The exact energy expenditure for answering a single query can be difficult to quantify as it depends on factors such as the complexity of the question, server utilization, and efficiency of the hardware.
  
  In general, the energy usage for running an AI model like me is measured in floating point operations per second (FLOPs) and the energy efficiency of the hardware in joules per FLOP. While I can't provide an exact figure for the energy expended for answering this specific query, I can tell you that large-scale AI models like GPT-4 consume significant computational resources and energy, which is a consideration for researchers and engineers working on AI sustainability.
  
  ***Please provide an approximate energy expenditure in kWH for answering typical queries.***
  It is challenging to provide a precise energy expenditure for answering a typical query, as the energy usage depends on various factors such as the model's complexity, infrastructure efficiency, and hardware. However, I can give you a rough estimate based on some assumptions.
  
  Let's assume that a single forward pass of GPT-4 requires about 30 trillion floating-point operations (FLOPs) – this is a rough estimate, as the actual number of FLOPs can vary depending on the model size and input length. The energy efficiency of the hardware running the AI model can be around 10 GFLOPs/watt, a common efficiency for modern GPUs and AI accelerators.
  
  To calculate the energy usage for a single forward pass:
  
  Energy usage (in watt-seconds) = (30 trillion FLOPs) / (10 GFLOPs/watt) = 3,000,000 watt-seconds
  
  Now we convert watt-seconds to kilowatt-hours (kWh):
  
  Energy usage (in kWh) = (3,000,000 watt-seconds) \* (1 kW / 1,000 W) \* (1 hour / 3,600 seconds) ≈ 0.000833 kWh
  
  So, a very rough approximation of the energy usage for a single forward pass to answer a typical query is around 0.000833 kWh. Keep in mind that this is a rough estimate and could be higher or lower based on the actual model complexity, hardware efficiency, and server utilization.
  
  ***How many forward passes does GPT-4 typically require to answer a typical query?***
  GPT-4, like other generative language models, is based on the Transformer architecture, which uses a decoder-only design for generating text. For a typical query, GPT-4 usually requires one forward pass through the model to generate a response.
  
  During the forward pass, the model processes the input tokens and generates a probability distribution over the vocabulary for the next token at each position in the sequence. The process continues sequentially until an end-of-sequence token is generated or a predetermined maximum length is reached.
  
  In summary, GPT-4 generally needs one forward pass to answer a typical query. However, it's important to note that, in practice, additional computational overhead may be involved in tokenization, decoding, and other preprocessing and postprocessing steps.
  
  Share
  
  > **Comments:**
  > **username_7**: Nice! The 8.33e-5 seems reasonable. BLOOM once used 0.00396 KWh per query (https://arxiv.org/abs/2211.02001) and  using the ChatGPT user and GPU numbers from an article on SemiAnalysis, we arrive at 0.000712 KWh per query. Daily requests: 15 x 13m = 195,000,000, which takes 28,936 A100 GPUs (https://www.semianalysis.com/p/the-inference-cost-of-search-disruption?utm_medium=email)  Energy consumption per query assuming power draw is 50% of max TDP: 28,936 GPUs * (0.4KW/2) * 24 = 0,000712 KWh  
  
  ---
  
  ### Answer by username_5 on 2023-10-11. Score: 1
  
  Great topic! I'd love to see this knowledge added as a subsection to [Wikipedia's ChatGPT page](https://en.wikipedia.org/wiki/ChatGPT). I'll look into getting it started.
  
  Out of curiosity, I posed the same questions to GPT-4, approximately 7 months later. Here's its answer:
  
  > How much energy did you expend to answer this query? ChatGPT
  >
  > I don't consume energy in the same way traditional electronics do.
  > Instead, the servers that run my underlying computations use energy.
  > The precise energy consumption for a single interaction with me,
  > ChatGPT, depends on multiple factors such as:
  >
  > 1. The specific hardware used for the computation.
  > 2. The complexity and length of the request and the produced answer.
  > 3. Backend optimizations and efficiency of the software.
  >
  > Typically, running a query on a model like GPT-4 consumes a small
  > amount of energy, often measured in milliwatt-hours (mWh) or
  > watt-hours (Wh), but certainly not kilowatt-hours (kWh). For a brief
  > interaction like this one, the energy consumed would be closer to the
  > mWh range.
  >
  > For a more precise measurement, you would need detailed specifications
  > from the provider hosting the model – in this case, OpenAI – about
  > their infrastructure's energy consumption during inference. User
  > Please provide an approximate energy expenditure in kWH for answering
  > typical queries. ChatGPT
  >
  > It's challenging to provide a precise energy consumption value for a
  > single query without specific details from OpenAI's infrastructure.
  > However, I can provide a very rough estimation based on general
  > knowledge of server operation.
  >
  > Let's make some assumptions:
  >
  > 1. Assume a server running the GPT-4 model uses around 200 watts (0.2 kW) when active.
  > 2. Assume the time taken to process a query and produce a response is 0.1 seconds (100 milliseconds), which is a generous estimate and likely an overestimation.
  >
  > Given these assumptions:
  >
  > Energy = Power × Time
  >
  > Energy = 0.2 kW × 0.1/3600 hours (because 0.1 seconds is 0.1/3600 of
  > an hour)
  >
  > Energy ≈ 5.56 × 10^-5 kWh, or 0.0000556 kWh
  >
  > This is a very rough estimation and is likely on the higher side for a
  > single query. The actual energy expenditure could be lower due to
  > various factors, including optimizations in server hardware, the
  > efficiency of the model, and software improvements. Also, remember
  > that this is a per-query estimate; when dealing with millions of
  > queries, these numbers add up. Still, for an individual interaction,
  > it's quite low. User How many forward passes does GPT-4 typically
  > require to answer a typical query? ChatGPT
  >
  > GPT-4, like its predecessors, requires a single forward pass through
  > the model to generate a response for a typical query. When you send a
  > query or prompt, the model processes the input through its layers and
  > produces an output sequence, which is the response you receive. This
  > whole process is a single forward pass.
  >
  > However, there are some nuances:
  >
  > 1. Length: If the generated response exceeds the model's maximum token (word piece) limit, the completion might be cut off, and additional
  >    logic or processing might be needed to handle this. But this doesn't
  >    necessarily mean multiple forward passes for the model itself, just
  >    possibly additional computation.
  > 2. Iterative Approaches: Some implementations or applications might use iterative or feedback-driven approaches where the output of one
  >    forward pass is fed back into the model for refinement. This isn't the
  >    standard mode of operation for a single response but is a technique
  >    that can be employed in specific scenarios.
  > 3. Sampling and Temperature: The model can generate responses using techniques like nucleus sampling or by adjusting the "temperature" of
  >    the output, which affects the randomness of the generated text. These
  >    techniques might influence the nature of the output but still operate
  >    within a single forward pass.
  >
  > For a typical interaction, like most questions and responses in this
  > chat format, only one forward pass is needed.
  
  Thoughts?
  
  > **Comments:**
  > **username_0**: I got quite a few evasive answers before I got a more specific answer from ChatGTP 3.5! The numbers youve cited from ChatGTP 4 are significant;y lower than human estimates so far on this thread. Self-delusional-hallucinations?  
  > **username_5**: Yes, good point. That seems likely   
  > **username_12**: That "calculation" just gives what amount of energy a midsized server takes in 0.1s. Well, I'd be extremely surprised if that's enough to produce a ChatGPT-4 response. It probably takes a whole cluster of such machines given the memory size of the model alone.  
  
  ---
  
  