Answers.Body: <p>The model infers in an autoregressive way. That is, it needs the previous output to produce the next one, so it's slow because it computes. More <a href="https://lilianweng.github.io/posts/2023-01-10-inference-optimization/" rel="nofollow noreferrer">details</a>.</p>
  <p>It also depends on the language. The model was trained primarily on the English corpus and typically generates one or two tokens for each word. But for other languages, it might not be as efficient and can generate a token for each symbol</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-05-03T09:45:48.457Z
Answers.Text: Also you might notice that sometimes it will type out a response and then replace the whole text with a standard response once some other system finds out that the GPT output is violating the their policy in some way. GPT is slow, and the policy checking algorithm is run on intermediate output from GPT.
Answers.UserDisplayName: Tim Kuipers
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-21T12:13:34.127Z
Answers.LastActivityDate: 2023-02-21T12:34:59.623Z
Answers.LastEditDate: 2023-02-21T12:34:59.623Z
Answers.UserDisplayName: Aray Karjauv
Body: <p>When you give ChatGPT a prompt, does it</p>
  <p>A) generate the entire response instantly (at least on the server), but then &quot;buffers&quot; it slowly to the end user (for whatever reason)</p>
  <p>or</p>
  <p>B) it's actually generating words (and sometimes entire sentences at once) one by one, and using (server) computational power all that while?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-21T19:14:17.863Z
Comments.Text: This has already been asked here.
Comments.UserDisplayName: nbro
Tags: <chatgpt>
Title: Does ChatGPT type slowly on purpose or is it computing?
UserDisplayName: laggingreflex
ThreadText: # Does ChatGPT type slowly on purpose or is it computing?
  
  (Asked by: username_0 on 2023-02-21)
  
  When you give ChatGPT a prompt, does it
  
  A) generate the entire response instantly (at least on the server), but then "buffers" it slowly to the end user (for whatever reason)
  
  or
  
  B) it's actually generating words (and sometimes entire sentences at once) one by one, and using (server) computational power all that while?
  
  **Tags:** chatgpt
  > **Comments:**
  > **username_1**: This has already been asked here.  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_2 on 2023-02-21. Score: 4 | Accepted answer
  
  The model infers in an autoregressive way. That is, it needs the previous output to produce the next one, so it's slow because it computes. More [details](https://lilianweng.github.io/posts/2023-01-10-inference-optimization/).
  
  It also depends on the language. The model was trained primarily on the English corpus and typically generates one or two tokens for each word. But for other languages, it might not be as efficient and can generate a token for each symbol
  
  > **Comments:**
  > **username_3**: Also you might notice that sometimes it will type out a response and then replace the whole text with a standard response once some other system finds out that the GPT output is violating the their policy in some way. GPT is slow, and the policy checking algorithm is run on intermediate output from GPT.  
  
  ---
  
  