Body: <p>I'm asking specifically about ChatGPT4, but the question could apply to either that or 3.5.</p>
  <p>When you use the ChatGPT API, it's of course up to you to manage conversation history and include that in successive API calls within available context length in whatever manner you choose.</p>
  <p>In the case of the web interface, they've obviously implemented some system to manage conversation history in context. It clearly doesn't &quot;remember&quot; the entire thing once the conversation gets very long, because it doesn't have infinite context length. So, what strategy does it use to send conversation history to the model once it's exceeded its context length? Does it truncate all content prior to the max context length? Does it summarize earlier parts of conversations to more efficiently fit them within the context? Does it do some dynamic strategy combining many inputs?</p>
  <p>Or is this just another case where we just don't know, and OpenAI is being tight-lipped about what it's actually doing?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-01T15:08:47.163Z
Comments.Text: well, either it uses some "last N tokens", or it might just ask itself to also summarize the history, and the next time to summarize the summarization+the new question, and so on
Comments.UserDisplayName: Alberto
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-03T07:09:09.070Z
Comments.Text: @Alberto is there a source for this? Transformers can be finetuned to handle longer contexts (e.g., see https://lmsys.org/blog/2023-06-29-longchat/), so compressing the input tokens doesn't seem absolutely necessary to me.
Comments.UserDisplayName: Alexander Wan
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-09-04T11:00:47.993Z
Comments.Text: @AlexanderWan it's not a matter of architecture, it's a matter of efficiency (price-wise)... no, "_it might just ask itself_" it was just a personal thought, but you can see something like this implemented in RMT
Comments.UserDisplayName: Alberto
Tags: <chatgpt>
Title: What strategy does ChatGPT use to manage its context in very lengthy conversations?
UserDisplayName: Ascendant
ThreadText: # What strategy does ChatGPT use to manage its context in very lengthy conversations?
  
  (Asked by: username_0 on 2023-08-31)
  
  I'm asking specifically about ChatGPT4, but the question could apply to either that or 3.5.
  
  When you use the ChatGPT API, it's of course up to you to manage conversation history and include that in successive API calls within available context length in whatever manner you choose.
  
  In the case of the web interface, they've obviously implemented some system to manage conversation history in context. It clearly doesn't "remember" the entire thing once the conversation gets very long, because it doesn't have infinite context length. So, what strategy does it use to send conversation history to the model once it's exceeded its context length? Does it truncate all content prior to the max context length? Does it summarize earlier parts of conversations to more efficiently fit them within the context? Does it do some dynamic strategy combining many inputs?
  
  Or is this just another case where we just don't know, and OpenAI is being tight-lipped about what it's actually doing?
  
  **Tags:** chatgpt
  
  > **Comments:**
  > **username_1**: well, either it uses some "last N tokens", or it might just ask itself to also summarize the history, and the next time to summarize the summarization+the new question, and so on  
  > **username_2**: @username_1 is there a source for this? Transformers can be finetuned to handle longer contexts (e.g., see https://lmsys.org/blog/2023-06-29-longchat/), so compressing the input tokens doesn't seem absolutely necessary to me.  
  > **username_1**: @username_2 it's not a matter of architecture, it's a matter of efficiency (price-wise)... no, "_it might just ask itself_" it was just a personal thought, but you can see something like this implemented in RMT  
  
  ---
  
  ## 0 answers
  
  