Answers.Body: <p>Based on nothing but my own experience trying to build a similar chatbot using text-davinci-003, I think they are using the model itself to summarize the conversation, then feeding that summary back into the prompt. I get good results when doing this - though obviously the team behind ChatGPT does a better job.</p>
  <p>Here is an example using the discussion under the question above:</p>
  <blockquote>
  <p><b>You are a user named &quot;AI&quot; in an online forum. Below is a conversation
  about a question. Please add your contribution to the conversation.
  <br>Question: &quot;&quot;&quot;<br> One of the innovations with OpenAI's ChatGPT is how
  natural it is for users to interact with it.<br>
  What is the technical enabler for ChatGPT to maintain the context of
  previous questions in its answers? For example, ChatGPT understands a
  prompt of &quot;tell me more&quot; and expands on it's previous answer.<br>
  Does it use activations from previous questions? Is there a separate
  input for the context? How does it work? &quot;&quot;&quot;<br>
  Rainb: &quot;&quot;&quot; <br>
  ChatGPT says:
  &quot;ChatGPT uses a type of natural language processing called
  &quot;transformer&quot; architecture, which is a type of neural network that is
  designed to process and generate language. One of the key features of
  transformer architecture is that it allows the model to maintain a
  &quot;context&quot; of previous inputs. This is achieved through the use of
  self-attention mechanisms, which allow the model to weight different
  parts of the input differently based on how relevant they are to the
  current context. This allows the model to generate more nuanced and
  context-aware responses.&quot; &quot;&quot;&quot;<br>
  milez: &quot;&quot;&quot; <br>It does, indeed, but the
  transformer architecture alone does not explain this capability.
  Maintaining context requires something else. In fact, I think being a
  transformer is irrelevant for this purpose. &quot;&quot;&quot;<br>
  Rexcirus: &quot;&quot;&quot; <br>My best
  guess is that the whole conversation is fed as input for the next
  reply. But looking into openai labelling guidance, it looks that the
  data collection is single Q&amp;A only. &quot;&quot;&quot;<br>
  Rexcirus: &quot;&quot;&quot;<br> If my guess is
  correct, when the conversation is very long ChatGPT should start
  forgetting previous bits, due to the maximum sequence size. &quot;&quot;&quot;<br>
  yters:
  &quot;&quot;&quot; <br>The ability to maintain context is very mysterious. What is most
  mysterious to me is that if I continuously converse with ChatGPT it
  has no problem maintaining context. On the other hand, if I break in
  the middle of a conversation, and then come back to it some time
  later, it's as if ChatGPT has had a memory wipe and has a lot of
  trouble regaining the thread of the conversation. Since ChatGPT is a
  neural network where the output is generated based on the previous
  2048 input tokens, the amount of time between responses should have no
  bearing on the output generated. &quot;&quot;&quot;<br>
  milez: &quot;&quot;&quot;<br> Great observation
  @yters. Could it be a bug: the interface correctly shows the previous
  conversation, but the service fails to send this to the OpenAI APIs
  after some time.  &quot;&quot;&quot;</b><br>
  <em>AI: &quot;&quot;&quot; <br>It is possible that the OpenAI API may
  be designed to automatically discard the context after some time to
  avoid the system being overburdened with large conversations. Another
  possibility is that the system may be storing the context, but using a
  different mechanism than the transformer architecture to access it.<br />
  &quot;&quot;&quot;</em></p>
  </blockquote>
  <p>Now, I'll ask it to summarize the conversation so far:</p>
  <blockquote>
  <p><b>You are a user named &quot;AI&quot; in an online forum. Below is a conversation
  about a question. Please summarize the interaction so far.</b></p>
  <p>[I then pasted in everything from the Question down in the previous prompt/response]</p>
  <p><em>The conversation so far has focused on understanding the technical
  enablers that allow ChatGPT to maintain context in its answers. Rainb
  has suggested that this is done through transformer architecture and
  self-attention mechanisms. Rexcirus has suggested that the whole
  conversation may be fed as input for the next reply. Yters has then
  posed the interesting question of why, when the conversation is broken
  off and then resumed, ChatGPT has difficulty regaining the thread.
  Milez has suggested that this could be a bug in the interface.</em></p>
  </blockquote>
  <p>Finally, I delete most of the conversation except for the summary and last few responses and alter the prompt a little to alert it to the presence of the summary:</p>
  <blockquote>
  <p><b>You are a user named &quot;AI&quot; in an online forum. Below is a summary of
  the conversation so far about a question, and the last few responses
  to that question. Please add your contribution to the conversation.</p>
  <p>Summary: The conversation so far has focused on understanding the
  technical enablers that allow ChatGPT to maintain context in its
  answers. Rainb has suggested that this is done through transformer
  architecture and self-attention mechanisms. Rexcirus has suggested
  that the whole conversation may be fed as input for the next reply.
  Yters has then posed the interesting question of why, when the
  conversation is broken off and then resumed, ChatGPT has difficulty
  regaining the thread. Milez has suggested that this could be a bug in
  the interface.</p>
  <p>yters: &quot;&quot;&quot;<br>
  The ability to maintain context is very mysterious. What is
  most mysterious to me is that if I continuously converse with ChatGPT
  it has no problem maintaining context. On the other hand, if I break
  in the middle of a conversation, and then come back to it some time
  later, it's as if ChatGPT has had a memory wipe and has a lot of
  trouble regaining the thread of the conversation. Since ChatGPT is a
  neural network where the output is generated based on the previous
  2048 input tokens, the amount of time between responses should have no
  bearing on the output generated. &quot;&quot;&quot;<br>
  milez: &quot;&quot;&quot;<br>
  Great observation
  @yters. Could it be a bug: the interface correctly shows the previous
  conversation, but the service fails to send this to the OpenAI APIs
  after some time.  &quot;&quot;&quot;</b></p>
  <p><em>AI: That is an interesting point, @Milez. It could be that the
  interface does not accurately display the previous conversation to the
  API, or that there is a bug in the API that causes it to not be able
  to accurately generate a response with the 2048 input tokens when the
  conversation is broken off and then resumed. It would be interesting
  to investigate further.</em></p>
  </blockquote>
  <p>I've had some success with this method, though it does still lose detail. Improving the prompt to create a more detailed summary helps. It's amazing how much of a conversation is full of repetitive filler, and how concisely it can be summarized. Maybe this is some kind of lingual compression? :)</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-03-26T20:33:10.147Z
Answers.Text: Lemme run your answer through GPT-4...
Answers.UserDisplayName: YakovK
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-02T16:45:54.397Z
Answers.LastActivityDate: 2023-02-02T16:45:54.397Z
Answers.UserDisplayName: RLC
Answers.Body: <p>Based on <a href="https://web.archive.org/web/20230228124048/https://help.openai.com/en/articles/6787051-does-chatgpt-remember-what-happened-earlier-in-the-conversation" rel="nofollow noreferrer">an answer by OpenAI</a>, it appears that the illusion of understanding context is created by the model's capacity to accept very long input sequences. The OpenAI FAQ states that approximately 3000 words can be given as input. This together with the fact that <a href="https://en.wikipedia.org/wiki/GPT-3" rel="nofollow noreferrer">GPT-3 was trained to produce text that continues a given prompt</a> could explain the context feature.</p>
  <p>In practice, each prompt is probably extended with the previous outputs and prompt, as much as the input sequence length allows. So, all of the context is actually in the prompt.</p>
  <p><strong>Edit 18.2.2023</strong></p>
  <p>After spending some time with large language models and reading up the theory, I think my old answer is an understatement of ChatGPT's capabilities.</p>
  <p>It is likely that there are several engineering approaches to improve the context after the maximum content length is exceeded. These include (but are probably not limited to)</p>
  <ol>
  <li>Using language models to summarize the conversation thus far, and using that as context</li>
  <li>Using language models to search for the relevant context from the previous discussion (can be done by embedding questions and answers and doing a distance-based lookup in vector space), and feeding those as context with clever prompting like &quot;If this information improves your answer, update your answer accordingly&quot;.</li>
  </ol>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-02-18T16:22:39.300Z
Answers.Text: Agreed. Mathematically, they can retain the previous content through marginalization (https://www.quora.com/What-is-marginalization-in-probability). This can be done using Bayesian assumptions or simply a DL model. Of course there are a lot of good engineering involved. The amazing part is most folks in NLU space knows how to create ChatGPT in theory but it takes a very good engineering / science team, plus resources, to get this done correctly and efficiently.
Answers.UserDisplayName: Keeeeeenw
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-15T15:49:08.853Z
Answers.Text: Do you think ChatGPT used my interactions in the current session to enhance their answers? I think so based on my experience, but not sure.
Answers.UserDisplayName: Avv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-22T05:03:04.017Z
Answers.Text: @Avv yes it can use previous interactions in the current chat. But those will not carry over to future discussions. Not in real time anyway. Unless you opt out of the training data collection, your discussions on the chat UI may be used to train OpenAI models.
Answers.UserDisplayName: milez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-06-22T15:40:51.123Z
Answers.Text: @milez. from my experience I noticed that it's aware of all my previous interactions and use them to answer current interactions, but you might be right that it does not use them in a separate chat unless I start all over.
Answers.UserDisplayName: Avv
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-08-05T06:09:44.933Z
Answers.Text: @Keeeeeenw What is marginalization for conversation or LLM specifically?
Answers.UserDisplayName: JP Zhang
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-09-01T02:45:18.503Z
Answers.Text: What I wish we knew, and as far as I know OpenAI hasn't disclosed, is the actual strategy they use for managing context in lengthy conversations. I've noticed it can still remember "old" bits of the convo while quickly forgetting "new" things sometimes, so I think it's not simply truncating the oldest messages. Rather, they're doing some more advanced processing to conversation history before loading it into context, but unfortunately, I don't know exactly what, and we may not have any avenue to find out. Like @RLC suggested, summarization probably plays some part in that.
Answers.UserDisplayName: Ascendant
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-17T09:03:20.017Z
Answers.Text: So does that imply that if you keep chatting long enough, that ChatGPT starts to lose context as well, since you pass the threshold? That would certainly explain some brain dead responses I get sometimes.... like "I just told you that, you already forgot!?"
Answers.UserDisplayName: Benny Bottema
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-20T06:26:04.417Z
Answers.Text: Yes, I think so. Most likely the latest messages (up to the token limit) are included as context, so that is the effective working memory ChatGPT has.
Answers.UserDisplayName: milez
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-01-21T09:42:10.537Z
Answers.Text: I did some experiments and it seems the token limit is not a hard one. Or at least the context doesn't completely go away. I don't know how they determine what is kept, but after flushing ChatGPT with a lot of content (or rather having it flood me), it still remembered the fine details of the conversation's start. It's rather impressive.
Answers.UserDisplayName: Benny Bottema
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2022-12-11T14:06:07.330Z
Answers.LastActivityDate: 2023-12-02T19:29:09.287Z
Answers.LastEditDate: 2023-12-02T19:29:09.287Z
Answers.UserDisplayName: milez
Body: <p>One of the innovations with OpenAI's ChatGPT is how natural it is for users to interact with it.</p>
  <p>What is the technical enabler for ChatGPT to maintain the context of previous questions in its answers? For example, ChatGPT understands a prompt of &quot;tell me more&quot; and expands on it's previous answer.</p>
  <p>Does it use activations from previous questions? Is there a separate input for the context? How does it work?</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-08T07:30:16.147Z
Comments.Text: ChatGPT says: "ChatGPT uses a type of natural language processing called "transformer" architecture, which is a type of neural network that is designed to process and generate language. One of the key features of transformer architecture is that it allows the model to maintain a "context" of previous inputs. This is achieved through the use of self-attention mechanisms, which allow the model to weight different parts of the input differently based on how relevant they are to the current context. This allows the model to generate more nuanced and context-aware responses."
Comments.UserDisplayName: Rainb
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-09T06:13:48.433Z
Comments.Text: It does, indeed, but the transformer architecture alone does not explain this capability. Maintaining context requires something else. In fact, I _think_ being a transformer is irrelevant for this purpose.
Comments.UserDisplayName: milez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-09T10:21:24.293Z
Comments.Text: My best guess is that the whole conversation is fed as input for the next reply. But looking into openai labelling guidance, it looks that the data collection is single Q&A only.
Comments.UserDisplayName: Rexcirus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2022-12-09T10:22:32.347Z
Comments.Text: If my guess is correct, when the conversation is very long ChatGPT should start forgetting previous bits, due to the maximum sequence size.
Comments.UserDisplayName: Rexcirus
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-18T09:21:33.320Z
Comments.Text: @Rainb why didn’t you put that as an answer?  Follow up question - do you have a bit of references about these “transformer” models? Does OpenAI have any available to public?
Comments.UserDisplayName: Antonio
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-18T12:11:44.630Z
Comments.Text: @Antonio the transformer is now a standard building block of deep neural networks used widely in image, audio and language processing. Hence that answer by ChatGPT is not really informative. There are plenty of transformer models available in Huggingface for example, and most definitely all of OpenAIs language and image processing models employ the architecture.
Comments.UserDisplayName: milez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-03-14T21:33:18.027Z
Comments.Text: I imagine something like the “memorizing Transformer” architecture will be adapted in time, if not already for GPT4. https://arxiv.org/abs/2203.08913
Comments.UserDisplayName: Evan Zamir
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-09T12:54:26.247Z
Comments.Text: The ability to maintain context is very mysterious.  What is most mysterious to me is that if I continuously converse with ChatGPT it has no problem maintaining context.  On the other hand, if I break in the middle of a conversation, and then come back to it some time later, it's as if ChatGPT has had a memory wipe and has a lot of trouble regaining the thread of the conversation.  Since ChatGPT is a neural network where the output is generated based on the previous 2048 input tokens, the amount of time between responses should have no bearing on the output generated.
Comments.UserDisplayName: yters
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-01-10T11:48:33.237Z
Comments.Text: Great observation @yters. Could it be a bug: the interface correctly shows the previous conversation, but the service fails to send this to the OpenAI APIs after some time.
Comments.UserDisplayName: milez
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-01T17:11:53.480Z
Comments.Text: I don't have enough reputation to answer this question, but I've been playing around with text-davinci-003 quite a bit and can simulate the history that ChatGPT achieves with a relatively simple trick: in the prompt I ask it to summarize the conversation so far. I then pass that summary into the next prompt along with as much of the conversation as I think is appropriate to fit inside the token limit. text-davinci-003 does a very nice job of concisely summarizing the conversation. It's not as good as ChatGPT, but I do believe this is part of the magic.
Comments.UserDisplayName: RLC
Tags: <natural-language-processing><chat-bots><natural-language-understanding><chatgpt>
Title: How does ChatGPT retain the context of previous questions?
UserDisplayName: milez
ThreadText: # How does ChatGPT retain the context of previous questions?
  
  (Asked by: username_0 on 2022-12-04)
  
  One of the innovations with OpenAI's ChatGPT is how natural it is for users to interact with it.
  
  What is the technical enabler for ChatGPT to maintain the context of previous questions in its answers? For example, ChatGPT understands a prompt of "tell me more" and expands on it's previous answer.
  
  Does it use activations from previous questions? Is there a separate input for the context? How does it work?
  
  **Tags:** natural-language-processing, chat-bots, natural-language-understanding, chatgpt
  > **Comments:**
  > **username_1**: ChatGPT says: "ChatGPT uses a type of natural language processing called "transformer" architecture, which is a type of neural network that is designed to process and generate language. One of the key features of transformer architecture is that it allows the model to maintain a "context" of previous inputs. This is achieved through the use of self-attention mechanisms, which allow the model to weight different parts of the input differently based on how relevant they are to the current context. This allows the model to generate more nuanced and context-aware responses."  
  > **username_0**: It does, indeed, but the transformer architecture alone does not explain this capability. Maintaining context requires something else. In fact, I _think_ being a transformer is irrelevant for this purpose.  
  > **username_2**: My best guess is that the whole conversation is fed as input for the next reply. But looking into openai labelling guidance, it looks that the data collection is single Q&A only.  
  > **username_2**: If my guess is correct, when the conversation is very long ChatGPT should start forgetting previous bits, due to the maximum sequence size.  
  > **username_5**: The ability to maintain context is very mysterious.  What is most mysterious to me is that if I continuously converse with ChatGPT it has no problem maintaining context.  On the other hand, if I break in the middle of a conversation, and then come back to it some time later, it's as if ChatGPT has had a memory wipe and has a lot of trouble regaining the thread of the conversation.  Since ChatGPT is a neural network where the output is generated based on the previous 2048 input tokens, the amount of time between responses should have no bearing on the output generated.  
  > **username_0**: Great observation @yters. Could it be a bug: the interface correctly shows the previous conversation, but the service fails to send this to the OpenAI APIs after some time.  
  > **username_6**: I don't have enough reputation to answer this question, but I've been playing around with text-davinci-003 quite a bit and can simulate the history that ChatGPT achieves with a relatively simple trick: in the prompt I ask it to summarize the conversation so far. I then pass that summary into the next prompt along with as much of the conversation as I think is appropriate to fit inside the token limit. text-davinci-003 does a very nice job of concisely summarizing the conversation. It's not as good as ChatGPT, but I do believe this is part of the magic.  
  > **username_3**: @Rainb why didn’t you put that as an answer?  Follow up question - do you have a bit of references about these “transformer” models? Does OpenAI have any available to public?  
  > **username_0**: @Antonio the transformer is now a standard building block of deep neural networks used widely in image, audio and language processing. Hence that answer by ChatGPT is not really informative. There are plenty of transformer models available in Huggingface for example, and most definitely all of OpenAIs language and image processing models employ the architecture.  
  > **username_4**: I imagine something like the “memorizing Transformer” architecture will be adapted in time, if not already for GPT4. https://arxiv.org/abs/2203.08913  
  
  ---
  
  ## 2 answers
  
  ### Answer by username_0 on 2022-12-11. Score: 32 | Accepted answer
  
  Based on [an answer by OpenAI](https://web.archive.org/web/20230228124048/https://help.openai.com/en/articles/6787051-does-chatgpt-remember-what-happened-earlier-in-the-conversation), it appears that the illusion of understanding context is created by the model's capacity to accept very long input sequences. The OpenAI FAQ states that approximately 3000 words can be given as input. This together with the fact that [GPT-3 was trained to produce text that continues a given prompt](https://en.wikipedia.org/wiki/GPT-3) could explain the context feature.
  
  In practice, each prompt is probably extended with the previous outputs and prompt, as much as the input sequence length allows. So, all of the context is actually in the prompt.
  
  **Edit 18.2.2023**
  
  After spending some time with large language models and reading up the theory, I think my old answer is an understatement of ChatGPT's capabilities.
  
  It is likely that there are several engineering approaches to improve the context after the maximum content length is exceeded. These include (but are probably not limited to)
  
  1. Using language models to summarize the conversation thus far, and using that as context
  2. Using language models to search for the relevant context from the previous discussion (can be done by embedding questions and answers and doing a distance-based lookup in vector space), and feeding those as context with clever prompting like "If this information improves your answer, update your answer accordingly".
  
  > **Comments:**
  > **username_12**: So does that imply that if you keep chatting long enough, that ChatGPT starts to lose context as well, since you pass the threshold? That would certainly explain some brain dead responses I get sometimes.... like "I just told you that, you already forgot!?"  
  > **username_0**: Yes, I think so. Most likely the latest messages (up to the token limit) are included as context, so that is the effective working memory ChatGPT has.  
  > **username_12**: I did some experiments and it seems the token limit is not a hard one. Or at least the context doesn't completely go away. I don't know how they determine what is kept, but after flushing ChatGPT with a lot of content (or rather having it flood me), it still remembered the fine details of the conversation's start. It's rather impressive.  
  > **username_8**: Agreed. Mathematically, they can retain the previous content through marginalization (https://www.quora.com/What-is-marginalization-in-probability). This can be done using Bayesian assumptions or simply a DL model. Of course there are a lot of good engineering involved. The amazing part is most folks in NLU space knows how to create ChatGPT in theory but it takes a very good engineering / science team, plus resources, to get this done correctly and efficiently.  
  > **username_9**: Do you think ChatGPT used my interactions in the current session to enhance their answers? I think so based on my experience, but not sure.  
  > **username_0**: @Avv yes it can use previous interactions in the current chat. But those will not carry over to future discussions. Not in real time anyway. Unless you opt out of the training data collection, your discussions on the chat UI may be used to train OpenAI models.  
  > **username_9**: @milez. from my experience I noticed that it's aware of all my previous interactions and use them to answer current interactions, but you might be right that it does not use them in a separate chat unless I start all over.  
  > **username_10**: @Keeeeeenw What is marginalization for conversation or LLM specifically?  
  > **username_11**: What I wish we knew, and as far as I know OpenAI hasn't disclosed, is the actual strategy they use for managing context in lengthy conversations. I've noticed it can still remember "old" bits of the convo while quickly forgetting "new" things sometimes, so I think it's not simply truncating the oldest messages. Rather, they're doing some more advanced processing to conversation history before loading it into context, but unfortunately, I don't know exactly what, and we may not have any avenue to find out. Like @RLC suggested, summarization probably plays some part in that.  
  
  ---
  
  ### Answer by username_6 on 2023-02-02. Score: 6
  
  Based on nothing but my own experience trying to build a similar chatbot using text-davinci-003, I think they are using the model itself to summarize the conversation, then feeding that summary back into the prompt. I get good results when doing this - though obviously the team behind ChatGPT does a better job.
  
  Here is an example using the discussion under the question above:
  
  > **You are a user named "AI" in an online forum. Below is a conversation
  > about a question. Please add your contribution to the conversation.
  >   
  > Question: """  
  >  One of the innovations with OpenAI's ChatGPT is how
  > natural it is for users to interact with it.  
  > What is the technical enabler for ChatGPT to maintain the context of
  > previous questions in its answers? For example, ChatGPT understands a
  > prompt of "tell me more" and expands on it's previous answer.  
  > Does it use activations from previous questions? Is there a separate
  > input for the context? How does it work? """  
  > Rainb: """   
  > ChatGPT says:
  > "ChatGPT uses a type of natural language processing called
  > "transformer" architecture, which is a type of neural network that is
  > designed to process and generate language. One of the key features of
  > transformer architecture is that it allows the model to maintain a
  > "context" of previous inputs. This is achieved through the use of
  > self-attention mechanisms, which allow the model to weight different
  > parts of the input differently based on how relevant they are to the
  > current context. This allows the model to generate more nuanced and
  > context-aware responses." """  
  > milez: """   
  > It does, indeed, but the
  > transformer architecture alone does not explain this capability.
  > Maintaining context requires something else. In fact, I think being a
  > transformer is irrelevant for this purpose. """  
  > Rexcirus: """   
  > My best
  > guess is that the whole conversation is fed as input for the next
  > reply. But looking into openai labelling guidance, it looks that the
  > data collection is single Q&A only. """  
  > Rexcirus: """  
  >  If my guess is
  > correct, when the conversation is very long ChatGPT should start
  > forgetting previous bits, due to the maximum sequence size. """  
  > yters:
  > """   
  > The ability to maintain context is very mysterious. What is most
  > mysterious to me is that if I continuously converse with ChatGPT it
  > has no problem maintaining context. On the other hand, if I break in
  > the middle of a conversation, and then come back to it some time
  > later, it's as if ChatGPT has had a memory wipe and has a lot of
  > trouble regaining the thread of the conversation. Since ChatGPT is a
  > neural network where the output is generated based on the previous
  > 2048 input tokens, the amount of time between responses should have no
  > bearing on the output generated. """  
  > milez: """  
  >  Great observation
  > @yters. Could it be a bug: the interface correctly shows the previous
  > conversation, but the service fails to send this to the OpenAI APIs
  > after some time. """**  
  > *AI: """   
  > It is possible that the OpenAI API may
  > be designed to automatically discard the context after some time to
  > avoid the system being overburdened with large conversations. Another
  > possibility is that the system may be storing the context, but using a
  > different mechanism than the transformer architecture to access it.*
  
  Now, I'll ask it to summarize the conversation so far:
  
  > **You are a user named "AI" in an online forum. Below is a conversation
  > about a question. Please summarize the interaction so far.**
  >
  > [I then pasted in everything from the Question down in the previous prompt/response]
  >
  > *The conversation so far has focused on understanding the technical
  > enablers that allow ChatGPT to maintain context in its answers. Rainb
  > has suggested that this is done through transformer architecture and
  > self-attention mechanisms. Rexcirus has suggested that the whole
  > conversation may be fed as input for the next reply. Yters has then
  > posed the interesting question of why, when the conversation is broken
  > off and then resumed, ChatGPT has difficulty regaining the thread.
  > Milez has suggested that this could be a bug in the interface.*
  
  Finally, I delete most of the conversation except for the summary and last few responses and alter the prompt a little to alert it to the presence of the summary:
  
  > **You are a user named "AI" in an online forum. Below is a summary of
  > the conversation so far about a question, and the last few responses
  > to that question. Please add your contribution to the conversation.**
  >
  > Summary: The conversation so far has focused on understanding the
  > technical enablers that allow ChatGPT to maintain context in its
  > answers. Rainb has suggested that this is done through transformer
  > architecture and self-attention mechanisms. Rexcirus has suggested
  > that the whole conversation may be fed as input for the next reply.
  > Yters has then posed the interesting question of why, when the
  > conversation is broken off and then resumed, ChatGPT has difficulty
  > regaining the thread. Milez has suggested that this could be a bug in
  > the interface.
  >
  > yters: """  
  > The ability to maintain context is very mysterious. What is
  > most mysterious to me is that if I continuously converse with ChatGPT
  > it has no problem maintaining context. On the other hand, if I break
  > in the middle of a conversation, and then come back to it some time
  > later, it's as if ChatGPT has had a memory wipe and has a lot of
  > trouble regaining the thread of the conversation. Since ChatGPT is a
  > neural network where the output is generated based on the previous
  > 2048 input tokens, the amount of time between responses should have no
  > bearing on the output generated. """  
  > milez: """  
  > Great observation
  > @yters. Could it be a bug: the interface correctly shows the previous
  > conversation, but the service fails to send this to the OpenAI APIs
  > after some time. """
  >
  > *AI: That is an interesting point, @Milez. It could be that the
  > interface does not accurately display the previous conversation to the
  > API, or that there is a bug in the API that causes it to not be able
  > to accurately generate a response with the 2048 input tokens when the
  > conversation is broken off and then resumed. It would be interesting
  > to investigate further.*
  
  I've had some success with this method, though it does still lose detail. Improving the prompt to create a more detailed summary helps. It's amazing how much of a conversation is full of repetitive filler, and how concisely it can be summarized. Maybe this is some kind of lingual compression? :)
  
  > **Comments:**
  > **username_7**: Lemme run your answer through GPT-4...  
  
  ---
  
  