Answers.Body: <p>The thread maintains a content window.</p>
  <p>All LLMs have fixed content window size which is measured in tokens like 2048 tokens. So, When generating a Question it will consider only that many tokens as context.</p>
  <p>As we do the conversation the model stores the token from the conversation inside the content window which includes user input and model response.</p>
  <p>When the size of the conversation exceeds the content window size it will drop the past tokens.</p>
  <p>So, with the next question or conversation sentence old content window also goes as context.</p>
  <p>In terms of memory, it is stored as a session content in DB or similar.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-11-29T09:43:15.150Z
Answers.LastActivityDate: 2023-11-29T09:51:37.377Z
Answers.LastEditDate: 2023-11-29T09:51:37.377Z
Answers.UserDisplayName: Hiren Namera
Body: <p><strong>How an LLM keeps the context</strong> (what has already been entered by the user) <strong>of a chat/thread?</strong></p>
  <p>For reference, in <a href="https://chat.openai.com/" rel="nofollow noreferrer">chat.openai.com</a>, for each chat we create (or a <a href="https://platform.openai.com/docs/assistants/how-it-works/managing-threads-and-messages" rel="nofollow noreferrer">Thread</a> according to their API), the LLM remembers what we have already input to the model, when answering a new question.</p>
  <p>I did some reading on the topic and found below possible ways:</p>
  <ol>
  <li>change the weights accordingly: but this seems not-practical for LLM given their size (even changing weights of the last layer seems an over-kill)</li>
  <li>output a context vector at each inference and re-use it for the next inference: this seems more likely. but I am not sure exactly how to do it.</li>
  </ol>
  <p>It would be great if someone can help me with this.</p>
  <p>Thanks.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-12-12T10:00:24.187Z
Comments.Text: This question is answered better at https://datascience.stackexchange.com/a/122226/85897
Comments.UserDisplayName: Wenuka
Tags: <open-ai><chatgpt><large-language-models>
Title: How LLM keeps the context of a chat/thread
UserDisplayName: Wenuka
ThreadText: # How LLM keeps the context of a chat/thread
  
  (Asked by: username_0 on 2023-11-26)
  
  **How an LLM keeps the context** (what has already been entered by the user) **of a chat/thread?**
  
  For reference, in [chat.openai.com](https://chat.openai.com/), for each chat we create (or a [Thread](https://platform.openai.com/docs/assistants/how-it-works/managing-threads-and-messages) according to their API), the LLM remembers what we have already input to the model, when answering a new question.
  
  I did some reading on the topic and found below possible ways:
  
  1. change the weights accordingly: but this seems not-practical for LLM given their size (even changing weights of the last layer seems an over-kill)
  2. output a context vector at each inference and re-use it for the next inference: this seems more likely. but I am not sure exactly how to do it.
  
  It would be great if someone can help me with this.
  
  Thanks.
  
  **Tags:** open-ai, chatgpt, large-language-models
  > **Comments:**
  > **username_0**: This question is answered better at https://datascience.stackexchange.com/a/122226/85897  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-11-29. Score: 3
  
  The thread maintains a content window.
  
  All LLMs have fixed content window size which is measured in tokens like 2048 tokens. So, When generating a Question it will consider only that many tokens as context.
  
  As we do the conversation the model stores the token from the conversation inside the content window which includes user input and model response.
  
  When the size of the conversation exceeds the content window size it will drop the past tokens.
  
  So, with the next question or conversation sentence old content window also goes as context.
  
  In terms of memory, it is stored as a session content in DB or similar.
  
  
  ---
  
  