Answers.Body: <p>I will base my answer on GPT-2, as most LLMs are not too different in their architecture.</p>
  <p>The output of the transformer in an LLM is not text or tokens; it is a categorical distribution over the space of next possible tokens.</p>
  <p>To calculate loss of the transformer, you (commonly) take negative log likelihood. If the training data says the next token was <span class="math-container">$i$</span> and the transformer's output was a vector <span class="math-container">$v$</span>, then we have this loss.
  <span class="math-container">$$L = -\log(v_i)$$</span>
  Backpropagating this encourages the transformer to assign higher confidence to the correct token, according to data.</p>
  <p>Reagarding the 'thumbs up' feature of ChatGPT, this is probably an auxiliary loss. OpenAI is known to use Reinforcement Learning to tune ChatGPT, in addition to the negative log likelihood which would've been used during pre-training. How exactly this is calculated, we don't know; ChatGPT is closed source.</p>
  
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-13T04:08:46.990Z
Answers.Text: "If the training data says the next token was " how would this apply to text it outputs?  This makes sense for training with user generated text.
Answers.UserDisplayName: Seph Reed
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-13T04:30:11.413Z
Answers.Text: @SephReed Most LLMs are trained on human data, so full text generated by the network is not taken into account, only the token probabilities. In RLHF, the full texts are used for fine-tuning. A human is able to assign scores to a token sequence that the LLM generates. 'credit' is then assigned to each of the tokens, and the transformer adapts its token probabilities to maximise score. You could have a look at InstructGPT which is an example of this.
Answers.UserDisplayName: Venna Banana
Answers.ContentLicense: CC BY-SA 4.0
Answers.CreationDate: 2023-07-12T16:42:29.800Z
Answers.LastActivityDate: 2023-07-12T16:42:29.800Z
Answers.UserDisplayName: Venna Banana
Body: <p>When an LLM creates an output, it seemingly has no way to check if its output was valid.  Therefore it wouldn't be able to back-propagate any changes to the weights is used to create that output.</p>
  <p>Right now, I suspect that all weight modification is done by training on input data, as that can (generally) be assumed to be human and valid.</p>
  <p>But perhaps it does have some way of checking if its output was good or not, and being modified based off of it.  For instance, there is the thumb up and down button on ChatGPT which could be used for that.</p>
  <p>Do LLM's modify their neural weights based off of their own answers?  If so, how?</p>
  
Tags: <backpropagation><chatgpt>
Title: Does the output of LLM's affect their neural weights?
UserDisplayName: Seph Reed
ThreadText: # Does the output of LLM's affect their neural weights?
  
  (Asked by: username_0 on 2023-07-12)
  
  When an LLM creates an output, it seemingly has no way to check if its output was valid. Therefore it wouldn't be able to back-propagate any changes to the weights is used to create that output.
  
  Right now, I suspect that all weight modification is done by training on input data, as that can (generally) be assumed to be human and valid.
  
  But perhaps it does have some way of checking if its output was good or not, and being modified based off of it. For instance, there is the thumb up and down button on ChatGPT which could be used for that.
  
  Do LLM's modify their neural weights based off of their own answers? If so, how?
  
  **Tags:** backpropagation, chatgpt
  
  
  ---
  
  ## 1 answers
  
  ### Answer by username_1 on 2023-07-12. Score: 1 | Accepted answer
  
  I will base my answer on GPT-2, as most LLMs are not too different in their architecture.
  
  The output of the transformer in an LLM is not text or tokens; it is a categorical distribution over the space of next possible tokens.
  
  To calculate loss of the transformer, you (commonly) take negative log likelihood. If the training data says the next token was $i$ and the transformer's output was a vector $v$, then we have this loss.
  $$L = -\log(v\_i)$$
  Backpropagating this encourages the transformer to assign higher confidence to the correct token, according to data.
  
  Reagarding the 'thumbs up' feature of ChatGPT, this is probably an auxiliary loss. OpenAI is known to use Reinforcement Learning to tune ChatGPT, in addition to the negative log likelihood which would've been used during pre-training. How exactly this is calculated, we don't know; ChatGPT is closed source.
  
  > **Comments:**
  > **username_0**: "If the training data says the next token was " how would this apply to text it outputs?  This makes sense for training with user generated text.  
  > **username_1**: @username_0 Most LLMs are trained on human data, so full text generated by the network is not taken into account, only the token probabilities. In RLHF, the full texts are used for fine-tuning. A human is able to assign scores to a token sequence that the LLM generates. 'credit' is then assigned to each of the tokens, and the transformer adapts its token probabilities to maximise score. You could have a look at InstructGPT which is an example of this.  
  
  ---
  
  