Body: <p>I was looking into the loss function in t5x <a href="https://github.com/google-research/t5x/blob/efce74c982db9937117ae37a873d2e4c67ab4d56/t5x/losses.py#L57" rel="nofollow noreferrer">here</a> and see there is a z-loss added to the typical log loss definition.</p>
  <p>The only paper I could surface on this was <a href="https://arxiv.org/abs/1604.08859" rel="nofollow noreferrer">https://arxiv.org/abs/1604.08859</a>, but I don't see the correlation. In the code, it is logsumexp of the logits squared.</p>
  <p>But in the paper it is calculating mean/variance for the z-loss you can see the author's implementation here.</p>
  <p>I am asking for help either connecting the two or looking for resources on the current implementation if they are in fact unrelated.</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-10T09:34:59.643Z
Comments.Text: Could you please reformulate the title as a question? Like "How is this z-loss implementation in t5x related to this paper's loss X?"
Comments.UserDisplayName: nbro
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2024-07-10T17:04:36.847Z
Comments.Text: Sure, I changed it.
Comments.UserDisplayName: Jacob B
Tags: <gradient-descent><gpt><loss>
Title: How is this z-loss implementation in t5x related to this paper's loss X?
UserDisplayName: Jacob B
ThreadText: # How is this z-loss implementation in t5x related to this paper's loss X?
  
  (Asked by: username_0 on 2024-07-10)
  
  I was looking into the loss function in t5x [here](https://github.com/google-research/t5x/blob/efce74c982db9937117ae37a873d2e4c67ab4d56/t5x/losses.py#L57) and see there is a z-loss added to the typical log loss definition.
  
  The only paper I could surface on this was <https://arxiv.org/abs/1604.08859>, but I don't see the correlation. In the code, it is logsumexp of the logits squared.
  
  But in the paper it is calculating mean/variance for the z-loss you can see the author's implementation here.
  
  I am asking for help either connecting the two or looking for resources on the current implementation if they are in fact unrelated.
  
  **Tags:** gradient-descent, gpt, loss
  
  > **Comments:**
  > **username_1**: Could you please reformulate the title as a question? Like "How is this z-loss implementation in t5x related to this paper's loss X?"  
  > **username_0**: Sure, I changed it.  
  
  ---
  
  ## 0 answers
  
  