Body: <p>Given the samples <span class="math-container">$\vec{x_i} \in \mathbb{R}^d, i \in [1,..,l]$</span> where <span class="math-container">$l$</span> is the number of training samples, <span class="math-container">$d$</span> is the number of input features, the related target values <span class="math-container">$y_i \in \mathbb{R}$</span>, and the <span class="math-container">$l \times l$</span> matrix defined below:</p>
  <p><span class="math-container">$$ S_{i,j} =  e^{-\gamma_S ||\vec{x}_i - \vec{x}_j||^2} = e^{-\gamma_S (\vec{x}_i' \vec{x}_i -2 \vec{x}_i' \vec{x}_j + \vec{x}_j'\vec{x}_j)}
  $$</span></p>
  <p>where <span class="math-container">$i \in [1,..,l], j \in [1,..,l]$</span>, and <span class="math-container">$\gamma_S$</span> is another hyper-parameter, we would like to use with PyTorch the following custom loss for a regression task:</p>
  <p><span class="math-container">$$\sum_{i=1}^l \sum_{j=1}^l \sqrt{|p_i-y_i|}\sqrt{|p_j-y_j|} S_{i,j}$$</span>
  where <span class="math-container">$p_i$</span> is the <span class="math-container">$i$</span>-th estimation.</p>
  <p>Our loss is implemented with this code:</p>
  <pre><code>def ourLoss(out, lab):
      global stra, sc
      abserr = torch.abs(out - lab).flatten().float()
      serr = torch.sqrt(abserr)
      bm = stra[sc : sc + out.shape[0], sc : sc + out.shape[0]].float()
      loss = torch.dot(serr, torch.matmul(bm, serr))
      return loss
  </code></pre>
  <p>where 'stra' is <span class="math-container">$S$</span>, sc is a counter used for batch evaluations, and then the Adam optimizer returns a nan loss value...</p>
  
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-25T16:26:44.493Z
Comments.Text: Can you try to lower your learning rate (substantially). It might be that it just explodes becuase of exploding gradients.
Comments.UserDisplayName: Robin van Hoorn
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-02-27T12:03:27.030Z
Comments.Text: We tried, other nan's compare.
  We think the problem is related with the fact that as some serr goes to $0$, its derivative goes to infinity.
  We tried to clip the gradient but it did not work.
Comments.UserDisplayName: Filippo Portera
Comments.ContentLicense: CC BY-SA 4.0
Comments.CreationDate: 2023-05-18T08:07:22.463Z
Comments.Text: We tried also to add a small value ( 1E-7 ) to serr, but we still have nan loss values.
Comments.UserDisplayName: Filippo Portera
Tags: <pytorch><loss><adam>
Title: What's wrong with our loss and PyTorch?
UserDisplayName: Filippo Portera
ThreadText: # What's wrong with our loss and PyTorch?
  
  (Asked by: username_0 on 2023-02-25)
  
  Given the samples $\vec{x_i} \in \mathbb{R}^d, i \in [1,..,l]$ where $l$ is the number of training samples, $d$ is the number of input features, the related target values $y_i \in \mathbb{R}$, and the $l \times l$ matrix defined below:
  $$ S_{i,j} =  e^{-\gamma_S ||\vec{x}_i - \vec{x}_j||^2} = e^{-\gamma_S (\vec{x}_i' \vec{x}_i -2 \vec{x}_i' \vec{x}_j + \vec{x}_j'\vec{x}_j)}
  $$
  where $i \in [1,..,l], j \in [1,..,l]$, and $\gamma_S$ is another hyper-parameter, we would like to use with PyTorch the following custom loss for a regression task:
  $$\sum_{i=1}^l \sum_{j=1}^l \sqrt{|p_i-y_i|}\sqrt{|p_j-y_j|} S_{i,j}$$
  where $p_i$ is the $i$-th estimation.
  Our loss is implemented with this code:
  def ourLoss(out, lab):
      global stra, sc
      abserr = torch.abs(out - lab).flatten().float()
      serr = torch.sqrt(abserr)
      bm = stra[sc : sc + out.shape[0], sc : sc + out.shape[0]].float()
      loss = torch.dot(serr, torch.matmul(bm, serr))
      return loss
  
  where 'stra' is $S$, sc is a counter used for batch evaluations, and then the Adam optimizer returns a nan loss value...
  
  **Tags:** pytorch, loss, adam
  > **Comments:**
  > **username_1**: Can you try to lower your learning rate (substantially). It might be that it just explodes becuase of exploding gradients.  
  > **username_0**: We tried, other nan's compare. We think the problem is related with the fact that as some serr goes to $0$, its derivative goes to infinity. We tried to clip the gradient but it did not work.  
  > **username_0**: We tried also to add a small value ( 1E-7 ) to serr, but we still have nan loss values.  
  
  ---
  
  ## 0 answers
  
  