title: RuntimeError 'DivBackward0' nan values in its 0th output, but works when tensors loaded from disk?
body: <p>I'm trying to implement a particular loss function in PyTorch called <a href="https://en.wikipedia.org/wiki/Symmetric_mean_absolute_percentage_error" rel="nofollow noreferrer">SMAPE</a> (commonly used in time series forecasting). I have two variables, <code>model_outputs</code> and <code>target_outputs</code>, and the formula for computing the element-wise SMAPE is straight-forward:</p>
  
  <pre><code>numerator = torch.abs(model_outputs - target_outputs)
  denominator = torch.abs(model_outputs) + torch.abs(target_outputs)
  elementwise_smape = torch.div(numerator, denominator)
  nan_mask = torch.isnan(elementwise_smape)
  loss = elementwise_smape[~nan_mask].mean()
  assert ~torch.isnan(loss)  # loss = 0.023207199
  loss.backward()
  </code></pre>
  
  <p>But when I ran my code, I received the following error: <code>RuntimeError: Function 'DivBackward0' returned nan values in its 0th output.</code></p>
  
  <p>I used <code>autograd.detect_anomaly()</code> to track down the error as stemming from <code>elementwise_smape = torch.div(numerator, denominator)</code>. I tried the suggested solutions I could find online (checking for NaN values, adding hooks, printing gradients) and nothing helped. So I decided to write the tensors to disk and write an independent script:</p>
  
  <pre><code>import torch
  
  
  model_outputs = torch.load('model_outputs.pt')
  target_outputs = torch.load('target_outputs.pt')
  print(model_outputs)
  print(target_outputs)
  
  model_outputs.requires_grad = True
  target_outputs.requires_grad = False
  
  numerator = torch.abs(model_outputs - target_outputs)
  denominator = torch.abs(model_outputs) + torch.abs(target_outputs)
  elementwise_smape = torch.div(numerator, denominator)
  nan_mask = torch.isnan(elementwise_smape)
  loss = elementwise_smape[~nan_mask].mean()
  assert ~torch.isnan(loss)
  print(loss.detach().numpy())
  loss.backward()
  print('Success!')
  </code></pre>
  
  <p>This worked completely fine! To test that the problem doesn't originate from any of the variables <code>elementwise_smape</code> depends on (<code>model_outputs</code>, <code>numerator</code>, <code>denominator</code>), I tried replacing the loss function with each of the dependent variables, but all worked!</p>
  
  <pre><code>nan_mask = torch.isnan(elementwise_smape)
  loss = model_outputs[~nan_mask].mean()
  loss.backward()  # succeeds
  </code></pre>
  
  <pre><code>nan_mask = torch.isnan(elementwise_smape)
  loss = numerator[~nan_mask].mean()
  loss.backward()  # succeeds
  </code></pre>
  
  <pre><code>nan_mask = torch.isnan(elementwise_smape)
  loss = denominator[~nan_mask].mean()
  loss.backward()  # succeeds
  </code></pre>
  
  <p>So what the hell is going on with the element-wise division that causes a problem when <code>model_outputs</code> are generated by the model, but not when <code>model_outputs</code> are loaded from disk?</p>
  
  <p>Update #1: Adding complete error:</p>
  
  <pre><code>Traceback of forward call that caused the error:
    File "/home/rylan/tsml/main.py", line 116, in &lt;module&gt;
      main(experiment_number)
    File "/home/rylan/tsml/main.py", line 23, in main
      loss_per_model_per_step=loss_per_model_per_step)
    File "/home/rylan/tsml/main.py", line 47, in run
      sequence_lengths)
    File "/home/rylan/tsml/main.py", line 80, in step
      sequence_lengths=sequence_lengths)
    File "/home/rylan/tsml/models.py", line 153, in loss
      model_output_log_probs=None)
    File "/home/rylan/tsml/losses.py", line 96, in calculate_mean_smape
      target_outputs=target_outputs)
    File "/home/rylan/tsml/losses.py", line 117, in calculate_elementwise_smape
      elementwise_smape = torch.div(numerator, denominator)
  
  
  Traceback (most recent call last):
    File "/home/rylan/tsml/main.py", line 116, in &lt;module&gt;
      main(experiment_number)
    File "/home/rylan/tsml/main.py", line 23, in main
      loss_per_model_per_step=loss_per_model_per_step)
    File "/home/rylan/tsml/main.py", line 47, in run
      sequence_lengths)
    File "/home/rylan/tsml/main.py", line 89, in step
      total_loss.backward()
    File "/home/rylan/.local/lib/python3.6/site-packages/torch/tensor.py", line 107, in backward
      torch.autograd.backward(self, gradient, retain_graph, create_graph)
    File "/home/rylan/.local/lib/python3.6/site-packages/torch/autograd/__init__.py", line 93, in backward
      allow_unreachable=True)  # allow_unreachable flag
  RuntimeError: Function 'DivBackward0' returned nan values in its 0th output.
  
  Process finished with exit code 1
  </code></pre>
  
  <p>Update #2: Finomnis suggested writing the tensors to disk, reading them back and comparing them to the original:</p>
  
  <pre><code>torch.save(model_outputs, 'model_outputs.pt')
  torch.save(target_outputs, 'target_outputs.pt')
  model_outputs_2 = torch.load('model_outputs.pt')
  target_outputs_2 = torch.load('target_outputs.pt')
  (model_outputs == model_outputs_2).all()  # tensor(1, dtype=torch.uint8)
  (target_outputs == target_outputs_2).all()  # tensor(0, dtype=torch.uint8)
  </code></pre>
  
  <p>I suspected that <code>(target_outputs == target_outputs_2).all()</code> evaluates to <code>False</code> due to <code>NaNs</code>. This is correct:</p>
  
  <pre><code>(torch.isnan(target_outputs) == torch.isnan(target_outputs_2)).all()  # tensor(1, dtype=torch.uint8)
  (target_outputs[~torch.isnan(target_outputs)] == target_outputs_2[~torch.isnan(target_outputs_2)]).all()  # tensor(1, dtype=torch.uint8)
  </code></pre>
  
  <p>Update #3: I tried the next suggestion of detaching <code>model_outputs</code> to ensure that the problem isn't originating from elsewhere:</p>
  
  <pre><code>def calculate_elementwise_smape(model_outputs,
                                  target_outputs):
      model_outputs = model_outputs.detach()
      model_outputs.requires_grad = True
  
      target_outputs = target_outputs.detach()
  
      numerator = torch.abs(model_outputs - target_outputs)
      denominator = torch.abs(model_outputs) + torch.abs(target_outputs)
      elementwise_smape = torch.div(numerator, denominator)
      return elementwise_smape
  </code></pre>
  
  <p>but this produced the same error <code>RuntimeError: Function 'DivBackward0' returned nan values in its 0th output.</code> at the same location <code>elementwise_smape = torch.div(numerator, denominator)</code></p>
  
  <p>To reiterate what I said earlier, this problem does not occur if I load the tensors from disk and make the exact same sequence of operations.</p>