Hi,
I’m training my own dataset with mrcnn(box head only).
At beginning, the loss value kept dropping in general until it suddenly popped out ‘nan’.
So I tried with autograd.detect_anomaly, and here is what I got:
RuntimeError                              Traceback (most recent call last)
<ipython-input-17-f414452ff0e1> in <module>()
     37               loss_train.append(smooth_tra.avg)
     38           optimizer.zero_grad()
---> 39           losses.backward()
     40           optimizer.step()
     41           #lr_scheduler.step()

1 frames
/usr/local/lib/python3.6/dist-packages/torch/tensor.py in backward(self, gradient, retain_graph, create_graph)
    196                 products. Defaults to ``False``.
    197         """
--> 198         torch.autograd.backward(self, gradient, retain_graph, create_graph)
    199 
    200     def register_hook(self, hook):

/usr/local/lib/python3.6/dist-packages/torch/autograd/__init__.py in backward(tensors, grad_tensors, retain_graph, create_graph, grad_variables)
     98     Variable._execution_engine.run_backward(
     99         tensors, grad_tensors, retain_graph, create_graph,
--> 100         allow_unreachable=True)  # allow_unreachable flag
    101 
    102 

RuntimeError: Function 'SmoothL1LossBackward' returned nan values in its 0th output.

What does this traceback mean? Any reason could cause this problem?
Thanks for any advice！