@albanD thanks for your tips. I got MulBackward0 nan error in anomaly detection mode.
the forward function that is causing the nan is below loss:

@weighted_loss
def smooth_l1_loss(pred, target, beta=1.0):
assert beta > 0
assert pred.size() == target.size() and target.numel() > 0
diff = torch.abs(pred - target)
loss = torch.where(diff < beta, 0.5 * diff * diff / beta, <–this line
diff - 0.5 * beta)
return loss

so the diff tensor is nan here?
is it the first place that nan occurs?