Thank you for the quick response.
1.The loss is quite normal, it’s 40 at first, and keep decreasing. But somehow the loss becomes Nan suddenly during training.
2. What do you mean by “method 1 may miss validation/saving”? I thought anomaly detection is to monitor gradients and find out the forward operation that created the failing backward. Since there is no backward in validation and model saving, how could it affect the anomaly detection?
3.I fixed the seed of random so I think the experiment is deterministic, which explains the repeated error at same place.
4.Method 2 implies that “ExpandBackward” returned nan values in its 0th output. This relates to the following line:
logits = F.linear(F.normalize(emb), F.normalize(weight)).
I guess the normalization computation process has some issue. How to interpret the error more specifically?