I meet with Nan loss issue in my training, so now I’m trying to use anomaly detection in autograd for debugging. I found 2 classes, torch.autograd.detect_anomaly and torch.autograd.set_detect_anomaly. But I’m getting different results with them. Method1 gives no feedback and the training can be conducted successfully, but method 2 always throw runtime error at the same number of iterations and return the nan gradient information. Why is this happening? Am I using the anomaly detection in a correct way?
Method 1:
for i in range(epoch):
    for batch in data_batches:
        with torch.autograd.detect_anomaly():
            output= model(batch)
            loss = calc_loss(output,label)
            loss.backlward()
            optimizer.step()
    validate_performance()    
    save_model()


Method 2:
torch.autograd.set_detect_anomaly(True)
for i in range(epoch):
    for batch in data_batches:
        output= model(batch)
        loss = calc_loss(output,label)
        loss.backlward()
        optimizer.step()
    validate_performance()    
    save_model()