Dear Alexey and KFrank,
I have tried both methods and they both worked!
Thank you for noting that zeroing out parts of .grad sometimes will not work.
I am using MSELoss() with ADAM Optimizer.
Regards,
Eric