Hi,
Yes I am sure about this. My hook always prints dropout as the first layer where Nan appears in the output. This has happening more than once.
It’s hard to write print statements as I am using huggungface Bert. That’s why used the hooks.
This Nan issue is very frequent at 217th epoch which reduces amp loss scaler to 0 resulting in division by zero error.
We are not sure how to solve this. What would you suggest for now ?