I provided more info in my reply to rasbt.
I believe all computations are on GPU. I’m using P100s in an Azure VM.
The tensor I apply log_softmax to has shape (64, 49, 64*49), and the normalization is along dim=2.
When tracking the [50, 90, 95] percentiles of grad norms observed in each epoch, the difference between training with 0.4.1 and 1.0.0 is stark. It is clear from the first epoch, and not at all subtle.
It is possible that something weird is going on with my conda envs or the VM environment. For now I am just going to work in 0.4.1. If I have time I will try and record a simpler and more precise set of conditions in which the observed differences arise.