@ptrblck according to the test code the weights and biases for the final linear layer and the final layer normalization (for the ltsm layer) is changing, however the weights and biases for the rest of the normalizations and layers are not updating. I printed out the gradients to check if there is a problem there and found that a large minority of the gradients are 0 with the rest mostly being in the 10^-1 to the 10^4 region. Is there a way to fix these two problems (which are probably related)?