I checked out the sum of the gradients of encoder.embedding.weight.grad during training for model that uses mask padding and for second one that uses criterion with ignore_index.
For model with padding mask the lower the loss was, the lower the gradients, but they decreased very slowly.
Iteration: 1; tensor(-0.1997)
Iteration: 35; tensor(-0.0057)
In contrast to preceding example, model with criterion begins with much lower gradients, and during the training gradients diminish much faster.
Iteration: 1; tensor(-5.2681e-05)
Iteration: 35; tensor(2.0964e-10)
The very small gradients cause that model doesn’t converge, but how to find a remedy to that problem? Could anybody please explain gradients behavior? Any other advice would be appreciated as well.