Solved by mariosasko in post #8 
              
              
                I did work with the BERT model very recently, but I’m not allowed to share the code publicly. 
This time did a more thorough pass through the code.  You are clearing the gradients (with optim.zero_grad() before applying them (optim.step()). Basically, the model is not learning anything. Put the opti…