Solved by MarcoMeter in post #4 
              
              
                The solution for me is to not used Packed Padded Sequences. The behavior of that object is very suspicious. 
Now, I pad all the training data and then mask the loss.