Solved by Abhilash_Srivastava in post #2 
              
              
                Use masking over the padded sequence. 
You can either use either create your own mask or used masked_fill. 
Checkout this blogpost.