Ok well in you original code in this line there is an extra comma
     attention_mask=attention_mask, <---- here
   )

but I don’t think that is the problem. What is the shape of your attention mask and what github are you using for the xlnet?