why xlnet requires a long prompt for short inputs while Bert does not ? 
 hey guys,

Q1)
can someone give some more insight what @thomwolf explaining about?
'''
#846
The main reason you get bad performance is that XLNet is not good on short inputs (comes from the way it is pretrained, always having a long memory and only guessing a few words in the sequence).

The run_generation example here will show you how to get better performances by adding a random text as initiator.
Aman Rusia also wrote a blog post about that here. We are using his solution in the run_generation example.
'''
I can't understand the difference the way both Bert and XLnetLM works for LMhead task.
Aren't both model having disadvantages if they have short sentence?

It seems he said XLnet has huge disadvantage on short input sentence
while Bert does not(or has less disadvantage). Any detail explanation could be useful !

Q2)
Also, I can't get the point of adding extra padding or adding random padding things to improve XLnetLMHead model. Any snippet or explanation could be appreciated too...(saw the link but could not fully understood). I experimented by just adding extra strings of line:'I believe my sister is because she is a blonde ' + ' ' and it gives much better result than not having at the end....

Q3)
#846 (comment)
Lastly, why do we have better result when we don't use perm_mask ? above link response shows that
not having perm_mask option does give at least better result...But isn't perm_mask supposed to help to get better prediction and what author of paper used for SOTA ?

isn't perm_mask allow model to not seeing the next tokens in the given input while can see the previous tokens? According to the paper and the original code, I could see that if permute order is 3->4->1->2, mask=1,3, then model cannot see masked<1> when it tried to predict masked<3> but the reverse is possible.

Many thanks in advance !
This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.
