guotong1988:

So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.
Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.
Am I right? Is it the reason that pretraining do not improve the BLEU score?
Thank you very much.


Definitely possible, there could also be a bug in your code. I  don’t have enough familiarity with your task to know what results to expect.