What is your question?
My task is to generate keywords from sentences.
I pretrain a text-generation model. I mask the sentences’ tokens and predict the whole sentences’ tokens.
Pretraining batch_size = 8 and step = 1000000
I haven’t observed improvement from pretraining. BLEU score is 10.5 for not pretraining, BLEU score is 9.5 for pretraining.
Code
I take the python code from

  
      github.com
  
  
    google-research/pegasus/blob/master/pegasus/models/transformer.py#L38 1

from pegasus.layers import attention
from pegasus.layers import decoding
from pegasus.layers import embedding
from pegasus.layers import timing
from pegasus.layers import transformer_block
from pegasus.models import base
import tensorflow as tf
from tensorflow.contrib import layers as contrib_layers


class TransformerEncoderDecoderModel(base.BaseModel):
  """Transformer encoder+decoder.

  Notations:
    B: batch_size, I: max_input_len, T: max_target/decode_len, D: hidden_size
    V: vocab_size
  """

  def __init__(self, vocab_size, hidden_size, filter_size, num_heads,
               num_encoder_layers, num_decoder_layers, label_smoothing,
               dropout):



  
  
    
    
  
  


hidden_size = 512
num_encoder_layers = 3
num_decoder_layers = 3
Discussion
The task is to generate keyword from sentences.
The keyword may not appear in the sentences.
So input masked sentences to predict whole sentences, it is not benefit the keywords generation task.
Input masked sentences to predict whole sentences, it do not have relation to the keywords generation task.
Am I right? Is it the reason that pretraining do not improve the BLEU score?
Thank you very much.