10X the amount of data.
Tips:
GPT-2 is a model with absolute position embeddings so it's usually advised to pad the inputs on the right rather than
  the left.
GPT-2 was trained with a causal language modeling (CLM) objective and is therefore powerful at predicting the next
  token in a sequence. Leveraging this feature allows GPT-2 to generate syntactically coherent text as it can be
  observed in the run_generation.py example script.
The model can take the past_key_values (for PyTorch) or past (for TF) as input, which is the previously computed
  key/value attention pairs. Using this (past_key_values or past) value prevents the model from re-computing
  pre-computed values in the context of text generation. For PyTorch, see past_key_values argument of the
  [GPT2Model.forward] method, or for TF the past argument of the
