The library provides a version of this model for conditional generation and sequence classification.
Pegasus
PEGASUS: Pre-training with Extracted Gap-sentences forAbstractive Summarization, Jingqing Zhang, Yao Zhao, Mohammad Saleh and Peter J. Liu on Dec 18, 2019.
Sequence-to-sequence model with the same encoder-decoder model architecture as BART. Pegasus is pre-trained jointly on
two self-supervised objective functions: Masked Language Modeling (MLM) and a novel summarization specific pretraining
objective, called Gap Sentence Generation (GSG).
MLM: encoder input tokens are randomly replaced by a mask tokens and have to be predicted by the encoder (like in
  BERT)
GSG: whole encoder input sentences are replaced by a second mask token and fed to the decoder, but which has a
  causal mask to hide the future words like a regular auto-regressive transformer decoder.
