I misunderstood the use case a little bit 
For back-translation, generating artificial examples with sampling works pretty well in my experience, especially if you’re iterating in both directions (summarization model and expansion model):

  
      
      arXiv.org
  
  
    

Unsupervised Text Summarization via Mixed Model Back-Translation 2

Back-translation based approaches have recently lead to significant progress
in unsupervised sequence-to-sequence tasks such as machine translation or style
transfer. In this work, we extend the paradigm to the problem of learning a
sentence...


  
  
    
    
  
  


There’s also a great paper from the FAIR NY folks on self-training with beam generated outputs:

  
      arxiv.org
  
  
    
1909.13788.pdf 1

1052.31 KB

  
  
    
    
  
  


Iirc the model basically learns to reproduce the beam search output with greedy search, so it would improve greedy search performance