Distillation of Pegasus using Pseudo labeling

### Who can help
@sgugger 
@patrickvonplaten 

Models:

- Distillation of Pegasus

## Information

The model I am using (google/pegasus-xsum):

The problem:
- Trying to implement Pegasus Distillation using [Pseudo Labeling](https://github.com/huggingface/transformers/blob/master/examples/research_projects/seq2seq-distillation/precomputed_pseudo_labels.md) in [PRE-TRAINED SUMMARIZATION DISTILLATION](https://arxiv.org/pdf/2010.13002v2.pdf)
- By copying layers from the Teacher model, freezing the positional, token embeddings, all Encoder layers 
- The model trained for two epochs on Xsum-Dataset using cross-entropy loss function between logits of student and output 
generated from the teacher model 
- Generating outputs from the student model gives repeated words and poor generation, although the losses function decreases from 8 to 0.7647 in training and 0.5424 in validation 

```python
[have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have have wing wing wing wing wing wing wing wing wing wing wing wing wing']
```
How can I improve the generation of the model 
