I can help telling you that you should use Adafactor with PEGASUS and not Adam.

For optimization, both pre-training and fine-tuning used Adafactor (Shazeer & Stern, 2018) with square root learning rate decay and dropout rate of 0.1.