hey @SteveMama, have you tried quantizing your model first? this might be a “quick win” to reduce the size of your model by 2-3x.
if that’s not good enough, then my suggestion would be to try movement pruning 3 which is designed for fine-tuning language models and is implemented in the nn_pruning library: GitHub - huggingface/nn_pruning: Prune a model while finetuning or training. 10
(i’m not sure if pegasus is supported in the library, but you can post and issue to find out!)