valhalla:

To use these models you should fine-tune them


Thanks for this. Shows that I am a newbie but I am still confused about this.
Given that I have 15,000 examples, of my own data, to work with, I am still unclear if I should be using pegasus-large, even though it has huge memory requirements, or if there is a distilled version of pegasus that I can use for fine-tuning on my own data.
It seems somehow illogical for me to START my fine tuning on a model that has ALREADY been fine-tuned on another task… especially because my summarization task doesn’t closely match the exists tasks like xsum, cnn, or etc.
I have been studying ML now for 2 months, and this basic question has escaped me:  "Should I always start with a large model, and fine-tune that, or is it general practice to start with a distilled model, which has already been fine-tuned, and essentially make ANOTHER layer of fine-tuning, on top of that?