I usually refer to the HuggingFace pre-trained models. You can refer here 1.
I had worked on a music generation project (not directly relying on pre-trained models though).
My guess is that pre-trained models (like BERT, Roberta) may not perform well on music-related tasks because they weren’t trained on music data (rather on more general datasets like Wiki, Book corpus). It is still worth a try though.