Papers have shown that NLP models pretrained on any dataset (even not language, e.g.: music, codes, brackets, etc) can improve the performance on other datasets.
So is there any pretrained nn.Transformer parameters available? I would really appreciate it if anyone could provide one. Specifically, I’m doing a music-related task, so it would be super amazing to find a nn.Transformer model pretrained on music data.