Hi @MoritzLaurer
Thanks for the suggestion.
Quality on supervised benchmarks like GLUE / SuperGLUE does sadly not correlate with performance for dense embedding models. Many of the new models, that perform better on GLUE / SuperGLUE, sadly fail to produce good vector spaces.
You can find some benchmarks here:
https://www.sbert.net/docs/pretrained_models.html#sentence-embedding-models 14
The ‘paraphrase-albert-base-v2’ model performed on par with ‘paraphrase-albert-small-v2’, and both performed worse than e.g. DistilRoBERTa.
So far I made the best experiences with MPNet, but which is sadly not yet available in JAX.
Second best results were achieved by BERT (and variations like TinyBERT, DistilBERT, MiniLM) and RoBERTa.
But the ALBERT model is interesting due to its small size. So if we have enough compute power in that week left, we could also try to tune with ALBERT.