Benchmark-wise speaking, I have some new idea : since SuperGLUE 2 is one of the  most difficult (multi-)task on language understanding. And since T5 is the current SOTA on this benchmark so we can also try embedding vectors from T5.
Previously, this may not be straightforward to extract (since T5 is encoder-decoder), but the latest master version of Huggingface now contains T5 encoder’s only model 8 which we can directly extract the vector of the pretrained model. (Thanks to @agemagician) … So this is interesting choice IMO