To get good quality language-agnostic sentence embeddings, LaBSE is a good choice. But due to the parameter size(Bert-base size, but #param is 471M), it is hard to fine-tune/deploy appropriately in a small GPU/machine.
So I applied the method of the paper “Load What You Need: Smaller Versions of Multilingual BERT” to get the smaller version of LaBSE, and I can reduce LaBSE’s parameters to 47% without a big performance drop using TF-hub and tensorflow/models.
GitHub: https://github.com/jeongukjae/smaller-labse 15

Relative Links:

Language-agnostic BERT Sentence Embedding(LaBSE) (Paper: [2007.01852] Language-agnostic BERT Sentence Embedding, TF-hub:TensorFlow Hub 1)
Load What You Need: Smaller Versions of Multilingual BERT (Paper: [2010.05609] Load What You Need: Smaller Versions of Multilingual BERT 1, GitHub: https://github.com/Geotrend-research/smaller-transformers 1)